snapshot of https://github.com/nats-io/nats-server.git (main, 2026-09-15; blobs > 131072B omitted)

server/README-MQTT.md

@@ -0,0 +1,595 @@
+**MQTT Implementation Overview**
+
+Revision 1.1
+
+Authors: Ivan Kozlovic, Lev Brouk
+
+NATS Server currently supports most of MQTT 3.1.1. This document describes how
+it is implemented.
+
+It is strongly recommended to review the [MQTT v3.1.1
+specifications](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html)
+and get a detailed understanding before proceeding with this document.
+
+# Contents
+
+1. [Concepts](#1-concepts)
+   - [Server, client](#server-client)
+   - [Connection, client ID, session](#connection-client-id-session)
+   - [Packets, messages, and subscriptions](#packets-messages-and-subscriptions)
+   - [Quality of Service (QoS), publish identifier (PI)](#quality-of-service-qos-publish-identifier-pi)
+   - [Retained message](#retained-message)
+   - [Will message](#will-message)
+2. [Use of JetStream](#2-use-of-jetstream)
+   - [JetStream API](#jetstream-api)
+   - [Streams](#streams)
+   - [Consumers and Internal NATS Subscriptions](#consumers-and-internal-nats-subscriptions)
+3. [Lifecycles](#3-lifecycles)
+   - [Connection, Session](#connection-session)
+   - [Subscription](#subscription)
+   - [Message](#message)
+   - [Retained messages](#retained-messages)
+4. [Implementation Notes](#4-implementation-notes)
+   - [Hooking into NATS I/O](#hooking-into-nats-io)
+   - [Session Management](#session-management)
+   - [Processing QoS acks: PUBACK, PUBREC, PUBCOMP](#processing-qos-acks-puback-pubrec-pubcomp)
+   - [Subject Wildcards](#subject-wildcards)
+5. [Known issues](#5-known-issues)
+
+# 1. Concepts
+
+## Server, client
+
+In the MQTT specification there are concepts of **Client** and **Server**, used
+somewhat interchangeably with those of **Sender** and **Receiver**. A **Server**
+acts as a **Receiver** when it gets `PUBLISH` messages from a **Sender**
+**Client**, and acts as a **Sender** when it delivers them to subscribed
+**Clients**.
+
+In the NATS server implementation there are also concepts (types) `server` and
+`client`. `client` is an internal representation of a (connected) client and
+runs its own read and write loops. Both of these have an `mqtt` field that if
+set makes them behave as MQTT-compliant.
+
+The code and comments may sometimes be confusing as they refer to `server` and
+`client` sometimes ambiguously between MQTT and NATS.
+
+## Connection, client ID, session
+
+When an MQTT client connects to a server, it must send a `CONNECT` packet to
+create an **MQTT Connection**. The packet must include a **Client Identifier**.
+The server will then create or load a previously saved **Session** for the (hash
+of) the client ID.
+
+## Packets, messages, and subscriptions
+
+The low level unit of transmission in MQTT is a **Packet**. Examples of packets
+are: `CONNECT`, `SUBSCRIBE`, `SUBACK`, `PUBLISH`, `PUBCOMP`, etc.
+
+An **MQTT Message** starts with a `PUBLISH` packet that a client sends to the
+server. It is then matched against the current **MQTT Subscriptions** and is
+delivered to them as appropriate. During the message delivery the server acts as
+an MQTT client, and the receiver acts as an MQTT server.
+
+Internally we use **NATS Messages** and **NATS Subscriptions** to facilitate
+message delivery. This may be somewhat confusing as the code refers to `msg` and
+`sub`. What may be even more confusing is that some MQTT packets (specifically,
+`PUBREL`) are represented as NATS messages, and that the original MQTT packet
+"metadata" may be encoded as NATS message headers.
+
+## Quality of Service (QoS), publish identifier (PI)
+
+MQTT specifies 3 levels of quality of service (**QoS**):
+
+- `0` for at most once. A single delivery attempt.
+- `1` for at least once. Will try to redeliver until acknowledged by the
+  receiver.
+- `2` for exactly once. See the [SPEC REF] for the acknowledgement flow.
+
+QoS 1 and 2 messages need to be identified with publish identifiers (**PI**s). A
+PI is a 16-bit integer that must uniquely identify a message for the duration of
+the required exchange of acknowledgment packets.
+
+Note that the QoS applies separately to the transmission of a message from a
+sender client to the server, and from the server to the receiving client. There
+is no protocol-level acknowledgements between the receiver and the original
+sender. The sender passes the ownership of messages to the server, and the
+server then delivers them at maximum possible QoS to the receivers
+(subscribers). The PIs for in-flight outgoing messages are issued and stored per
+session.
+
+## Retained message
+
+A **Retained Message** is not part of any MQTT session and is not removed when the
+session that produced it goes away. Instead, the server needs to persist a
+_single_ retained message per topic. When a subscription is started, the server
+needs to send the “matching” retained messages, that is, messages that would
+have been delivered to the new subscription should that subscription had been
+running prior to the publication of this message.
+
+Retained messages are removed when the server receives a retained message with
+an empty body. Still, this retained message that serves as a “delete” of a
+retained message will be processed as a normal published message.
+
+Retained messages can have QoS.
+
+## Will message
+
+The `CONNECT` packet can contain information about a **Will Message** that needs to
+be sent to any client subscribing on the Will topic/subject in the event that
+the client is disconnected implicitly, that is, not as a result as the client
+sending the `DISCONNECT` packet.
+
+Will messages can have the retain flag and QoS.
+
+# 2. Use of JetStream
+
+The MQTT implementation relies heavily on JetStream. We use it to:
+
+- Persist (and restore) the [Session](#connection-client-id-session) state.
+- Store and retrieve [Retained messages](#retained-message).
+- Persist incoming [QoS 1 and
+  2](#quality-of-service-qos-publish-identifier-pi) messages, and
+  re-deliver if needed.
+- Store and de-duplicate incoming [QoS
+  2](#quality-of-service-qos-publish-identifier-pi) messages.
+- Persist and re-deliver outgoing [QoS
+  2](#quality-of-service-qos-publish-identifier-pi) `PUBREL` packets.
+
+Here is the overview of how we set up and use JetStream **streams**,
+**consumers**, and **internal NATS subscriptions**.
+
+## JetStream API
+
+All interactions with JetStream are performed via `mqttJSA` that sends NATS
+requests to JetStream. Most are processed synchronously and await a response,
+some (e.g. `jsa.sendAck()`) are sent asynchronously. JetStream API is usually
+referred to as `jsa` in the code. No special locking is required to use `jsa`,
+however the asynchronous use of JetStream may create race conditions with
+delivery callbacks.
+
+## Streams
+
+We create the following streams unless they already exist. Failing to ensure the
+streams would prevent the client from connecting.
+
+Each stream is created with a replica value that is determined by the size of
+the cluster but limited to 3. It can also be overwritten by the stream_replicas
+option in the MQTT configuration block.
+
+The streams are created the first time an Account Session Manager is initialized
+and are used by all sessions in it. Note that to avoid race conditions, some
+subscriptions are created first. The streams are never deleted. See
+`mqttCreateAccountSessionManager()` for details.
+
+1. `$MQTT_sess` stores persisted **Session** records. It filters on
+   `"$MQTT.sess.>` subject and has a “limits” policy with `MaxMsgsPer` setting
+   of 1.
+2. `$MQTT_msgs` is used for **QoS 1 and 2 message delivery**.
+   It filters on `$MQTT.msgs.>` subject and has an “interest” policy.
+3. `$MQTT_rmsgs` stores **Retained Messages**. They are all
+   stored (and filtered) on a single subject `$MQTT.rmsg`. This stream has a
+   limits policy.
+4. `$MQTT_qos2in` stores and deduplicates **Incoming QoS 2 Messages**. It
+   filters on `$MQTT.qos2.in.>` and has a "limits" policy with `MaxMsgsPer` of
+   1.
+5. `$MQTT_out` stores **Outgoing QoS 2** `PUBREL` packets. It filters on
+   `$MQTT.out.>` and has a "interest" retention policy.
+
+## Consumers and Internal NATS Subscriptions
+
+### Account Scope
+
+- A durable consumer for [Retained Messages](#retained-message) -
+  `$MQTT_rmsgs_<server name hash>`
+- A subscription to handle all [jsa](#jetstream-api) replies for the account.
+- A subscription to replies to "session persist" requests, so that we can detect
+  the use of a session with the same client ID anywhere in the cluster.
+- 2 subscriptions to support [retained messages](#retained-message):
+  `$MQTT.sub.<nuid>` for the messages themselves, and one to receive replies to
+  "delete retained message" JS API (on the JS reply subject var).
+
+### Session Scope
+
+When a new QoS 2 MQTT subscription is detected in a session, we ensure that
+there is a durable consumer for [QoS
+2](#quality-of-service-qos-publish-identifier-pi) `PUBREL`s out for delivery -
+`$MQTT_PUBREL_<session id hash>`
+
+### Subscription Scope
+
+For all MQTT subscriptions, regardless of their QoS, we create internal NATS subscriptions to
+
+- `subject` (directly encoded from `topic`). This subscription is used to
+  deliver QoS 0 messages, and messages originating from NATS.
+- if needed, `subject fwc` complements `subject` for topics like `topic.#` to
+  include `topic` itself, see [top-level wildcards](#subject-wildcards)
+
+For QoS 1 or 2 MQTT subscriptions we ensure:
+
+- A durable consumer for messages out for delivery - `<session ID hash>_<nuid>`
+- An internal subscription to `$MQTT.sub.<nuid>` to deliver the messages to the
+  receiving client.
+
+### (Old) Notes
+
+As indicated before, for a QoS1 or QoS2 subscription, the server will create a
+JetStream consumer with the appropriate subject filter. If the subscription
+already existed, then only the NATS subscription is created for the JetStream
+consumer’s delivery subject.
+
+Note that JS consumers can be created with an “Replicas” override, which from
+recent discussion is problematic with “Interest” policy streams, which
+“$MQTT_msgs” is.
+
+We do handle situations where a subscription on the same subject filter is sent
+with a different QoS as per MQTT specifications. If the existing was on QoS 1 or
+2, and the “new” is for QoS 0, then we delete the existing JS consumer.
+
+Subscriptions that are QoS 0 have a NATS subscription with the callback function
+being `mqttDeliverMsgCbQos0()`; while QoS 1 and 2 have a NATS subscription with
+callback `mqttDeliverMsgCbQos12()`. Both those functions have comments that
+describe the reason for their existence and what they are doing. For instance
+the `mqttDeliverMsgCbQos0()` callback will reject any producing client that is
+of type JETSTREAM, so that it handles only non JetStream (QoS 1 and 2) messages.
+
+Both these functions end-up calling mqttDeliver() which will first enqueue the
+possible retained messages buffer before delivering any new message. The message
+itself being delivered is serialized in MQTT format and enqueued to the client’s
+outbound buffer and call to addToPCD is made so that it is flushed out of the
+readloop.
+
+# 3. Lifecycles
+
+## Connection, Session
+
+An MQTT connection is created when a listening MQTT server receives a `CONNECT`
+packet. See `mqttProcessConnect()`. A connection is associated with a session.
+Steps:
+
+1. Ensure that we have an `AccountSessionManager` so we can have an
+   `mqttSession`. Lazily initialize JetStream streams, and internal consumers
+   and subscriptions. See `getOrCreateMQTTAccountSessionManager()`.
+2. Find and disconnect any previous session/client for the same ID. See
+   `mqttProcessConnect()`.
+3. Ensure we have an `mqttSession` - create a new or load a previously persisted
+   one. If the clean flag is set in `CONNECT`, clean the session. see
+   `mqttSession.clear()`
+4. Initialize session's subscriptions, if any.
+5. Always send back a `CONNACK` packet. If there were errors in previous steps,
+   include the error.
+
+An MQTT connection can be closed for a number of reasons, including receiving a
+`DISCONNECT` from the client, explicit internal errors processing MQTT packets,
+or the server receiving another `CONNECT` packet with the same client ID. See
+`mqttHandleClosedClient()` and `mqttHandleWill()`. Steps:
+
+1. Send out the Will Message if applicable (if not caused by a `DISCONNECT` packet)
+2. Delete the JetStream consumers for to QoS 1 and 2 packet delivery through
+   JS API calls (if "clean" session flag is set)
+3. Delete the session record from the “$MQTT_sess” stream, based on recorded
+   stream sequence. (if "clean" session flag is set)
+4. Close the client connection.
+
+On an explicit disconnect, that is, the client sends the DISCONNECT packet, the
+server will NOT send the Will, as per specifications.
+
+For sessions that had the “clean” flag, the JS consumers corresponding to QoS 1
+subscriptions are deleted through JS API calls, the session record is then
+deleted (based on recorded stream sequence) from the “$MQTT_sess” stream.
+
+Finally, the client connection is closed
+
+Sessions are persisted on disconnect, and on subscriptions changes.
+
+## Subscription
+
+Receiving an MQTT `SUBSCRIBE` packet creates new subscriptions, or updates
+existing subscriptions in a session. Each `SUBSCRIBE` packet may contain several
+specific subscriptions (`topic` + QoS in each). We always respond with a
+`SUBACK`, which may indicate which subscriptions errored out.
+
+For each subscription in the packet, we:
+
+1. Ignore it if `topic` starts with `$MQTT.sub.`.
+2. Set up QoS 0 message delivery - an internal NATS subscription on `topic`.
+3. Replay any retained messages for `topic`, once as QoS 0.
+4. If we already have a subscription on `topic`, update its QoS
+5. If this is a QoS 2 subscription in the session, ensure we have the [PUBREL
+   consumer](#session-scope) for the session.
+6. If this is a QoS 1 or 2 subscription, ensure we have the [Message
+   consumer](#subscription-scope) for this subscription (or delete one if it
+   exists and this is now a QoS 0 sub).
+7. Add an extra subscription for the [top-level wildcard](#subject-wildcards) case.
+8. Update the session, persist it if changed.
+
+When a session is restored (no clean flag), we go through the same steps to
+re-subscribe to its stored subscription, except step #8 which would have been
+redundant.
+
+When we get an `UNSUBSCRIBE` packet, it can contain multiple subscriptions to
+unsubscribe. The parsing will generate a slice of mqttFilter objects that
+contain the “filter” (the topic with possibly wildcard of the subscription) and
+the QoS value. The server goes through the list and deletes the JS consumer (if
+QoS 1 or 2) and unsubscribes the NATS subscription for the delivery subject (if
+it was a QoS 1 or 2) or on the actual topic/subject. In case of the “#”
+wildcard, the server will handle the “level up” subscriptions that NATS had to
+create.
+
+Again, we update the session and persist it as needed in the `$MQTT_sess`
+stream.
+
+## Message
+
+1. Detect an incoming PUBLISH packet, parse and check the message QoS. Fill out
+   the session's `mqttPublish` struct that contains information about the
+   published message. (see `mqttParse()`, `mqttParsePub()`)
+2. Process the message according to its QoS (see `mqttProcessPub()`)
+
+   - QoS 0:
+     - Initiate message delivery
+   - QoS 1:
+     - Initiate message delivery
+     - Send back a `PUBACK`
+   - QoS 2:
+     - Store the message in `$MQTT_qos2in` stream, using a PI-specific subject.
+       Since `MaxMsgsPer` is set to 1, we will ignore duplicates on the PI.
+     - Send back a `PUBREC`
+     - "Wait" for a `PUBREL`, then initiate message delivery
+     - Remove the previously stored QoS2 message
+     - Send back a `PUBCOMP`
+
+3. Initiate message delivery (see `mqttInitiateMsgDelivery()`)
+
+   - Convert the MQTT `topic` into a NATS `subject` using
+     `mqttTopicToNATSPubSubject()` function. If there is a known subject
+     mapping, then we select the new subject using `selectMappedSubject()`
+     function and then convert back this subject into an MQTT topic using
+     `natsSubjectToMQTTTopic()` function.
+   - Re-serialize the `PUBLISH` packet received as a NATS message. Use NATS
+     headers for the metadata, and the deliverable MQTT `PUBLISH` packet as the
+     contents.
+   - Publish the messages as `subject` (and `subject fwc` if applicable, see
+     [subject wildcards](#subject-wildcards)). Use the "standard" NATS
+     `c.processInboundClientMsg()` to do that. `processInboundClientMsg()` will
+     distribute the message to any NATS subscriptions (including routes,
+     gateways, leafnodes) and the relevant MQTT subscriptions.
+   - Check for retained messages, process as needed. See
+     `c.processInboundClientMsg()` calling `c.mqttHandlePubRetain()` For MQTT
+     clients.
+   - If the message QoS is 1 or 2, store it in `$MQTT_msgs` stream as
+     `$MQTT.msgs.<subject>` for "at least once" delivery with retries.
+
+4. Let NATS and JetStream deliver to the internal subscriptions, and to the
+   receiving clients. See `mqttDeliverMsgCb...()`
+
+   - The NATS message posted to `subject` (and `subject fwc`) will be delivered
+     to each relevant internal subscription by calling `mqttDeliverMsgCbQoS0()`.
+     The function has access to both the publishing and the receiving clients.
+
+     - Ignore all irrelevant invocations. Specifically, do nothing if the
+       message needs to be delivered with a higher QoS - that will be handled by
+       the other, `...QoS12` callback. Note that if the original message was
+       publuished with a QoS 1 or 2, but the subscription has its maximum QoS
+       set to 0, the message will be delivered by this callback.
+     - Ignore "reserved" subscriptions, as per MQTT spec.
+     - Decode delivery `topic` from the NATS `subject`.
+     - Write (enqueue) outgoing `PUBLISH` packet.
+     - **DONE for QoS 0**
+
+   - The NATS message posted to JetStream as `$MQTT.msgs.subject` will be
+     consumed by subscription-specific consumers. Note that MQTT subscriptions
+     with max QoS 0 do not have JetStream consumers. They are handled by the
+     QoS0 callback.
+
+     The consumers will deliver it to the `$MQTT.sub.<nuid>`
+     subject for their respective NATS subscriptions by calling
+     `mqttDeliverMsgCbQoS12()`. This callback too has access to both the
+     publishing and the receiving clients.
+
+     - Ignore "reserved" subscriptions, as per MQTT spec.
+     - See if this is a re-delivery from JetStream by checking `sess.cpending`
+       for the JS reply subject. If so, use the existing PI and treat this as a
+       duplicate redelivery.
+     - Otherwise, assign the message a new PI (see `trackPublish()` and
+       `bumpPI()`) and store it in `sess.cpending` and `sess.pendingPublish`,
+       along with the JS reply subject that can be used to remove this pending
+       message from the consumer once it's delivered to the receipient.
+     - Decode delivery `topic` from the NATS `subject`.
+     - Write (enqueue) outgoing `PUBLISH` packet.
+
+5. QoS 1: "Wait" for a `PUBACK`. See `mqttProcessPubAck()`.
+
+   - When received, remove the PI from the tracking maps, send an ACK to
+     consumer to remove the message.
+   - **DONE for QoS 1**
+
+6. QoS 2: "Wait" for a `PUBREC`. When received, we need to do all the same
+   things as in the QoS 1 `PUBACK` case, but we need to send out a `PUBREL`, and
+   continue using the same PI until the delivery flow is complete and we get
+   back a `PUBCOMP`. For that, we add the PI to `sess.pendingPubRel`, and to
+   `sess.cpending` with the PubRel consumer durable name.
+
+   We also compose and store a headers-only NATS message signifying a `PUBREL`
+   out for delivery, and store it in the `$MQTT_qos2out` stream, as
+   `$MQTT.qos2.out.<session-id>`.
+
+7. QoS 2: Deliver `PUBREL`. The PubRel session-specific consumer will publish to
+   internal subscription on `$MQTT.qos2.delivery`, calling
+   `mqttDeliverPubRelCb()`. We store the ACK reply subject in `cpending` to
+   remove the JS message on `PUBCOMP`, compose and send out a `PUBREL` packet.
+
+8. QoS 2: "Wait" for a `PUBCOMP`. See `mqttProcessPubComp()`.
+   - When received, remove the PI from the tracking maps, send an ACK to
+     consumer to remove the `PUBREL` message.
+   - **DONE for QoS 2**
+
+## Retained messages
+
+When we process an inbound `PUBLISH` and submit it to
+`processInboundClientMsg()` function, for MQTT clients it will invoke
+`mqttHandlePubRetain()` which checks if the published message is “retained” or
+not.
+
+If it is, then we construct a record representing the retained message and store
+it in the `$MQTT_rmsg` stream, under the single `$MQTT.rmsg` subject. The stored
+record (in JSON) contains information about the subject, topic, MQTT flags, user
+that produced this message and the message content itself. It is stored and the
+stream sequence is remembered in the memory structure that contains retained
+messages.
+
+Note that when creating an account session manager, the retained messages stream
+is read from scratch to load all the messages through the use of a JS consumer.
+The associated subscription will process the recovered retained messages or any
+new that comes from the network.
+
+A retained message is added to a map and a subscription is created and inserted
+into a sublist that will be used to perform a ReverseMatch() when a subscription
+is started and we want to find all retained messages that the subscription would
+have received if it had been running prior to the message being published.
+
+If a retained message on topic “foo” already exists, then the server has to
+delete the old message at the stream sequence we saved when storing it.
+
+This could have been done with having retained messages stored under
+`$MQTT.rmsg.<subject>` as opposed to all under a single subject, and make use of
+the `MaxMsgsPer` field set to 1. The `MaxMsgsPer` option was introduced well into
+the availability of MQTT and changes to the sessions was made in [PR
+#2501](https://github.com/nats-io/nats-server/pull/2501), with a conversion of
+existing streams such as `$MQTT*sess*<sess ID>` into a single stream with unique
+subjects, but the changes were not made to the retained messages stream.
+
+There are also subscriptions for the handling of retained messages which are
+messages that are asked by the publisher to be retained by the MQTT server to be
+delivered to matching subscriptions when they start. There is a single message
+per topic. Retained messages are deleted when the user sends a retained message
+(there is a flag in the PUBLISH protocol) on a given topic with an empty body.
+The difficulty with retained messages is to handle them in a cluster since all
+servers need to be aware of their presence so that they can deliver them to
+subscriptions that those servers may become the leader for.
+
+- `$MQTT_rmsgs` which has a “limits” policy and holds retained messages, all
+  under `$MQTT.rmsg` single subject. Not sure why I did not use MaxMsgsPer for
+  this stream and not filter `$MQTT.rmsg.>`.
+
+The first step when processing a new subscription is to gather the retained
+messages that would be a match for this subscription. To do so, the server will
+serialize into a buffer all messages for the account session manager’s sublist’s
+ReverseMatch result. We use the returned subscriptions’ subject to find from a
+map appropriate retained message (see `serializeRetainedMsgsForSub()` for
+details).
+
+# 4. Implementation Notes
+
+## Hooking into NATS I/O
+
+### Starting the accept loop
+
+The MQTT accept loop is started when the server detects that an MQTT port has
+been defined in the configuration file. It works similarly to all other accept
+loops. Note that for MQTT over websocket, the websocket port has to be defined
+and MQTT clients will connect to that port instead of the MQTT port and need to
+provide `/mqtt` as part of the URL to redirect the creation of the client to an
+MQTT client (with websocket support) instead of a regular NATS with websocket.
+See the branching done in `startWebsocketServer()`. See `startMQTT()`.
+
+### Starting the read/write loops
+
+When a TCP connection is accepted, the internal go routine will invoke
+`createMQTTClient()`. This function will set a `c.mqtt` object that will make it
+become an MQTT client (through the `isMqtt()` helper function). The `readLoop()`
+and `writeLoop()` are started similarly to other clients. However, the read loop
+will branch out to `mqttParse()` instead when detecting that this is an MQTT
+client.
+
+## Session Management
+
+### Account Session Manager
+
+`mqttAccountSessionManager` is an object that holds the state of all sessions in
+an account. It also manages the lifecycle of JetStream streams and internal
+subscriptions for processing JS API replies, session updates, etc. See
+`mqttCreateAccountSessionManager()`. It is lazily initialized upon the first
+MQTT `CONNECT` packet received. Account session manager is referred to as `asm`
+in the code.
+
+Note that creating the account session manager (and attempting to create the
+streams) is done only once per account on a given server, since once created the
+account session manager for a given account would be found in the sessions map
+of the mqttSessionManager object.
+
+### Find and disconnect previous session/client
+
+Once all that is done, we now go to the creation of the session object itself.
+For that, we first need to make sure that it does not already exist, meaning
+that it is registered on the server - or anywhere in the cluster. Note that MQTT
+dictates that if a session with the same ID connects, the OLD session needs to
+be closed, not the new one being created. NATS Server complies with this
+requirement.
+
+Once a session is detected to already exists, the old one (as described above)
+is closed and the new one accepted, however, the session ID is maintained in a
+flappers map so that we detect situations where sessions with the same ID are
+started multiple times causing the previous one to be closed. When that
+detection occurs, the newly created session is put in “jail” for a second to
+avoid a very rapid succession of connect/disconnect. This has already been seen
+by users since there was some issue there where we would schedule the connection
+closed instead of waiting in place which was causing a panic.
+
+We also protect from multiple clients on a given server trying to connect with
+the same ID at the “same time” while the processing of a CONNECT of a session is
+not yet finished. This is done with the use of a sessLocked map, keyed by the
+session ID.
+
+### Create or restore the session
+
+If everything is good up to that point, the server will either create or restore
+a session from the stream. This is done in the `createOrRestoreSession()`
+function. The client/session ID is hashed and added to the session’s stream
+subject along with the JS domain to prevent clients connecting from different
+domains to “pollute” the session stream of a given domain.
+
+Since each session constitutes a subject and the stream has a maximum of 1
+message per subject, we attempt to load the last message on the formed subject.
+If we don’t find it, then the session object is created “empty”, while if we
+find a record, we create the session object based on the record persisted on the
+stream.
+
+If the session was restored from the JS stream, we keep track of the stream
+sequence where the record was located. When we save the session (even if it
+already exists) we will use this sequence number to set the
+`JSExpectedLastSubjSeq` header so that we handle possibly different servers in a
+(super)cluster to detect the race of clients trying to use the same session ID,
+since only one of the write should succeed. On success, the session’s new
+sequence is remembered by the server that did the write.
+
+When created or restored, the CONNACK can now be sent back to the client, and if
+there were any recovered subscriptions, they are now processed.
+
+## Processing QoS acks: PUBACK, PUBREC, PUBCOMP
+
+When the server delivers a message with QoS 1 or 2 (also a `PUBREL` for QoS 2) to a subscribed client, the client will send back an acknowledgement. See `mqttProcessPubAck()`, `mqttProcessPubRec()`, and `mqttProcessPubComp()`
+
+While the specific logic for each packet differs, these handlers all update the
+session's PI mappings (`cpending`, `pendingPublish`, `pendingPubRel`), and if
+needed send an ACK to JetStream to remove the message from its consumer and stop
+the re-delivery attempts.
+
+## Subject Wildcards
+
+Note that MQTT subscriptions have wildcards too, the `“+”` wildcard is equivalent
+to NATS’s `“*”` wildcard, however, MQTT’s wildcard `“#”` is similar to `“>”`, except
+that it also includes the level above. That is, a subscription on `“foo/#”` would
+receive messages on `“foo/bar/baz”`, but also on `“foo”`.
+
+So, for MQTT subscriptions enging with a `'#'` we are forced to create 2
+internal NATS subscriptions, one on `“foo”` and one on `“foo.>”`.
+
+# 5. Known issues
+- "active" redelivery for QoS from JetStream (compliant, just a note)
+- JetStream QoS redelivery happens out of (original) order
+- finish delivery of in-flight messages after UNSUB
+- finish delivery of in-flight messages after a reconnect
+- consider replacing `$MQTT_msgs` with `$MQTT_out`.
+- consider using unique `$MQTT.rmsg.>` and `MaxMsgsPer` for retained messages.
+- add a cli command to list/clean old sessions

server/README.md

@@ -0,0 +1,17 @@
+# Tests
+
+Tests that run on Travis have been split into jobs that run in their own VM in parallel. This reduces the overall running time but also is allowing recycling of a job when we get a flapper as opposed to have to recycle the whole test suite.
+
+## JetStream Tests
+
+For JetStream tests, we need to observe a naming convention so that no tests are omitted when running on Travis.
+
+The script `runTestsOnTravis.sh` will run a given job based on the definition found in "`.travis.yml`".
+
+As for the naming convention:
+
+- All JetStream test name should start with `TestJetStream`
+- Cluster tests should go into `jetstream_cluster_test.go` and start with `TestJetStreamCluster`
+- Super-cluster tests should go into `jetstream_super_cluster_test.go` and start with `TestJetStreamSuperCluster`
+
+Not following this convention means that some tests may not be executed on Travis.

server/archive/archive.go

@@ -0,0 +1,356 @@
+// Copyright 2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package archive
+
+import (
+	"bufio"
+	"encoding/binary"
+	"errors"
+	"io"
+)
+
+const MagicBytes = "NATSARC1"
+
+// maxNameLen bounds the entry name length accepted when reading an archive,
+// avoiding an unbounded allocation on corrupt or malicious input. The name is
+// the only variable-length field read directly from the stream.
+const maxNameLen = 1 << 20
+
+var (
+	ErrClosed            = errors.New("archive: closed")
+	ErrInvalidArchive    = errors.New("archive: invalid archive stream")
+	ErrIncompleteEntry   = errors.New("archive: entry not fully written")
+	ErrNoActiveEntry     = errors.New("archive: no active entry")
+	ErrWriteTooLong      = errors.New("archive: write exceeds declared entry size")
+	ErrNilHeader         = errors.New("archive: nil header")
+	ErrNegativeEntrySize = errors.New("archive: negative entry size")
+)
+
+// Header describes one archive entry.
+//
+// On the wire each entry is the encoded header fields followed by the payload.
+// HeaderSize and PayloadSize describe how that payload is split (e.g. message
+// headers vs. body); the payload length is their sum and is not stored
+// separately. Sequence is always encoded, with 0 meaning "unset".
+type Header struct {
+	Name        string
+	HeaderSize  int64
+	PayloadSize int64
+	Timestamp   int64
+	Sequence    uint64
+}
+
+func (h *Header) clone() *Header {
+	if h == nil {
+		return nil
+	}
+	c := *h
+	return &c
+}
+
+// Writer writes a stream of archive entries.
+type Writer struct {
+	w          io.Writer
+	wroteMagic bool
+	header     *Header
+	remaining  int64
+	closed     bool
+}
+
+// NewWriter returns a new stream writer.
+func NewWriter(w io.Writer) *Writer {
+	return &Writer{w: w}
+}
+
+// WriteHeader starts a new archive entry.
+func (a *Writer) WriteHeader(hdr *Header) error {
+	if a.closed {
+		return ErrClosed
+	}
+	if hdr == nil {
+		return ErrNilHeader
+	}
+	if hdr.HeaderSize < 0 || hdr.PayloadSize < 0 {
+		return ErrNegativeEntrySize
+	}
+	if a.header != nil && a.remaining > 0 {
+		return ErrIncompleteEntry
+	}
+	totalPayloadSize := hdr.HeaderSize + hdr.PayloadSize
+	if totalPayloadSize < 0 {
+		return ErrNegativeEntrySize
+	}
+	if !a.wroteMagic {
+		if _, err := io.WriteString(a.w, MagicBytes); err != nil {
+			return err
+		}
+		a.wroteMagic = true
+	}
+	if err := a.writeUvarint(uint64(len(hdr.Name))); err != nil {
+		return err
+	}
+	if err := a.writeVarint(hdr.Timestamp); err != nil {
+		return err
+	}
+	if err := a.writeUvarint(hdr.Sequence); err != nil {
+		return err
+	}
+	if err := a.writeUvarint(uint64(hdr.HeaderSize)); err != nil {
+		return err
+	}
+	if err := a.writeUvarint(uint64(hdr.PayloadSize)); err != nil {
+		return err
+	}
+	if _, err := io.WriteString(a.w, hdr.Name); err != nil {
+		return err
+	}
+	a.header = hdr.clone()
+	a.remaining = totalPayloadSize
+	if a.remaining == 0 {
+		a.header = nil
+	}
+	return nil
+}
+
+// Write writes payload bytes for the current entry.
+func (a *Writer) Write(p []byte) (int, error) {
+	if a.closed {
+		return 0, ErrClosed
+	}
+	if len(p) == 0 {
+		return 0, nil
+	}
+	if a.header == nil {
+		return 0, ErrNoActiveEntry
+	}
+	wanted := len(p)
+	if int64(wanted) > a.remaining {
+		wanted = int(a.remaining)
+	}
+	n, err := a.w.Write(p[:wanted])
+	a.remaining -= int64(n)
+	if err != nil {
+		return n, err
+	}
+	if n < wanted {
+		return n, io.ErrShortWrite
+	}
+	if len(p) > wanted {
+		if a.remaining == 0 {
+			a.header = nil
+		}
+		return n, ErrWriteTooLong
+	}
+	if a.remaining == 0 {
+		a.header = nil
+	}
+	return n, nil
+}
+
+// Close finalizes the stream writer.
+func (a *Writer) Close() error {
+	if a.closed {
+		return nil
+	}
+	a.closed = true
+	if a.header != nil && a.remaining > 0 {
+		return ErrIncompleteEntry
+	}
+	return nil
+}
+
+// Flush flushes the underlying writer if it supports flushing.
+func (a *Writer) Flush() error {
+	if a.closed {
+		return ErrClosed
+	}
+	if f, ok := a.w.(interface{ Flush() error }); ok {
+		return f.Flush()
+	}
+	if f, ok := a.w.(interface{ Flush() }); ok {
+		f.Flush()
+	}
+	return nil
+}
+
+// Reader reads a stream of archive entries.
+type Reader struct {
+	r         *bufio.Reader
+	header    *Header
+	remaining int64
+	seenMagic bool
+	err       error
+}
+
+// NewReader returns a new stream reader.
+func NewReader(r io.Reader) *Reader {
+	return &Reader{r: bufio.NewReader(r)}
+}
+
+func (u *Reader) discardCurrent() error {
+	if u.remaining == 0 {
+		u.header = nil
+		return nil
+	}
+	_, err := io.CopyN(io.Discard, u.r, u.remaining)
+	u.remaining = 0
+	u.header = nil
+	if errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) {
+		return ErrInvalidArchive
+	}
+	return err
+}
+
+// Next advances to the next entry and returns its header.
+func (u *Reader) Next() (*Header, error) {
+	if u.err != nil {
+		return nil, u.err
+	}
+	if err := u.discardCurrent(); err != nil {
+		u.err = err
+		return nil, err
+	}
+	if !u.seenMagic {
+		magic := make([]byte, len(MagicBytes))
+		if _, err := io.ReadFull(u.r, magic); err != nil {
+			if errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) {
+				u.err = io.EOF
+				return nil, io.EOF
+			}
+			u.err = err
+			return nil, err
+		}
+		if string(magic) != MagicBytes {
+			u.err = ErrInvalidArchive
+			return nil, ErrInvalidArchive
+		}
+		u.seenMagic = true
+	}
+	// A clean io.EOF on the first field is the normal end of the stream.
+	nameLen, err := u.readUvarint()
+	if err != nil {
+		u.err = err
+		return nil, err
+	}
+	timestamp, err := u.readVarint()
+	if err != nil {
+		u.err = noEOF(err)
+		return nil, u.err
+	}
+	sequence, err := u.readUvarint()
+	if err != nil {
+		u.err = noEOF(err)
+		return nil, u.err
+	}
+	headerSize, err := u.readUvarint()
+	if err != nil {
+		u.err = noEOF(err)
+		return nil, u.err
+	}
+	payloadSize, err := u.readUvarint()
+	if err != nil {
+		u.err = noEOF(err)
+		return nil, u.err
+	}
+	// HeaderSize/PayloadSize are stored as int64: a value above MaxInt64
+	// converts to a negative int64, and an overflowing sum lands negative too,
+	// so the sign checks below cover both cases.
+	hdrSize, plSize := int64(headerSize), int64(payloadSize)
+	total := hdrSize + plSize
+	if nameLen > maxNameLen || hdrSize < 0 || plSize < 0 || total < 0 {
+		u.err = ErrInvalidArchive
+		return nil, ErrInvalidArchive
+	}
+	name := make([]byte, nameLen)
+	if _, err := io.ReadFull(u.r, name); err != nil {
+		u.err = ErrInvalidArchive
+		return nil, ErrInvalidArchive
+	}
+	u.header = &Header{
+		Name:        string(name),
+		Timestamp:   timestamp,
+		Sequence:    sequence,
+		HeaderSize:  hdrSize,
+		PayloadSize: plSize,
+	}
+	u.remaining = total
+	return u.header.clone(), nil
+}
+
+// Read reads bytes from the current entry payload.
+func (u *Reader) Read(p []byte) (int, error) {
+	if u.err != nil {
+		return 0, u.err
+	}
+	if len(p) == 0 {
+		return 0, nil
+	}
+	if u.header == nil || u.remaining == 0 {
+		return 0, io.EOF
+	}
+
+	if int64(len(p)) > u.remaining {
+		p = p[:int(u.remaining)]
+	}
+	n, err := io.ReadFull(u.r, p)
+	u.remaining -= int64(n)
+	if err != nil {
+		u.err = ErrInvalidArchive
+		return n, ErrInvalidArchive
+	}
+	if u.remaining == 0 {
+		u.header = nil
+	}
+	return n, nil
+}
+
+// noEOF converts a clean io.EOF into ErrInvalidArchive. It is used for header
+// fields after the first: reaching EOF there means the entry was truncated
+// rather than that the stream ended cleanly on an entry boundary.
+func noEOF(err error) error {
+	if err != nil && errors.Is(err, io.EOF) {
+		return ErrInvalidArchive
+	}
+	return err
+}
+
+func (a *Writer) writeVarint(v int64) error {
+	var tmp [binary.MaxVarintLen64]byte
+	n := binary.PutVarint(tmp[:], v)
+	_, err := a.w.Write(tmp[:n])
+	return err
+}
+
+func (a *Writer) writeUvarint(v uint64) error {
+	var tmp [binary.MaxVarintLen64]byte
+	n := binary.PutUvarint(tmp[:], v)
+	_, err := a.w.Write(tmp[:n])
+	return err
+}
+
+func (u *Reader) readVarint() (int64, error) {
+	v, err := binary.ReadVarint(u.r)
+	if err != nil && !errors.Is(err, io.EOF) {
+		return v, ErrInvalidArchive
+	}
+	return v, err
+}
+
+func (u *Reader) readUvarint() (uint64, error) {
+	v, err := binary.ReadUvarint(u.r)
+	if err != nil && !errors.Is(err, io.EOF) {
+		return v, ErrInvalidArchive
+	}
+	return v, err
+}

server/archive/archive_test.go

@@ -0,0 +1,244 @@
+// Copyright 2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package archive
+
+import (
+	"bytes"
+	"errors"
+	"io"
+	"testing"
+)
+
+func TestRoundTrip(t *testing.T) {
+	var buf bytes.Buffer
+
+	w := NewWriter(&buf)
+
+	want := []struct {
+		hdr  Header
+		data string
+	}{
+		{
+			hdr: Header{
+				Name:        "alpha.txt",
+				Timestamp:   100,
+				HeaderSize:  4,
+				PayloadSize: 7,
+			},
+			data: "hello alpha",
+		},
+		{
+			hdr: Header{
+				Name:        "nested/beta.txt",
+				Timestamp:   100,
+				Sequence:    42,
+				HeaderSize:  3,
+				PayloadSize: 9,
+			},
+			data: "beta payload",
+		},
+	}
+
+	for i := range want {
+		if err := w.WriteHeader(&want[i].hdr); err != nil {
+			t.Fatalf("write header %d: %v", i, err)
+		}
+		if _, err := io.WriteString(w, want[i].data); err != nil {
+			t.Fatalf("write payload %d: %v", i, err)
+		}
+	}
+	if err := w.Close(); err != nil {
+		t.Fatalf("close writer: %v", err)
+	}
+
+	r := NewReader(bytes.NewReader(buf.Bytes()))
+
+	for i := range want {
+		hdr, err := r.Next()
+		if err != nil {
+			t.Fatalf("next %d: %v", i, err)
+		}
+		if hdr.Name != want[i].hdr.Name {
+			t.Fatalf("header %d name mismatch: got %q want %q", i, hdr.Name, want[i].hdr.Name)
+		}
+		if hdr.Timestamp != want[i].hdr.Timestamp {
+			t.Fatalf("header %d timestamp mismatch: got %d want %d", i, hdr.Timestamp, want[i].hdr.Timestamp)
+		}
+		if hdr.Sequence != want[i].hdr.Sequence {
+			t.Fatalf("header %d sequence mismatch: got %d want %d", i, hdr.Sequence, want[i].hdr.Sequence)
+		}
+		if hdr.HeaderSize != want[i].hdr.HeaderSize {
+			t.Fatalf("header %d header size mismatch: got %d want %d", i, hdr.HeaderSize, want[i].hdr.HeaderSize)
+		}
+		if hdr.PayloadSize != want[i].hdr.PayloadSize {
+			t.Fatalf("header %d payload size mismatch: got %d want %d", i, hdr.PayloadSize, want[i].hdr.PayloadSize)
+		}
+		got, err := io.ReadAll(r)
+		if err != nil {
+			t.Fatalf("read payload %d: %v", i, err)
+		}
+		if string(got) != want[i].data {
+			t.Fatalf("payload %d mismatch: got %q want %q", i, got, want[i].data)
+		}
+	}
+
+	if _, err := r.Next(); !errors.Is(err, io.EOF) {
+		t.Fatalf("expected EOF, got %v", err)
+	}
+}
+
+func TestNextSkipsUnreadPayload(t *testing.T) {
+	var buf bytes.Buffer
+
+	w := NewWriter(&buf)
+	first := Header{Name: "first", HeaderSize: 0, PayloadSize: int64(len("abcdef"))}
+	if err := w.WriteHeader(&first); err != nil {
+		t.Fatalf("write first header: %v", err)
+	}
+	if _, err := io.WriteString(w, "abcdef"); err != nil {
+		t.Fatalf("write first payload: %v", err)
+	}
+
+	second := Header{Name: "second", HeaderSize: 0, PayloadSize: int64(len("second-body"))}
+	if err := w.WriteHeader(&second); err != nil {
+		t.Fatalf("write second header: %v", err)
+	}
+	if _, err := io.WriteString(w, "second-body"); err != nil {
+		t.Fatalf("write second payload: %v", err)
+	}
+
+	r := NewReader(bytes.NewReader(buf.Bytes()))
+
+	hdr, err := r.Next()
+	if err != nil {
+		t.Fatalf("next first: %v", err)
+	}
+	if hdr.Name != "first" {
+		t.Fatalf("unexpected first header name %q", hdr.Name)
+	}
+
+	buf1 := make([]byte, 3)
+	if _, err := io.ReadFull(r, buf1); err != nil {
+		t.Fatalf("read partial first payload: %v", err)
+	}
+	if string(buf1) != "abc" {
+		t.Fatalf("unexpected partial payload %q", buf1)
+	}
+
+	hdr, err = r.Next()
+	if err != nil {
+		t.Fatalf("next second: %v", err)
+	}
+	if hdr.Name != "second" {
+		t.Fatalf("unexpected second header name %q", hdr.Name)
+	}
+
+	got, err := io.ReadAll(r)
+	if err != nil {
+		t.Fatalf("read second payload: %v", err)
+	}
+	if string(got) != "second-body" {
+		t.Fatalf("unexpected second payload %q", got)
+	}
+}
+
+// validArchive returns a well-formed archive holding a single entry whose
+// payload is the 6-byte string "abcdef" split as HeaderSize=2, PayloadSize=4.
+func validArchive(t *testing.T) []byte {
+	t.Helper()
+	var buf bytes.Buffer
+	w := NewWriter(&buf)
+	hdr := Header{Name: "first", Timestamp: 7, Sequence: 3, HeaderSize: 2, PayloadSize: 4}
+	if err := w.WriteHeader(&hdr); err != nil {
+		t.Fatalf("write header: %v", err)
+	}
+	if _, err := io.WriteString(w, "abcdef"); err != nil {
+		t.Fatalf("write payload: %v", err)
+	}
+	if err := w.Close(); err != nil {
+		t.Fatalf("close: %v", err)
+	}
+	return buf.Bytes()
+}
+
+func TestEmptyArchiveReturnsEOF(t *testing.T) {
+	r := NewReader(bytes.NewReader(nil))
+	if _, err := r.Next(); !errors.Is(err, io.EOF) {
+		t.Fatalf("expected io.EOF on empty archive, got %v", err)
+	}
+}
+
+func TestInvalidMagicIsRejected(t *testing.T) {
+	r := NewReader(bytes.NewReader([]byte("not-a-valid-archive-stream")))
+	if _, err := r.Next(); !errors.Is(err, ErrInvalidArchive) {
+		t.Fatalf("expected ErrInvalidArchive, got %v", err)
+	}
+}
+
+// A clean EOF on the first header field means end-of-stream, but an EOF
+// partway through the header means the entry was truncated.
+func TestTruncatedHeaderIsInvalid(t *testing.T) {
+	full := validArchive(t)
+	// Keep the magic plus the single name-length byte, dropping the rest.
+	r := NewReader(bytes.NewReader(full[:len(MagicBytes)+1]))
+	if _, err := r.Next(); !errors.Is(err, ErrInvalidArchive) {
+		t.Fatalf("expected ErrInvalidArchive, got %v", err)
+	}
+}
+
+func TestTruncatedPayloadIsInvalid(t *testing.T) {
+	full := validArchive(t)
+	// Drop the last 3 payload bytes; the header still decodes cleanly.
+	r := NewReader(bytes.NewReader(full[:len(full)-3]))
+	hdr, err := r.Next()
+	if err != nil {
+		t.Fatalf("next: %v", err)
+	}
+	if hdr.HeaderSize+hdr.PayloadSize != 6 {
+		t.Fatalf("unexpected payload size %d", hdr.HeaderSize+hdr.PayloadSize)
+	}
+	if _, err := io.ReadAll(r); !errors.Is(err, ErrInvalidArchive) {
+		t.Fatalf("expected ErrInvalidArchive reading payload, got %v", err)
+	}
+}
+
+// The header round-trips when its fields are read directly from the stream,
+// including a large Sequence (uvarint) and a negative Timestamp (varint).
+func TestHeaderFieldEncoding(t *testing.T) {
+	var buf bytes.Buffer
+	w := NewWriter(&buf)
+	want := Header{Name: "nested/x.dat", Timestamp: -123, Sequence: 1 << 60, HeaderSize: 0, PayloadSize: 0}
+	if err := w.WriteHeader(&want); err != nil {
+		t.Fatalf("write header: %v", err)
+	}
+	if err := w.Close(); err != nil {
+		t.Fatalf("close: %v", err)
+	}
+
+	r := NewReader(bytes.NewReader(buf.Bytes()))
+	got, err := r.Next()
+	if err != nil {
+		t.Fatalf("next: %v", err)
+	}
+	if *got != want {
+		t.Fatalf("header round-trip mismatch: got %+v want %+v", *got, want)
+	}
+	// A zero-length payload yields immediate EOF, then a clean end-of-stream.
+	if n, err := r.Read(make([]byte, 4)); n != 0 || !errors.Is(err, io.EOF) {
+		t.Fatalf("expected (0, EOF) reading empty payload, got (%d, %v)", n, err)
+	}
+	if _, err := r.Next(); !errors.Is(err, io.EOF) {
+		t.Fatalf("expected io.EOF after last entry, got %v", err)
+	}
+}

server/ats/ats.go

@@ -0,0 +1,86 @@
+// Copyright 2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// ats controls the go routines for the access time service.
+// This allows more efficient unixnano operations for cache access times.
+// We will have one per binary (usually per server).
+package ats
+
+import (
+	"sync/atomic"
+	"time"
+)
+
+// Update every 100ms for gathering access time in unix nano.
+const TickInterval = 100 * time.Millisecond
+
+var (
+	// Our unix nano time.
+	utime atomic.Int64
+	// How may registered users do we have, controls lifetime of Go routine.
+	refs atomic.Int64
+	// To signal the shutdown of the Go routine.
+	done chan struct{}
+)
+
+func init() {
+	// Initialize our done chan.
+	done = make(chan struct{}, 1)
+}
+
+// Register usage. This will happen on filestore creation.
+func Register() {
+	if v := refs.Add(1); v == 1 {
+		// This is the first to register (could also go up and down),
+		// so spin up Go routine and grab initial time.
+		utime.Store(time.Now().UnixNano())
+
+		go func() {
+			ticker := time.NewTicker(TickInterval)
+			defer ticker.Stop()
+			for {
+				select {
+				case <-ticker.C:
+					utime.Store(time.Now().UnixNano())
+				case <-done:
+					return
+				}
+			}
+		}()
+	}
+}
+
+// Unregister usage. We will shutdown the go routine if no more registered users.
+func Unregister() {
+	if v := refs.Add(-1); v == 0 {
+		done <- struct{}{}
+	} else if v < 0 {
+		refs.Store(0)
+		panic("unbalanced unregister for access time state")
+	}
+}
+
+// Will load the access time from an atomic.
+// If no one has registered this will return 0 or stale data.
+// It is the responsibility of the user to properly register and unregister.
+func AccessTime() int64 {
+	// Return last updated time.
+	v := utime.Load()
+	if v == 0 {
+		// Always register a time, the worst case is a stale time.
+		// On startup, we can register in parallel and could previously panic.
+		v = time.Now().UnixNano()
+		utime.Store(v)
+	}
+	return v
+}

server/ats/ats_test.go

@@ -0,0 +1,98 @@
+// Copyright 2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ats
+
+import (
+	"runtime"
+	"testing"
+	"time"
+)
+
+func TestNotRunningValue(t *testing.T) {
+	// Set back to zero in case this test gets run multiple times via --count.
+	utime.Store(0)
+	at := AccessTime()
+	if at == 0 {
+		t.Fatal("Expected non-zero access time")
+	}
+
+	atn := AccessTime()
+	if atn != at {
+		t.Fatal("Did not expect updates to access time")
+	}
+}
+
+func TestRegisterAndUnregister(t *testing.T) {
+	ngrp := runtime.NumGoroutine()
+
+	Register()
+	if r := refs.Load(); r != 1 {
+		t.Fatalf("Expected refs to be 1, got %d", r)
+	}
+	// Make sure we have a non-zero time.
+	at := AccessTime()
+	if at == 0 {
+		t.Fatal("Expected non-zero access time")
+	}
+	// Make sure we are updating access time.
+	time.Sleep(2 * TickInterval)
+	atn := AccessTime()
+	if atn == at {
+		t.Fatal("Expected access time to be updated but it was not")
+	}
+
+	// Now unregister.
+	Unregister()
+	if r := refs.Load(); r != 0 {
+		t.Fatalf("Expected refs to be 0, got %d", r)
+	}
+	time.Sleep(TickInterval)
+
+	at = AccessTime()
+	time.Sleep(2 * TickInterval)
+	atn = AccessTime()
+	if atn != at {
+		t.Fatal("Did not expect updates to access time")
+	}
+
+	// Check that we have no additional go routines running.
+	ngra := runtime.NumGoroutine()
+	if ngra != ngrp {
+		t.Fatalf("Expected same number of go routines after removing all registered: %d vs %d", ngrp, ngra)
+	}
+
+	// Check that we spin back up on going from zero to one registered after spinning down.
+	Register()
+	defer Unregister()
+
+	at = AccessTime()
+	time.Sleep(2 * TickInterval)
+	atn = AccessTime()
+	if atn == at {
+		t.Fatal("Expected access time to be updated but it was not")
+	}
+	ngra = runtime.NumGoroutine()
+	if ngra != ngrp+1 {
+		t.Fatalf("Expected to see additional go routine: %d vs %d", ngrp, ngra)
+	}
+}
+
+func TestUnbalancedUnregister(t *testing.T) {
+	defer func() {
+		if r := recover(); r == nil {
+			t.Errorf("Expected function to panic, but it did not")
+		}
+	}()
+	Unregister()
+}

server/auth_callout.go

@@ -0,0 +1,503 @@
+// Copyright 2022-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"bytes"
+	"crypto/tls"
+	"encoding/pem"
+	"errors"
+	"fmt"
+	"time"
+	"unicode"
+
+	"github.com/nats-io/jwt/v2"
+	"github.com/nats-io/nkeys"
+)
+
+const (
+	AuthCalloutSubject    = "$SYS.REQ.USER.AUTH"
+	AuthRequestSubject    = "nats-authorization-request"
+	AuthRequestXKeyHeader = "Nats-Server-Xkey"
+)
+
+func titleCase(m string) string {
+	r := []rune(m)
+	if len(r) == 0 {
+		return _EMPTY_
+	}
+	return string(append([]rune{unicode.ToUpper(r[0])}, r[1:]...))
+}
+
+// Process a callout on this client's behalf.
+func (s *Server) processClientOrLeafCallout(c *client, opts *Options, proxyRequired, trustedProxy bool, ujwt string) (authorized bool, errStr string) {
+	isOperatorMode := len(opts.TrustedKeys) > 0
+
+	// this is the account the user connected in, or the one running the callout
+	var acc *Account
+	if !isOperatorMode && opts.AuthCallout != nil && opts.AuthCallout.Account != _EMPTY_ {
+		aname := opts.AuthCallout.Account
+		var err error
+		acc, err = s.LookupAccount(aname)
+		if err != nil {
+			errStr = fmt.Sprintf("No valid account %q for auth callout request: %v", aname, err)
+			s.Warnf(errStr)
+			return false, errStr
+		}
+	} else {
+		acc = c.acc
+	}
+	if acc == nil {
+		// FIX for https://github.com/nats-io/nats-server/issues/7841
+		// hand rolled creds on leafnode became crasher here
+		errStr = fmt.Sprintf("%s not mapped to a callout account", c.kindString())
+		s.Warnf(errStr)
+		return false, errStr
+	}
+
+	// Check if we have been requested to encrypt.
+	var xkp nkeys.KeyPair
+	var xkey string
+	var pubAccXKey string
+	if !isOperatorMode && opts.AuthCallout != nil && opts.AuthCallout.XKey != _EMPTY_ {
+		pubAccXKey = opts.AuthCallout.XKey
+	} else if isOperatorMode {
+		pubAccXKey = acc.externalAuthXKey()
+	}
+	// If set grab server's xkey keypair and public key.
+	if pubAccXKey != _EMPTY_ {
+		// These are only set on creation, so lock not needed.
+		xkp, xkey = s.xkp, s.info.XKey
+	}
+
+	// Create a keypair for the user. We will expect this public user to be in the signed response.
+	// This prevents replay attacks.
+	ukp, _ := nkeys.CreateUser()
+	pub, _ := ukp.PublicKey()
+
+	reply := s.newRespInbox()
+	respCh := make(chan string, 1)
+
+	decodeResponse := func(rc *client, rmsg []byte, acc *Account) (*jwt.UserClaims, error) {
+		account := acc.Name
+		_, msg := rc.msgParts(rmsg)
+
+		// This signals not authorized.
+		// Since this is an account subscription will always have "\r\n".
+		if len(msg) <= LEN_CR_LF {
+			return nil, fmt.Errorf("auth callout violation: %q on account %q", "no reason supplied", account)
+		}
+		// Strip trailing CRLF.
+		msg = msg[:len(msg)-LEN_CR_LF]
+		encrypted := false
+		// If we sent an encrypted request the response could be encrypted as well.
+		// we are expecting the input to be `eyJ` if it is a JWT
+		if xkp != nil && len(msg) > 0 && !bytes.HasPrefix(msg, []byte(jwtPrefix)) {
+			var err error
+			msg, err = xkp.Open(msg, pubAccXKey)
+			if err != nil {
+				return nil, fmt.Errorf("error decrypting auth callout response on account %q: %v", account, err)
+			}
+			encrypted = true
+		}
+
+		cr, err := jwt.DecodeAuthorizationResponseClaims(string(msg))
+		if err != nil {
+			return nil, err
+		}
+		vr := jwt.CreateValidationResults()
+		cr.Validate(vr)
+		if len(vr.Issues) > 0 {
+			return nil, fmt.Errorf("authorization response had validation errors: %v", vr.Issues[0])
+		}
+
+		// the subject is the user id
+		if cr.Subject != pub {
+			return nil, errors.New("auth callout violation: auth callout response is not for expected user")
+		}
+
+		// check the audience to be the server ID
+		if cr.Audience != s.info.ID {
+			return nil, errors.New("auth callout violation: auth callout response is not for server")
+		}
+
+		// check if had an error message from the auth account
+		if cr.Error != _EMPTY_ {
+			return nil, fmt.Errorf("auth callout service returned an error: %v", cr.Error)
+		}
+
+		// if response is encrypted none of this is needed
+		if isOperatorMode && !encrypted {
+			pkStr := cr.Issuer
+			if cr.IssuerAccount != _EMPTY_ {
+				pkStr = cr.IssuerAccount
+			}
+			if pkStr != account {
+				if _, ok := acc.hasIssuer(pkStr); !ok {
+					return nil, errors.New("auth callout signing key is unknown")
+				}
+			}
+		}
+
+		return jwt.DecodeUserClaims(cr.Jwt)
+	}
+
+	// getIssuerAccount returns the issuer (as per JWT) - it also asserts that
+	// only in operator mode we expect to receive `issuer_account`.
+	getIssuerAccount := func(arc *jwt.UserClaims, account string) (string, error) {
+		// Make sure correct issuer.
+		var issuer string
+		if opts.AuthCallout != nil {
+			issuer = opts.AuthCallout.Issuer
+		} else {
+			// Operator mode is who we send the request on unless switching accounts.
+			issuer = acc.Name
+		}
+
+		// the jwt issuer can be a signing key
+		jwtIssuer := arc.Issuer
+		if arc.IssuerAccount != _EMPTY_ {
+			if !isOperatorMode {
+				// this should be invalid - effectively it would allow the auth callout
+				// to issue on another account which may be allowed given the configuration
+				// where the auth callout account can handle multiple different ones..
+				return _EMPTY_, fmt.Errorf("error non operator mode account %q: attempted to use issuer_account", account)
+			}
+			jwtIssuer = arc.IssuerAccount
+		}
+
+		if jwtIssuer != issuer {
+			if !isOperatorMode {
+				return _EMPTY_, fmt.Errorf("wrong issuer for auth callout response on account %q, expected %q got %q", account, issuer, jwtIssuer)
+			} else if !acc.isAllowedAcount(jwtIssuer) {
+				return _EMPTY_, fmt.Errorf("account %q not permitted as valid account option for auth callout for account %q",
+					arc.Issuer, account)
+			}
+		}
+		return jwtIssuer, nil
+	}
+
+	getExpirationAndAllowedConnections := func(arc *jwt.UserClaims, account string) (time.Duration, map[string]struct{}, error) {
+		allowNow, expiration := validateTimes(arc)
+		if !allowNow {
+			c.Errorf("Outside connect times")
+			return 0, nil, fmt.Errorf("authorized user on account %q outside of valid connect times", account)
+		}
+
+		allowedConnTypes, err := convertAllowedConnectionTypes(arc.User.AllowedConnectionTypes)
+		if err != nil {
+			c.Debugf("%v", err)
+			if len(allowedConnTypes) == 0 {
+				return 0, nil, fmt.Errorf("authorized user on account %q using invalid connection type", account)
+			}
+		}
+		return expiration, allowedConnTypes, nil
+	}
+
+	assignAccountAndPermissions := func(arc *jwt.UserClaims, account string) (*Account, error) {
+		// Apply to this client.
+		var err error
+		issuerAccount, err := getIssuerAccount(arc, account)
+		if err != nil {
+			return nil, err
+		}
+
+		// if we are not in operator mode, they can specify placement as a tag
+		var placement string
+		if !isOperatorMode {
+			// only allow placement if we are not in operator mode
+			placement = arc.Audience
+		} else {
+			placement = issuerAccount
+		}
+
+		targetAcc, err := s.LookupAccount(placement)
+		if err != nil {
+			return nil, fmt.Errorf("no valid account %q for auth callout response on account %q: %v", placement, account, err)
+		}
+		if isOperatorMode {
+			// this will validate the signing key that emitted the user, and if it is a signing
+			// key it assigns the permissions from the target account
+			if scope, ok := targetAcc.hasIssuer(arc.Issuer); !ok {
+				return nil, fmt.Errorf("user JWT issuer %q is not known", arc.Issuer)
+			} else if scope != nil {
+				// this possibly has to be different because it could just be a plain issued by a non-scoped signing key
+				if err := scope.ValidateScopedSigner(arc); err != nil {
+					return nil, fmt.Errorf("user JWT is not valid: %v", err)
+				} else if uSc, ok := scope.(*jwt.UserScope); !ok {
+					return nil, fmt.Errorf("user JWT is not a valid scoped user")
+				} else if arc.User.UserPermissionLimits, err = processUserPermissionsTemplate(uSc.Template, arc, targetAcc); err != nil {
+					return nil, fmt.Errorf("user JWT generated invalid permissions: %v", err)
+				}
+			}
+		}
+		return targetAcc, nil
+	}
+
+	processReply := func(_ *subscription, rc *client, racc *Account, subject, reply string, rmsg []byte) {
+		arc, err := decodeResponse(rc, rmsg, racc)
+		if err != nil {
+			c.authViolation()
+			respCh <- titleCase(err.Error())
+			return
+		}
+		// If the caller had established that the user should go through a proxy,
+		// or if the `arc` JWT requires it, and we don't have a trusted proxy,
+		// reject the connection.
+		if (proxyRequired || arc.ProxyRequired) && !trustedProxy {
+			err = ErrAuthProxyRequired
+			c.setAuthError(err)
+			c.authViolation()
+			respCh <- titleCase(err.Error())
+			return
+		}
+		vr := jwt.CreateValidationResults()
+		arc.Validate(vr)
+		if len(vr.Issues) > 0 {
+			c.authViolation()
+			respCh <- fmt.Sprintf("Error validating user JWT: %v", vr.Issues[0])
+			return
+		}
+
+		// Make sure that the user is what we requested.
+		if arc.Subject != pub {
+			c.authViolation()
+			respCh <- fmt.Sprintf("Expected authorized user of %q but got %q on account %q", pub, arc.Subject, racc.Name)
+			return
+		}
+
+		expiration, allowedConnTypes, err := getExpirationAndAllowedConnections(arc, racc.Name)
+		if err != nil {
+			c.authViolation()
+			respCh <- titleCase(err.Error())
+			return
+		}
+
+		targetAcc, err := assignAccountAndPermissions(arc, racc.Name)
+		if err != nil {
+			c.authViolation()
+			respCh <- titleCase(err.Error())
+			return
+		}
+
+		// the JWT is cleared, because if in operator mode it may hold the JWT
+		// for the bearer token that connected to the callout if in operator mode
+		// the permissions are already set on the client, this prevents a decode
+		// on c.RegisterNKeyUser which would have wrong values
+		c.mu.Lock()
+		c.opts.JWT = _EMPTY_
+		c.mu.Unlock()
+
+		// Build internal user and bind to the targeted account.
+		nkuser := buildInternalNkeyUser(arc, allowedConnTypes, targetAcc)
+		if err := c.RegisterNkeyUser(nkuser); err != nil {
+			c.authViolation()
+			respCh <- fmt.Sprintf("Could not register auth callout user: %v", err)
+			return
+		}
+
+		// See if the response wants to override the username.
+		if arc.Name != _EMPTY_ {
+			c.mu.Lock()
+			c.opts.Username = arc.Name
+			// Clear any others.
+			c.opts.Nkey = _EMPTY_
+			c.pubKey = _EMPTY_
+			c.opts.Token = _EMPTY_
+			c.mu.Unlock()
+		}
+
+		// Check if we need to set an auth timer if the user jwt expires.
+		c.setExpiration(arc.Claims(), expiration)
+
+		respCh <- _EMPTY_
+	}
+
+	// create a subscription to receive a response from the authcallout
+	sub, err := acc.subscribeInternal(reply, processReply)
+	if err != nil {
+		errStr = fmt.Sprintf("Error setting up reply subscription for auth request: %v", err)
+		s.Warnf(errStr)
+		return false, errStr
+	}
+	defer acc.unsubscribeInternal(sub)
+
+	// Build our request claims - jwt subject should be nkey
+	jwtSub := acc.Name
+	if opts.AuthCallout != nil {
+		jwtSub = opts.AuthCallout.Issuer
+	}
+
+	// The public key of the server, if set is available on Varz.Key
+	// This means that when a service connects, it can now peer
+	// authenticate if it wants to - but that also means that it needs to be
+	// listening to cluster changes
+	claim := jwt.NewAuthorizationRequestClaims(jwtSub)
+	claim.Audience = AuthRequestSubject
+	// Set expected public user nkey.
+	claim.UserNkey = pub
+
+	s.mu.RLock()
+	claim.Server = jwt.ServerID{
+		Name:    s.info.Name,
+		Host:    s.info.Host,
+		ID:      s.info.ID,
+		Version: s.info.Version,
+		Cluster: s.info.Cluster,
+	}
+	s.mu.RUnlock()
+
+	// Tags
+	claim.Server.Tags = s.getOpts().Tags
+
+	// Check if we have been requested to encrypt.
+	// FIXME: possibly this public key also needs to be on the
+	//  Varz, because then it can be peer verified?
+	if xkp != nil {
+		claim.Server.XKey = xkey
+	}
+
+	authTimeout := secondsToDuration(s.getOpts().AuthTimeout)
+	claim.Expires = time.Now().Add(time.Duration(authTimeout)).UTC().Unix()
+
+	// Grab client info for the request.
+	c.mu.Lock()
+	c.fillClientInfo(&claim.ClientInformation)
+	c.fillConnectOpts(&claim.ConnectOptions, ujwt)
+	// If we have a sig in the client opts, fill in nonce.
+	if claim.ConnectOptions.SignedNonce != _EMPTY_ {
+		claim.ClientInformation.Nonce = string(c.nonce)
+	}
+
+	// TLS
+	if c.flags.isSet(handshakeComplete) && c.nc != nil {
+		var ct jwt.ClientTLS
+		conn := c.nc.(*tls.Conn)
+		cs := conn.ConnectionState()
+		ct.Version = tlsVersion(cs.Version)
+		ct.Cipher = tls.CipherSuiteName(cs.CipherSuite)
+		// Check verified chains.
+		for _, vs := range cs.VerifiedChains {
+			var certs []string
+			for _, c := range vs {
+				blk := &pem.Block{
+					Type:  "CERTIFICATE",
+					Bytes: c.Raw,
+				}
+				certs = append(certs, string(pem.EncodeToMemory(blk)))
+			}
+			ct.VerifiedChains = append(ct.VerifiedChains, certs)
+		}
+		// If we do not have verified chains put in peer certs.
+		if len(ct.VerifiedChains) == 0 {
+			for _, c := range cs.PeerCertificates {
+				blk := &pem.Block{
+					Type:  "CERTIFICATE",
+					Bytes: c.Raw,
+				}
+				ct.Certs = append(ct.Certs, string(pem.EncodeToMemory(blk)))
+			}
+		}
+		claim.TLS = &ct
+	}
+	c.mu.Unlock()
+
+	b, err := claim.Encode(s.kp)
+	if err != nil {
+		errStr = fmt.Sprintf("Error encoding auth request claim on account %q: %v", acc.Name, err)
+		s.Warnf(errStr)
+		return false, errStr
+	}
+	req := []byte(b)
+	var hdr []byte
+
+	// Check if we have been asked to encrypt.
+	if xkp != nil {
+		req, err = xkp.Seal([]byte(req), pubAccXKey)
+		if err != nil {
+			errStr = fmt.Sprintf("Error encrypting auth request claim on account %q: %v", acc.Name, err)
+			s.Warnf(errStr)
+			return false, errStr
+		}
+		hdr = genHeader(hdr, AuthRequestXKeyHeader, xkey)
+	}
+
+	// Send out our request.
+	if err := s.sendInternalAccountMsgWithReply(acc, AuthCalloutSubject, reply, hdr, req, false); err != nil {
+		errStr = fmt.Sprintf("Error sending authorization request: %v", err)
+		s.Debugf(errStr)
+		return false, errStr
+	}
+	select {
+	case errStr = <-respCh:
+		if authorized = errStr == _EMPTY_; !authorized {
+			s.Warnf(errStr)
+		}
+	case <-time.After(authTimeout):
+		s.Debugf(fmt.Sprintf("Authorization callout response not received in time on account %q", acc.Name))
+	}
+
+	return authorized, errStr
+}
+
+// Fill in client information for the request.
+// Lock should be held.
+func (c *client) fillClientInfo(ci *jwt.ClientInformation) {
+	if c == nil || (c.kind != CLIENT && c.kind != LEAF && c.kind != JETSTREAM && c.kind != ACCOUNT) {
+		return
+	}
+
+	// Do it this way to fail to compile if fields are added to jwt.ClientInformation.
+	*ci = jwt.ClientInformation{
+		Host:    c.host,
+		ID:      c.cid,
+		User:    c.getRawAuthUser(),
+		Name:    c.opts.Name,
+		Tags:    c.tags,
+		NameTag: c.nameTag,
+		Kind:    c.kindString(),
+		Type:    c.clientTypeString(),
+		MQTT:    c.getMQTTClientID(),
+	}
+}
+
+// Fill in client options.
+// Lock should be held.
+func (c *client) fillConnectOpts(opts *jwt.ConnectOptions, ujwt string) {
+	if c == nil || (c.kind != CLIENT && c.kind != LEAF && c.kind != JETSTREAM && c.kind != ACCOUNT) {
+		return
+	}
+
+	o := c.opts
+	if ujwt == _EMPTY_ {
+		// The caller may supply a reconstructed JWT that should be sent to auth
+		// callout without storing it in c.opts.JWT. If not, fall back to the client
+		// option as before.
+		ujwt = o.JWT
+	}
+
+	// Do it this way to fail to compile if fields are added to jwt.ClientInformation.
+	*opts = jwt.ConnectOptions{
+		JWT:         ujwt,
+		Nkey:        o.Nkey,
+		SignedNonce: o.Sig,
+		Token:       o.Token,
+		Username:    o.Username,
+		Password:    o.Password,
+		Name:        o.Name,
+		Lang:        o.Lang,
+		Version:     o.Version,
+		Protocol:    o.Protocol,
+	}
+}

server/avl/norace_test.go

@@ -0,0 +1,198 @@
+// Copyright 2023-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build !race && !skip_no_race_tests && !skip_no_race_1_tests
+
+package avl
+
+import (
+	"flag"
+	"fmt"
+	"math"
+	"math/rand/v2"
+	"runtime"
+	"runtime/debug"
+	"testing"
+	"time"
+)
+
+// Print Results: go test -v  --args --results
+var printResults = flag.Bool("results", false, "Enable Results Logging")
+
+// SequenceSet memory tests vs dmaps.
+func TestNoRaceSeqSetSizeComparison(t *testing.T) {
+	// Create 5M random entries (dupes possible but ok for this test) out of 8M range.
+	num := 5_000_000
+	max := 7_000_000
+
+	seqs := make([]uint64, 0, num)
+	for i := 0; i < num; i++ {
+		n := uint64(rand.Int64N(int64(max + 1)))
+		seqs = append(seqs, n)
+	}
+
+	runtime.GC()
+	// Disable to get stable results.
+	gcp := debug.SetGCPercent(-1)
+	defer debug.SetGCPercent(gcp)
+
+	mem := runtime.MemStats{}
+	runtime.ReadMemStats(&mem)
+	inUseBefore := mem.HeapInuse
+
+	dmap := make(map[uint64]struct{}, num)
+	for _, n := range seqs {
+		dmap[n] = struct{}{}
+	}
+	runtime.ReadMemStats(&mem)
+	dmapUse := mem.HeapInuse - inUseBefore
+	inUseBefore = mem.HeapInuse
+
+	// Now do SequenceSet on same dataset.
+	var sset SequenceSet
+	for _, n := range seqs {
+		sset.Insert(n)
+	}
+
+	runtime.ReadMemStats(&mem)
+	seqSetUse := mem.HeapInuse - inUseBefore
+
+	if seqSetUse > 2*1024*1024 {
+		t.Fatalf("Expected SequenceSet size to be < 2M, got %v", friendlyBytes(seqSetUse))
+	}
+	if seqSetUse*50 > dmapUse {
+		t.Fatalf("Expected SequenceSet to be at least 50x better then dmap approach: %v vs %v",
+			friendlyBytes(seqSetUse),
+			friendlyBytes(dmapUse),
+		)
+	}
+}
+
+func TestNoRaceSeqSetEncodeLarge(t *testing.T) {
+	num := 2_500_000
+	max := 5_000_000
+
+	dmap := make(map[uint64]struct{}, num)
+	var ss SequenceSet
+	for i := 0; i < num; i++ {
+		n := uint64(rand.Int64N(int64(max + 1)))
+		ss.Insert(n)
+		dmap[n] = struct{}{}
+	}
+
+	// Disable to get stable results.
+	gcp := debug.SetGCPercent(-1)
+	defer debug.SetGCPercent(gcp)
+
+	// In general should be about the same, but can see some variability.
+	expected := time.Millisecond
+
+	start := time.Now()
+	b := ss.Encode(nil)
+
+	if elapsed := time.Since(start); elapsed > expected {
+		t.Fatalf("Expected encode of %d items with encoded size %v to take less than %v, got %v",
+			num, friendlyBytes(len(b)), expected, elapsed)
+	} else {
+		logResults("Encode time for %d items was %v, encoded size is %v\n", num, elapsed, friendlyBytes(len(b)))
+	}
+
+	start = time.Now()
+	ss2, _, err := Decode(b)
+	require_NoError(t, err)
+	if elapsed := time.Since(start); elapsed > expected {
+		t.Fatalf("Expected decode to take less than %v, got %v", expected, elapsed)
+	} else {
+		logResults("Decode time is %v\n", elapsed)
+	}
+	require_True(t, ss.Nodes() == ss2.Nodes())
+	require_True(t, ss.Size() == ss2.Size())
+}
+
+func TestNoRaceSeqSetRelativeSpeed(t *testing.T) {
+	// Create 1M random entries (dupes possible but ok for this test) out of 3M range.
+	num := 1_000_000
+	max := 3_000_000
+
+	seqs := make([]uint64, 0, num)
+	for i := 0; i < num; i++ {
+		n := uint64(rand.Int64N(int64(max + 1)))
+		seqs = append(seqs, n)
+	}
+
+	start := time.Now()
+	// Now do SequenceSet on same dataset.
+	var sset SequenceSet
+	for _, n := range seqs {
+		sset.Insert(n)
+	}
+	ssInsertElapsed := time.Since(start)
+	logResults("Inserts SequenceSet: %v for %d items\n", ssInsertElapsed, num)
+
+	start = time.Now()
+	for _, n := range seqs {
+		if ok := sset.Exists(n); !ok {
+			t.Fatalf("Should exist")
+		}
+	}
+	ssLookupElapsed := time.Since(start)
+	logResults("Lookups: %v\n", ssLookupElapsed)
+
+	// Now do a map.
+	dmap := make(map[uint64]struct{})
+	start = time.Now()
+	for _, n := range seqs {
+		dmap[n] = struct{}{}
+	}
+	mapInsertElapsed := time.Since(start)
+	logResults("Inserts Map[uint64]: %v for %d items\n", mapInsertElapsed, num)
+
+	start = time.Now()
+	for _, n := range seqs {
+		if _, ok := dmap[n]; !ok {
+			t.Fatalf("Should exist")
+		}
+	}
+	mapLookupElapsed := time.Since(start)
+	logResults("Lookups: %v\n", mapLookupElapsed)
+
+	// In general we are between 1.5 and 1.75 times slower atm then a straight map.
+	// Let's test an upper bound of 2x for now.
+	if mapInsertElapsed*2 <= ssInsertElapsed {
+		t.Fatalf("Expected SequenceSet insert to be no more than 2x slower (%v vs %v)", mapInsertElapsed, ssInsertElapsed)
+	}
+
+	if mapLookupElapsed*3 <= ssLookupElapsed {
+		t.Fatalf("Expected SequenceSet lookups to be no more than 3x slower (%v vs %v)", mapLookupElapsed, ssLookupElapsed)
+	}
+}
+
+// friendlyBytes returns a string with the given bytes int64
+// represented as a size, such as 1KB, 10MB, etc...
+func friendlyBytes[T int | uint64 | int64](bytes T) string {
+	fbytes := float64(bytes)
+	base := 1024
+	pre := []string{"K", "M", "G", "T", "P", "E"}
+	if fbytes < float64(base) {
+		return fmt.Sprintf("%v B", fbytes)
+	}
+	exp := int(math.Log(fbytes) / math.Log(float64(base)))
+	index := exp - 1
+	return fmt.Sprintf("%.2f %sB", fbytes/math.Pow(float64(base), float64(exp)), pre[index])
+}
+
+func logResults(format string, args ...any) {
+	if *printResults {
+		fmt.Printf(format, args...)
+	}
+}

server/avl/seqset.go

@@ -0,0 +1,715 @@
+// Copyright 2023-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avl
+
+import (
+	"cmp"
+	"encoding/binary"
+	"errors"
+	"math/bits"
+	"slices"
+)
+
+// SequenceSet is a memory and encoding optimized set for storing unsigned ints.
+//
+// SequenceSet is ~80-100 times more efficient memory wise than a map[uint64]struct{}.
+// SequenceSet is ~1.75 times slower at inserts than the same map.
+// SequenceSet is not thread safe.
+//
+// We use an AVL tree with nodes that hold bitmasks for set membership.
+//
+// Encoding will convert to a space optimized encoding using bitmasks.
+type SequenceSet struct {
+	root  *node // root node
+	size  int   // number of items
+	nodes int   // number of nodes
+	// Having this here vs on the stack in Insert/Delete
+	// makes a difference in memory usage.
+	changed bool
+}
+
+// Insert will insert the sequence into the set.
+// The tree will be balanced inline.
+func (ss *SequenceSet) Insert(seq uint64) {
+	// If a node covering seq already exists, setting a bit can not change the
+	// tree shape, so skip the recursive descent and rebalance checks.
+	for n := ss.root; n != nil; {
+		if seq < n.base {
+			n = n.l
+		} else if seq >= n.base+numEntries {
+			n = n.r
+		} else {
+			n.set(seq, &ss.changed)
+			if ss.changed {
+				ss.changed = false
+				ss.size++
+			}
+			return
+		}
+	}
+	if ss.root = ss.root.insert(seq, &ss.changed, &ss.nodes); ss.changed {
+		ss.changed = false
+		ss.size++
+	}
+}
+
+// Exists will return true iff the sequence is a member of this set.
+func (ss *SequenceSet) Exists(seq uint64) bool {
+	for n := ss.root; n != nil; {
+		if seq < n.base {
+			n = n.l
+			continue
+		} else if seq >= n.base+numEntries {
+			n = n.r
+			continue
+		}
+		return n.exists(seq)
+	}
+	return false
+}
+
+// SetInitialMin should be used to set the initial minimum sequence when known.
+// This will more effectively utilize space versus self selecting.
+// The set should be empty.
+func (ss *SequenceSet) SetInitialMin(min uint64) error {
+	if !ss.IsEmpty() {
+		return ErrSetNotEmpty
+	}
+	ss.root, ss.nodes = &node{base: min, h: 1}, 1
+	return nil
+}
+
+// Delete will remove the sequence from the set.
+// Will optionally remove nodes and rebalance.
+// Returns where the sequence was set.
+func (ss *SequenceSet) Delete(seq uint64) bool {
+	if ss == nil || ss.root == nil {
+		return false
+	}
+	ss.root = ss.root.delete(seq, &ss.changed, &ss.nodes)
+	if ss.changed {
+		ss.changed = false
+		ss.size--
+		if ss.size == 0 {
+			ss.Empty()
+		}
+		return true
+	}
+	return false
+}
+
+// Size returns the number of items in the set.
+func (ss *SequenceSet) Size() int {
+	return ss.size
+}
+
+// Nodes returns the number of nodes in the tree.
+func (ss *SequenceSet) Nodes() int {
+	return ss.nodes
+}
+
+// Empty will clear all items from a set.
+func (ss *SequenceSet) Empty() {
+	ss.root = nil
+	ss.size = 0
+	ss.nodes = 0
+}
+
+// IsEmpty is a fast check of the set being empty.
+func (ss *SequenceSet) IsEmpty() bool {
+	if ss == nil || ss.root == nil {
+		return true
+	}
+	return false
+}
+
+// Range will invoke the given function for each item in the set.
+// They will range over the set in ascending order.
+// If the callback returns false we terminate the iteration.
+func (ss *SequenceSet) Range(f func(uint64) bool) {
+	ss.root.iter(f)
+}
+
+// Heights returns the left and right heights of the tree.
+func (ss *SequenceSet) Heights() (l, r int) {
+	if ss.root == nil {
+		return 0, 0
+	}
+	if ss.root.l != nil {
+		l = ss.root.l.h
+	}
+	if ss.root.r != nil {
+		r = ss.root.r.h
+	}
+	return l, r
+}
+
+// Returns min, max and number of set items.
+func (ss *SequenceSet) State() (min, max, num uint64) {
+	if ss == nil || ss.root == nil {
+		return 0, 0, 0
+	}
+	min, max = ss.MinMax()
+	return min, max, uint64(ss.Size())
+}
+
+// MinMax will return the minunum and maximum values in the set.
+func (ss *SequenceSet) MinMax() (min, max uint64) {
+	if ss.root == nil {
+		return 0, 0
+	}
+	for l := ss.root; l != nil; l = l.l {
+		if l.l == nil {
+			min = l.min()
+		}
+	}
+	for r := ss.root; r != nil; r = r.r {
+		if r.r == nil {
+			max = r.max()
+		}
+	}
+	return min, max
+}
+
+func clone(src *node, target **node) {
+	if src == nil {
+		return
+	}
+	n := &node{base: src.base, bits: src.bits, h: src.h}
+	*target = n
+	clone(src.l, &n.l)
+	clone(src.r, &n.r)
+}
+
+// Clone will return a clone of the given SequenceSet.
+func (ss *SequenceSet) Clone() *SequenceSet {
+	if ss == nil {
+		return nil
+	}
+	css := &SequenceSet{nodes: ss.nodes, size: ss.size}
+	clone(ss.root, &css.root)
+
+	return css
+}
+
+// Equal returns whether the two sets contain exactly the same sequences.
+func (ss *SequenceSet) Equal(other *SequenceSet) bool {
+	if ss.IsEmpty() || other.IsEmpty() {
+		return ss.IsEmpty() && other.IsEmpty()
+	}
+	if ss.size != other.size {
+		return false
+	}
+	// Sizes are equal, so a one-way membership check suffices.
+	equal := true
+	ss.Range(func(seq uint64) bool {
+		equal = other.Exists(seq)
+		return equal
+	})
+	return equal
+}
+
+// Union will union this SequenceSet with ssa.
+func (ss *SequenceSet) Union(ssa ...*SequenceSet) {
+	for _, sa := range ssa {
+		sa.root.nodeIter(func(n *node) {
+			for nb, b := range n.bits {
+				base := n.base + uint64(nb)*bitsPerBucket
+				for b != 0 {
+					ss.Insert(base + uint64(bits.TrailingZeros64(b)))
+					b &= b - 1
+				}
+			}
+		})
+	}
+}
+
+// Union will return a union of all sets.
+func Union(ssa ...*SequenceSet) *SequenceSet {
+	if len(ssa) == 0 {
+		return nil
+	}
+	// Sort so we can clone largest.
+	slices.SortFunc(ssa, func(i, j *SequenceSet) int { return -cmp.Compare(i.Size(), j.Size()) }) // reverse order
+	ss := ssa[0].Clone()
+
+	// Insert the rest through range call.
+	for i := 1; i < len(ssa); i++ {
+		ssa[i].Range(func(n uint64) bool {
+			ss.Insert(n)
+			return true
+		})
+	}
+	return ss
+}
+
+const (
+	// Magic is used to identify the encode binary state..
+	magic = uint8(22)
+	// Version
+	version = uint8(2)
+	// hdrLen
+	hdrLen = 2
+	// minimum length of an encoded SequenceSet.
+	minLen = 2 + 8 // magic + version + num nodes + num entries.
+)
+
+// EncodeLen returns the bytes needed for encoding.
+func (ss SequenceSet) EncodeLen() int {
+	return minLen + (ss.Nodes() * ((numBuckets+1)*8 + 2))
+}
+
+func (ss SequenceSet) Encode(buf []byte) []byte {
+	nn, encLen := ss.Nodes(), ss.EncodeLen()
+
+	if cap(buf) < encLen {
+		buf = make([]byte, encLen)
+	} else {
+		buf = buf[:encLen]
+	}
+
+	// TODO(dlc) - Go 1.19 introduced Append to not have to keep track.
+	// Once 1.20 is out we could change this over.
+	// Also binary.Write() is way slower, do not use.
+
+	var le = binary.LittleEndian
+	buf[0], buf[1] = magic, version
+	i := hdrLen
+	le.PutUint32(buf[i:], uint32(nn))
+	le.PutUint32(buf[i+4:], uint32(ss.size))
+	i += 8
+	ss.root.nodeIter(func(n *node) {
+		le.PutUint64(buf[i:], n.base)
+		i += 8
+		for _, b := range n.bits {
+			le.PutUint64(buf[i:], b)
+			i += 8
+		}
+		le.PutUint16(buf[i:], uint16(n.h))
+		i += 2
+	})
+	return buf[:i]
+}
+
+// ErrBadEncoding is returned when we can not decode properly.
+var (
+	ErrBadEncoding = errors.New("ss: bad encoding")
+	ErrBadVersion  = errors.New("ss: bad version")
+	ErrSetNotEmpty = errors.New("ss: set not empty")
+)
+
+// Decode returns the sequence set and number of bytes read from the buffer on success.
+func Decode(buf []byte) (*SequenceSet, int, error) {
+	if len(buf) < minLen || buf[0] != magic {
+		return nil, -1, ErrBadEncoding
+	}
+
+	switch v := buf[1]; v {
+	case 1:
+		return decodev1(buf)
+	case 2:
+		return decodev2(buf)
+	default:
+		return nil, -1, ErrBadVersion
+	}
+}
+
+// Helper to decode v2.
+func decodev2(buf []byte) (*SequenceSet, int, error) {
+	var le = binary.LittleEndian
+	index := 2
+	nn := int(le.Uint32(buf[index:]))
+	sz := int(le.Uint32(buf[index+4:]))
+	index += 8
+
+	// nn is decoded as a uint32 but held in an int. On 32-bit builds a value
+	// above MaxInt32 turns negative and nn*perNode below overflows, so the
+	// length check would pass for a short buffer and the following make/reads
+	// run off the end. Compare with division so the bound holds on every arch.
+	if nn < 0 || nn > (len(buf)-minLen)/((numBuckets+1)*8+2) {
+		return nil, -1, ErrBadEncoding
+	}
+
+	ss, nodes := SequenceSet{size: sz}, make([]node, nn)
+
+	for i := 0; i < nn; i++ {
+		n := &nodes[i]
+		n.base = le.Uint64(buf[index:])
+		index += 8
+		for bi := range n.bits {
+			n.bits[bi] = le.Uint64(buf[index:])
+			index += 8
+		}
+		n.h = int(le.Uint16(buf[index:]))
+		index += 2
+		ss.insertNode(n)
+	}
+
+	return &ss, index, nil
+}
+
+// Helper to decode v1 into v2 which has fixed buckets of 32 vs 64 originally.
+func decodev1(buf []byte) (*SequenceSet, int, error) {
+	var le = binary.LittleEndian
+	index := 2
+	nn := int(le.Uint32(buf[index:]))
+	sz := int(le.Uint32(buf[index+4:]))
+	index += 8
+
+	const v1NumBuckets = 64
+
+	// See decodev2: guard the node count without overflowing the multiply so
+	// the bound stays correct on 32-bit builds too.
+	if nn < 0 || nn > (len(buf)-minLen)/((v1NumBuckets+1)*8+2) {
+		return nil, -1, ErrBadEncoding
+	}
+
+	var ss SequenceSet
+	for i := 0; i < nn; i++ {
+		base := le.Uint64(buf[index:])
+		index += 8
+		for nb := uint64(0); nb < v1NumBuckets; nb++ {
+			n := le.Uint64(buf[index:])
+			// Walk all set bits and insert sequences manually for this decode from v1.
+			for n != 0 {
+				ss.Insert(base + (nb * uint64(bitsPerBucket)) + uint64(bits.TrailingZeros64(n)))
+				n &= n - 1
+			}
+			index += 8
+		}
+		// Skip over encoded height.
+		index += 2
+	}
+
+	// Sanity check.
+	if ss.Size() != sz {
+		return nil, -1, ErrBadEncoding
+	}
+
+	return &ss, index, nil
+
+}
+
+// insertNode places a decoded node into the tree.
+// These should be done in tree order as defined by Encode()
+// This allows us to not have to calculate height or do rebalancing.
+// So much better performance this way.
+func (ss *SequenceSet) insertNode(n *node) {
+	ss.nodes++
+
+	if ss.root == nil {
+		ss.root = n
+		return
+	}
+	// Walk our way to the insertion point.
+	for p := ss.root; p != nil; {
+		if n.base < p.base {
+			if p.l == nil {
+				p.l = n
+				return
+			}
+			p = p.l
+		} else {
+			if p.r == nil {
+				p.r = n
+				return
+			}
+			p = p.r
+		}
+	}
+}
+
+const (
+	bitsPerBucket = 64 // bits in uint64
+	numBuckets    = 32
+	numEntries    = numBuckets * bitsPerBucket
+)
+
+type node struct {
+	//v dvalue
+	base uint64
+	bits [numBuckets]uint64
+	l    *node
+	r    *node
+	h    int
+}
+
+// Set the proper bit.
+// seq should have already been qualified and inserted should be non nil.
+func (n *node) set(seq uint64, inserted *bool) {
+	seq -= n.base
+	i := seq / bitsPerBucket
+	mask := uint64(1) << (seq % bitsPerBucket)
+	if (n.bits[i] & mask) == 0 {
+		n.bits[i] |= mask
+		*inserted = true
+	}
+}
+
+func (n *node) insert(seq uint64, inserted *bool, nodes *int) *node {
+	if n == nil {
+		base := (seq / numEntries) * numEntries
+		n := &node{base: base, h: 1}
+		n.set(seq, inserted)
+		*nodes++
+		return n
+	}
+
+	if seq < n.base {
+		n.l = n.l.insert(seq, inserted, nodes)
+	} else if seq >= n.base+numEntries {
+		n.r = n.r.insert(seq, inserted, nodes)
+	} else {
+		n.set(seq, inserted)
+	}
+
+	n.h = maxH(n) + 1
+
+	// Don't make a function, impacts performance.
+	if bf := balanceF(n); bf > 1 {
+		// Left unbalanced.
+		if balanceF(n.l) < 0 {
+			n.l = n.l.rotateL()
+		}
+		return n.rotateR()
+	} else if bf < -1 {
+		// Right unbalanced.
+		if balanceF(n.r) > 0 {
+			n.r = n.r.rotateR()
+		}
+		return n.rotateL()
+	}
+	return n
+}
+
+func (n *node) rotateL() *node {
+	r := n.r
+	if r != nil {
+		n.r = r.l
+		r.l = n
+		n.h = maxH(n) + 1
+		r.h = maxH(r) + 1
+	} else {
+		n.r = nil
+		n.h = maxH(n) + 1
+	}
+	return r
+}
+
+func (n *node) rotateR() *node {
+	l := n.l
+	if l != nil {
+		n.l = l.r
+		l.r = n
+		n.h = maxH(n) + 1
+		l.h = maxH(l) + 1
+	} else {
+		n.l = nil
+		n.h = maxH(n) + 1
+	}
+	return l
+}
+
+func balanceF(n *node) int {
+	if n == nil {
+		return 0
+	}
+	var lh, rh int
+	if n.l != nil {
+		lh = n.l.h
+	}
+	if n.r != nil {
+		rh = n.r.h
+	}
+	return lh - rh
+}
+
+func maxH(n *node) int {
+	if n == nil {
+		return 0
+	}
+	var lh, rh int
+	if n.l != nil {
+		lh = n.l.h
+	}
+	if n.r != nil {
+		rh = n.r.h
+	}
+	if lh > rh {
+		return lh
+	}
+	return rh
+}
+
+// Clear the proper bit.
+// seq should have already been qualified and deleted should be non nil.
+// Will return true if this node is now empty.
+func (n *node) clear(seq uint64, deleted *bool) bool {
+	seq -= n.base
+	i := seq / bitsPerBucket
+	mask := uint64(1) << (seq % bitsPerBucket)
+	if (n.bits[i] & mask) == 0 {
+		// Nothing cleared, and nodes in the tree are never empty,
+		// so no need to scan the buckets.
+		return false
+	}
+	n.bits[i] &^= mask
+	*deleted = true
+	for _, b := range n.bits {
+		if b != 0 {
+			return false
+		}
+	}
+	return true
+}
+
+func (n *node) delete(seq uint64, deleted *bool, nodes *int) *node {
+	if n == nil {
+		return nil
+	}
+
+	if seq < n.base {
+		n.l = n.l.delete(seq, deleted, nodes)
+	} else if seq >= n.base+numEntries {
+		n.r = n.r.delete(seq, deleted, nodes)
+	} else if empty := n.clear(seq, deleted); empty {
+		*nodes--
+		if n.l == nil {
+			n = n.r
+		} else if n.r == nil {
+			n = n.l
+		} else {
+			// We have both children.
+			n.r = n.r.insertNodePrev(n.l)
+			n = n.r
+		}
+	}
+
+	if n != nil {
+		n.h = maxH(n) + 1
+	}
+
+	// Check balance.
+	if bf := balanceF(n); bf > 1 {
+		// Left unbalanced.
+		if balanceF(n.l) < 0 {
+			n.l = n.l.rotateL()
+		}
+		return n.rotateR()
+	} else if bf < -1 {
+		// right unbalanced.
+		if balanceF(n.r) > 0 {
+			n.r = n.r.rotateR()
+		}
+		return n.rotateL()
+	}
+
+	return n
+}
+
+// Will insert nn into the node assuming it is less than all other nodes in n.
+// Will re-calculate height and balance.
+func (n *node) insertNodePrev(nn *node) *node {
+	if n.l == nil {
+		n.l = nn
+	} else {
+		n.l = n.l.insertNodePrev(nn)
+	}
+	n.h = maxH(n) + 1
+
+	// Check balance.
+	if bf := balanceF(n); bf > 1 {
+		// Left unbalanced.
+		if balanceF(n.l) < 0 {
+			n.l = n.l.rotateL()
+		}
+		return n.rotateR()
+	} else if bf < -1 {
+		// right unbalanced.
+		if balanceF(n.r) > 0 {
+			n.r = n.r.rotateR()
+		}
+		return n.rotateL()
+	}
+	return n
+}
+
+func (n *node) exists(seq uint64) bool {
+	seq -= n.base
+	i := seq / bitsPerBucket
+	mask := uint64(1) << (seq % bitsPerBucket)
+	return n.bits[i]&mask != 0
+}
+
+// Return minimum sequence in the set.
+// This node can not be empty.
+func (n *node) min() uint64 {
+	for i, b := range n.bits {
+		if b != 0 {
+			return n.base +
+				uint64(i*bitsPerBucket) +
+				uint64(bits.TrailingZeros64(b))
+		}
+	}
+	return 0
+}
+
+// Return maximum sequence in the set.
+// This node can not be empty.
+func (n *node) max() uint64 {
+	for i := numBuckets - 1; i >= 0; i-- {
+		if b := n.bits[i]; b != 0 {
+			return n.base +
+				uint64(i*bitsPerBucket) +
+				uint64(bitsPerBucket-bits.LeadingZeros64(b>>1))
+		}
+	}
+	return 0
+}
+
+// This is done in tree order.
+func (n *node) nodeIter(f func(n *node)) {
+	if n == nil {
+		return
+	}
+	f(n)
+	n.l.nodeIter(f)
+	n.r.nodeIter(f)
+}
+
+// iter will iterate through the set's items in this node.
+// If the supplied function returns false we terminate the iteration.
+func (n *node) iter(f func(uint64) bool) bool {
+	if n == nil {
+		return true
+	}
+
+	if ok := n.l.iter(f); !ok {
+		return false
+	}
+	for i, b := range n.bits {
+		base := n.base + uint64(i)*bitsPerBucket
+		for b != 0 {
+			if ok := f(base + uint64(bits.TrailingZeros64(b))); !ok {
+				return false
+			}
+			b &= b - 1
+		}
+	}
+	if ok := n.r.iter(f); !ok {
+		return false
+	}
+
+	return true
+}

server/avl/seqset_test.go

@@ -0,0 +1,556 @@
+// Copyright 2023-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avl
+
+import (
+	"encoding/base64"
+	"encoding/binary"
+	"math/rand/v2"
+	"testing"
+)
+
+func TestSeqSetBasics(t *testing.T) {
+	var ss SequenceSet
+
+	seqs := []uint64{22, 222, 2000, 2, 2, 4}
+	for _, seq := range seqs {
+		ss.Insert(seq)
+		require_True(t, ss.Exists(seq))
+	}
+
+	require_True(t, ss.Nodes() == 1)
+	require_True(t, ss.Size() == len(seqs)-1) // We have one dup in there.
+	lh, rh := ss.Heights()
+	require_True(t, lh == 0)
+	require_True(t, rh == 0)
+}
+
+func TestSeqSetLeftLean(t *testing.T) {
+	var ss SequenceSet
+
+	for i := uint64(4 * numEntries); i > 0; i-- {
+		ss.Insert(i)
+	}
+
+	require_True(t, ss.Nodes() == 5)
+	require_True(t, ss.Size() == 4*numEntries)
+	lh, rh := ss.Heights()
+	require_True(t, lh == 2)
+	require_True(t, rh == 1)
+}
+
+func TestSeqSetRightLean(t *testing.T) {
+	var ss SequenceSet
+
+	for i := uint64(0); i < uint64(4*numEntries); i++ {
+		ss.Insert(i)
+	}
+
+	require_True(t, ss.Nodes() == 4)
+	require_True(t, ss.Size() == 4*numEntries)
+	lh, rh := ss.Heights()
+	require_True(t, lh == 1)
+	require_True(t, rh == 2)
+}
+
+func TestSeqSetCorrectness(t *testing.T) {
+	// Generate 100k sequences across 500k range.
+	num := 100_000
+	max := 500_000
+
+	set := make(map[uint64]struct{}, num)
+	var ss SequenceSet
+	for i := 0; i < num; i++ {
+		n := uint64(rand.Int64N(int64(max + 1)))
+		ss.Insert(n)
+		set[n] = struct{}{}
+	}
+
+	for i := uint64(0); i <= uint64(max); i++ {
+		_, exists := set[i]
+		require_True(t, ss.Exists(i) == exists)
+	}
+}
+
+func TestSeqSetRange(t *testing.T) {
+	num := 2*numEntries + 22
+	nums := make([]uint64, 0, num)
+	for i := 0; i < num; i++ {
+		nums = append(nums, uint64(i))
+	}
+	rand.Shuffle(len(nums), func(i, j int) { nums[i], nums[j] = nums[j], nums[i] })
+
+	var ss SequenceSet
+	for _, n := range nums {
+		ss.Insert(n)
+	}
+
+	nums = nums[:0]
+	ss.Range(func(n uint64) bool {
+		nums = append(nums, n)
+		return true
+	})
+	require_True(t, len(nums) == num)
+	for i := uint64(0); i < uint64(num); i++ {
+		require_True(t, nums[i] == i)
+	}
+
+	// Test truncating the range call.
+	nums = nums[:0]
+	ss.Range(func(n uint64) bool {
+		if n >= 10 {
+			return false
+		}
+		nums = append(nums, n)
+		return true
+	})
+	require_True(t, len(nums) == 10)
+	for i := uint64(0); i < 10; i++ {
+		require_True(t, nums[i] == i)
+	}
+}
+
+func TestSeqSetDelete(t *testing.T) {
+	var ss SequenceSet
+
+	// Simple single node.
+	seqs := []uint64{22, 222, 2222, 2, 2, 4}
+	for _, seq := range seqs {
+		ss.Insert(seq)
+	}
+
+	for _, seq := range seqs {
+		ss.Delete(seq)
+		require_True(t, !ss.Exists(seq))
+	}
+	require_True(t, ss.root == nil)
+}
+
+func TestSeqSetInsertAndDeletePedantic(t *testing.T) {
+	var ss SequenceSet
+
+	num := 50*numEntries + 22
+	nums := make([]uint64, 0, num)
+	for i := 0; i < num; i++ {
+		nums = append(nums, uint64(i))
+	}
+	rand.Shuffle(len(nums), func(i, j int) { nums[i], nums[j] = nums[j], nums[i] })
+
+	// Make sure always balanced.
+	testBalanced := func() {
+		t.Helper()
+		// Check heights.
+		ss.root.nodeIter(func(n *node) {
+			if n != nil && n.h != maxH(n)+1 {
+				t.Fatalf("Node height is wrong: %+v", n)
+			}
+		})
+		// Check balance factor.
+		if bf := balanceF(ss.root); bf > 1 || bf < -1 {
+			t.Fatalf("Unbalanced tree")
+		}
+	}
+
+	for _, n := range nums {
+		ss.Insert(n)
+		testBalanced()
+	}
+	require_True(t, ss.root != nil)
+
+	for _, n := range nums {
+		ss.Delete(n)
+		testBalanced()
+		require_True(t, !ss.Exists(n))
+		if ss.Size() > 0 {
+			require_True(t, ss.root != nil)
+		}
+	}
+	require_True(t, ss.root == nil)
+}
+
+func TestSeqSetMinMax(t *testing.T) {
+	var ss SequenceSet
+
+	// Simple single node.
+	seqs := []uint64{22, 222, 2222, 2, 2, 4}
+	for _, seq := range seqs {
+		ss.Insert(seq)
+	}
+
+	min, max := ss.MinMax()
+	require_True(t, min == 2 && max == 2222)
+
+	// Multi-node
+	ss.Empty()
+
+	num := 22*numEntries + 22
+	nums := make([]uint64, 0, num)
+	for i := 0; i < num; i++ {
+		nums = append(nums, uint64(i))
+	}
+	rand.Shuffle(len(nums), func(i, j int) { nums[i], nums[j] = nums[j], nums[i] })
+	for _, n := range nums {
+		ss.Insert(n)
+	}
+
+	min, max = ss.MinMax()
+	require_True(t, min == 0 && max == uint64(num-1))
+}
+
+func TestSeqSetClone(t *testing.T) {
+	// Generate 100k sequences across 500k range.
+	num := 100_000
+	max := 500_000
+
+	var ss SequenceSet
+	for i := 0; i < num; i++ {
+		ss.Insert(uint64(rand.Int64N(int64(max + 1))))
+	}
+
+	ssc := ss.Clone()
+	require_True(t, ss.Size() == ssc.Size())
+	require_True(t, ss.Nodes() == ssc.Nodes())
+}
+
+func TestSeqSetEqual(t *testing.T) {
+	var ss1, ss2, ss3 SequenceSet
+
+	// Empty and nil sets are all equal.
+	var nilSet *SequenceSet
+	require_True(t, ss1.Equal(&ss2))
+	require_True(t, ss1.Equal(nilSet))
+	require_True(t, nilSet.Equal(&ss1))
+
+	// Same state (first, last, size) but different contents.
+	for _, seq := range []uint64{1, 3, 5} {
+		ss1.Insert(seq)
+	}
+	for _, seq := range []uint64{1, 4, 5} {
+		ss2.Insert(seq)
+	}
+	require_False(t, ss1.Equal(&ss2))
+	require_False(t, ss2.Equal(&ss1))
+	require_False(t, ss1.Equal(nilSet))
+	require_False(t, nilSet.Equal(&ss1))
+
+	// Identical contents, insertion order should not matter.
+	for _, seq := range []uint64{5, 1, 3} {
+		ss3.Insert(seq)
+	}
+	require_True(t, ss1.Equal(&ss3))
+	require_True(t, ss3.Equal(&ss1))
+
+	// Differing sizes.
+	ss3.Insert(7)
+	require_False(t, ss1.Equal(&ss3))
+}
+
+func TestSeqSetUnion(t *testing.T) {
+	var ss1, ss2 SequenceSet
+
+	seqs1 := []uint64{22, 222, 2222, 2, 2, 4}
+	for _, seq := range seqs1 {
+		ss1.Insert(seq)
+	}
+
+	seqs2 := []uint64{33, 333, 3333, 3, 33_333, 333_333}
+	for _, seq := range seqs2 {
+		ss2.Insert(seq)
+	}
+
+	ss := Union(&ss1, &ss2)
+	require_True(t, ss.Size() == 11)
+
+	seqs := append(seqs1, seqs2...)
+	for _, n := range seqs {
+		require_True(t, ss.Exists(n))
+	}
+}
+
+func TestSeqSetFirst(t *testing.T) {
+	var ss SequenceSet
+
+	seqs := []uint64{22, 222, 2222, 222_222}
+	for _, seq := range seqs {
+		// Normal case where we pick first/base.
+		ss.Insert(seq)
+		require_True(t, ss.root.base == (seq/numEntries)*numEntries)
+		ss.Empty()
+		// Where we set the minimum start value.
+		ss.SetInitialMin(seq)
+		ss.Insert(seq)
+		require_True(t, ss.root.base == seq)
+		ss.Empty()
+	}
+}
+
+// Test that we can union with nodes vs individual sequence insertion.
+func TestSeqSetDistinctUnion(t *testing.T) {
+	// Distinct sets.
+	var ss1 SequenceSet
+	seqs1 := []uint64{1, 10, 100, 200}
+	for _, seq := range seqs1 {
+		ss1.Insert(seq)
+	}
+
+	var ss2 SequenceSet
+	seqs2 := []uint64{5000, 6100, 6200, 6222}
+	for _, seq := range seqs2 {
+		ss2.Insert(seq)
+	}
+
+	ss := ss1.Clone()
+	allSeqs := append(seqs1, seqs2...)
+
+	ss.Union(&ss2)
+	require_True(t, ss.Size() == len(allSeqs))
+	for _, seq := range allSeqs {
+		require_True(t, ss.Exists(seq))
+	}
+}
+
+func TestSeqSetDecodeV1(t *testing.T) {
+	// Encoding from v1 which was 64 buckets.
+	seqs := []uint64{22, 222, 2222, 222_222, 2_222_222}
+	encStr := `
+FgEDAAAABQAAAABgAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAADgIQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAA==
+`
+
+	enc, err := base64.StdEncoding.DecodeString(encStr)
+	require_NoError(t, err)
+
+	ss, _, err := Decode(enc)
+	require_NoError(t, err)
+
+	require_True(t, ss.Size() == len(seqs))
+	for _, seq := range seqs {
+		require_True(t, ss.Exists(seq))
+	}
+}
+
+func TestSeqSetDecodeNodeCountOverflow(t *testing.T) {
+	// A replicated stream state carries an avl-encoded delete map decoded from
+	// a peer. The node count is read as a uint32 into an int; on 32-bit builds
+	// a value above MaxInt32 turns negative (and a smaller value can overflow
+	// the nn*perNode multiply), defeating the length check so the decode then
+	// panics in make([]node, nn) or reads past the buffer. Both encodings must
+	// reject such counts on every architecture instead.
+	le := binary.LittleEndian
+	for _, ver := range []uint8{1, 2} {
+		buf := make([]byte, minLen)
+		buf[0], buf[1] = magic, ver
+		le.PutUint32(buf[2:], 0xFFFFFFFF) // node count, negative as a 32-bit int
+		if _, _, err := Decode(buf); err != ErrBadEncoding {
+			t.Fatalf("v%d: expected ErrBadEncoding for oversized node count, got %v", ver, err)
+		}
+
+		// Positive on 32-bit but the multiply still overflows for a short buffer.
+		le.PutUint32(buf[2:], 0x00800000)
+		if _, _, err := Decode(buf); err != ErrBadEncoding {
+			t.Fatalf("v%d: expected ErrBadEncoding for overflowing node count, got %v", ver, err)
+		}
+	}
+
+	// A well-formed single-node v2 buffer must still decode.
+	valid := make([]byte, minLen+(numBuckets+1)*8+2)
+	valid[0], valid[1] = magic, version
+	le.PutUint32(valid[2:], 1)
+	if _, _, err := Decode(valid); err != nil {
+		t.Fatalf("valid single-node buffer should decode, got %v", err)
+	}
+}
+
+func TestSeqSetInsertExistingNodeFastPath(t *testing.T) {
+	var ss SequenceSet
+
+	// Build a multi-node tree with only even sequences set.
+	for i := uint64(0); i < 8*numEntries; i += 2 {
+		ss.Insert(i)
+	}
+	nodes, size := ss.Nodes(), ss.Size()
+
+	// Inserting into existing nodes must not change the tree structure.
+	for i := uint64(1); i < 8*numEntries; i += 2 {
+		ss.Insert(i)
+		require_True(t, ss.Exists(i))
+	}
+	require_True(t, ss.Nodes() == nodes)
+	require_True(t, ss.Size() == size+4*numEntries)
+
+	// Duplicate inserts must not change the size.
+	for i := uint64(0); i < 8*numEntries; i++ {
+		ss.Insert(i)
+	}
+	require_True(t, ss.Size() == 8*numEntries)
+
+	// Heights and balance must still be intact.
+	ss.root.nodeIter(func(n *node) {
+		if n.h != maxH(n)+1 {
+			t.Fatalf("Node height is wrong: %+v", n)
+		}
+	})
+	if bf := balanceF(ss.root); bf > 1 || bf < -1 {
+		t.Fatalf("Unbalanced tree")
+	}
+}
+
+func TestSeqSetInsertUnalignedBase(t *testing.T) {
+	var ss SequenceSet
+
+	// SetInitialMin can create a node whose base is not aligned to numEntries.
+	require_NoError(t, ss.SetInitialMin(1000))
+	for _, seq := range []uint64{1000, 1001, 1000 + numEntries - 1} {
+		ss.Insert(seq)
+		require_True(t, ss.Exists(seq))
+	}
+	require_True(t, ss.Nodes() == 1)
+	require_True(t, ss.Size() == 3)
+
+	// Outside the node's range still creates a new node.
+	ss.Insert(1000 + numEntries)
+	require_True(t, ss.Exists(1000+numEntries))
+	require_True(t, ss.Nodes() == 2)
+	require_True(t, ss.Size() == 4)
+}
+
+func TestSeqSetDeleteNotPresent(t *testing.T) {
+	var ss SequenceSet
+
+	seqs := []uint64{22, 222, 2222, 222_222}
+	for _, seq := range seqs {
+		ss.Insert(seq)
+	}
+	nodes, size := ss.Nodes(), ss.Size()
+
+	// Deleting sequences that are not set, both inside and outside
+	// existing node ranges, must be a no-op and report false.
+	for _, seq := range []uint64{0, 23, 2223, 5000, 100_000, 999_999} {
+		require_True(t, !ss.Delete(seq))
+	}
+	require_True(t, ss.Nodes() == nodes)
+	require_True(t, ss.Size() == size)
+	for _, seq := range seqs {
+		require_True(t, ss.Exists(seq))
+	}
+
+	// Real deletes still work and empty the set.
+	for _, seq := range seqs {
+		require_True(t, ss.Delete(seq))
+	}
+	require_True(t, ss.root == nil)
+}
+
+func TestSeqSetRangeSparse(t *testing.T) {
+	// Cover bucket boundaries, node boundaries and large gaps.
+	seqs := []uint64{0, 1, 63, 64, 127, 128, 2047, 2048, 4095, 4096, 100_000, 1_000_000}
+	var ss SequenceSet
+	for _, seq := range seqs {
+		ss.Insert(seq)
+	}
+
+	var got []uint64
+	ss.Range(func(n uint64) bool {
+		got = append(got, n)
+		return true
+	})
+	require_True(t, len(got) == len(seqs))
+	for i, seq := range seqs {
+		require_True(t, got[i] == seq)
+	}
+
+	// Terminating mid-bucket should stop the iteration immediately.
+	got = got[:0]
+	ss.Range(func(n uint64) bool {
+		got = append(got, n)
+		return n != 64
+	})
+	require_True(t, len(got) == 4)
+	require_True(t, got[3] == 64)
+}
+
+func BenchmarkSeqSetInsertDense(b *testing.B) {
+	for i := 0; i < b.N; i++ {
+		var ss SequenceSet
+		for s := uint64(0); s < 200_000; s++ {
+			ss.Insert(s)
+		}
+	}
+}
+
+func BenchmarkSeqSetInsertSparse(b *testing.B) {
+	for i := 0; i < b.N; i++ {
+		var ss SequenceSet
+		for s := uint64(0); s < 200_000; s++ {
+			ss.Insert(s * 7)
+		}
+	}
+}
+
+func BenchmarkSeqSetRangeDense(b *testing.B) {
+	var ss SequenceSet
+	for s := uint64(0); s < 200_000; s++ {
+		ss.Insert(s)
+	}
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		var sum uint64
+		ss.Range(func(n uint64) bool { sum += n; return true })
+	}
+}
+
+func BenchmarkSeqSetRangeSparse(b *testing.B) {
+	var ss SequenceSet
+	for s := uint64(0); s < 10_000; s++ {
+		ss.Insert(s * 211) // ~10 bits per 2048-entry node
+	}
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		var sum uint64
+		ss.Range(func(n uint64) bool { sum += n; return true })
+	}
+}
+
+func BenchmarkSeqSetDeleteNotPresent(b *testing.B) {
+	var ss SequenceSet
+	for s := uint64(0); s < 200_000; s += 2 {
+		ss.Insert(s)
+	}
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		for s := uint64(1); s < 200_000; s += 2 {
+			ss.Delete(s)
+		}
+	}
+}
+
+func require_NoError(t *testing.T, err error) {
+	t.Helper()
+	if err != nil {
+		t.Fatalf("require no error, but got: %v", err)
+	}
+}
+
+func require_True(t *testing.T, b bool) {
+	t.Helper()
+	if !b {
+		t.Fatalf("require true")
+	}
+}
+
+func require_False(t *testing.T, b bool) {
+	t.Helper()
+	if b {
+		t.Fatalf("require false")
+	}
+}

server/benchmark_publish_test.go

@@ -0,0 +1,185 @@
+// Copyright 2022-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"fmt"
+	"math/rand/v2"
+	"sync/atomic"
+	"testing"
+
+	"github.com/nats-io/nats.go"
+)
+
+func BenchmarkPublish(b *testing.B) {
+
+	const (
+		verbose     = false
+		minMessages = 10_000
+		subject     = "S"
+		queue       = "Q"
+	)
+
+	const (
+		KB = 1024
+		MB = KB * KB
+	)
+
+	type SubscriberType string
+	const (
+		Async      SubscriberType = "Async"
+		QueueAsync SubscriberType = "AsyncQueue"
+		None       SubscriberType = "None"
+	)
+
+	benchmarksCases := []struct {
+		messageSize int
+	}{
+		{0},
+		{1},
+		{32},
+		{128},
+		{512},
+		{4 * KB},
+		{32 * KB},
+		{128 * KB},
+		{512 * KB},
+		{1 * MB},
+	}
+
+	// All the cases above are run for each of the subscriber cases below
+	subscribersCases := []struct {
+		numSubs int
+		subType SubscriberType
+	}{
+		{0, None},
+		{1, Async},
+		{1, QueueAsync},
+		{10, Async},
+		{10, QueueAsync},
+	}
+
+	for _, bc := range benchmarksCases {
+		bcName := fmt.Sprintf(
+			"MsgSz=%db",
+			bc.messageSize,
+		)
+
+		b.Run(
+			bcName,
+			func(b *testing.B) {
+
+				for _, sc := range subscribersCases {
+
+					scName := fmt.Sprintf("Subs=%dx%v", sc.numSubs, sc.subType)
+					if sc.subType == None {
+						scName = fmt.Sprintf("Subs=%v", sc.subType)
+					}
+
+					b.Run(
+						scName,
+						func(b *testing.B) {
+							// Skip short runs, benchmark gets re-executed with a larger N
+							if b.N < minMessages {
+								b.ResetTimer()
+								return
+							}
+
+							if verbose {
+								b.Logf("Running %s/%s with %d ops", bcName, scName, b.N)
+							}
+
+							subErrors := uint64(0)
+							handleSubError := func(_ *nats.Conn, _ *nats.Subscription, _ error) {
+								atomic.AddUint64(&subErrors, 1)
+							}
+
+							// Start single server (no JS)
+							opts := DefaultTestOptions
+							opts.Port = -1
+							s := RunServer(&opts)
+							defer s.Shutdown()
+
+							// Create subscribers
+							for i := 0; i < sc.numSubs; i++ {
+								subConn, connErr := nats.Connect(s.ClientURL(), nats.ErrorHandler(handleSubError))
+								if connErr != nil {
+									b.Fatalf("Failed to connect: %v", connErr)
+								}
+								defer subConn.Close()
+
+								var sub *nats.Subscription
+								var subErr error
+
+								switch sc.subType {
+								case None:
+									// No subscription
+								case Async:
+									sub, subErr = subConn.Subscribe(subject, func(*nats.Msg) {})
+								case QueueAsync:
+									sub, subErr = subConn.QueueSubscribe(subject, queue, func(*nats.Msg) {})
+								default:
+									b.Fatalf("Unknow subscribers type: %v", sc.subType)
+								}
+
+								if subErr != nil {
+									b.Fatalf("Failed to subscribe: %v", subErr)
+								}
+								defer sub.Unsubscribe()
+								// Do not drop messages due to slow subscribers:
+								sub.SetPendingLimits(-1, -1)
+							}
+
+							// Create publisher connection
+							nc, err := nats.Connect(s.ClientURL())
+							if err != nil {
+								b.Fatalf("Failed to connect: %v", err)
+							}
+							defer nc.Close()
+
+							rng := rand.NewChaCha8([32]byte{42})
+							message := make([]byte, bc.messageSize)
+							var published, errors int
+
+							// Benchmark starts here
+							b.ResetTimer()
+
+							for i := 0; i < b.N; i++ {
+								rng.Read(message)
+								pubErr := nc.Publish(subject, message)
+								if pubErr != nil {
+									errors++
+								} else {
+									published++
+									b.SetBytes(int64(bc.messageSize))
+								}
+							}
+
+							// Benchmark ends here
+							b.StopTimer()
+
+							if published+errors != b.N {
+								b.Fatalf("Something doesn't add up: %d + %d != %d", published, errors, b.N)
+							} else if subErrors > 0 {
+								b.Fatalf("Subscribers errors: %d", subErrors)
+							}
+
+							b.ReportMetric(float64(errors)*100/float64(b.N), "%error")
+						},
+					)
+				}
+			},
+		)
+	}
+}

server/certidp/certidp.go

@@ -0,0 +1,312 @@
+// Copyright 2023-2024 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package certidp
+
+import (
+	"crypto/sha256"
+	"crypto/x509"
+	"encoding/base64"
+	"encoding/json"
+	"fmt"
+	"net/url"
+	"strings"
+	"time"
+
+	"golang.org/x/crypto/ocsp"
+)
+
+const (
+	DefaultAllowedClockSkew     = 30 * time.Second
+	DefaultOCSPResponderTimeout = 2 * time.Second
+	DefaultTTLUnsetNextUpdate   = 1 * time.Hour
+)
+
+type StatusAssertion int
+
+var (
+	StatusAssertionStrToVal = map[string]StatusAssertion{
+		"good":    ocsp.Good,
+		"revoked": ocsp.Revoked,
+		"unknown": ocsp.Unknown,
+	}
+	StatusAssertionValToStr = map[StatusAssertion]string{
+		ocsp.Good:    "good",
+		ocsp.Revoked: "revoked",
+		ocsp.Unknown: "unknown",
+	}
+	StatusAssertionIntToVal = map[int]StatusAssertion{
+		0: ocsp.Good,
+		1: ocsp.Revoked,
+		2: ocsp.Unknown,
+	}
+)
+
+// GetStatusAssertionStr returns the corresponding string representation of the StatusAssertion.
+func GetStatusAssertionStr(sa int) string {
+	// If the provided status assertion value is not found in the map (StatusAssertionIntToVal),
+	// the function defaults to "unknown" to avoid defaulting to "good," which is the default iota value
+	// for the ocsp.StatusAssertion enumeration (https://pkg.go.dev/golang.org/x/crypto/ocsp#pkg-constants).
+	// This ensures that we don't unintentionally default to "good" when there's no map entry.
+	v, ok := StatusAssertionIntToVal[sa]
+	if !ok {
+		// set unknown as fallback
+		v = ocsp.Unknown
+	}
+
+	return StatusAssertionValToStr[v]
+}
+
+func (sa StatusAssertion) MarshalJSON() ([]byte, error) {
+	// This ensures that we don't unintentionally default to "good" when there's no map entry.
+	// (see more details in the GetStatusAssertionStr() comment)
+	str, ok := StatusAssertionValToStr[sa]
+	if !ok {
+		// set unknown as fallback
+		str = StatusAssertionValToStr[ocsp.Unknown]
+	}
+	return json.Marshal(str)
+}
+
+func (sa *StatusAssertion) UnmarshalJSON(in []byte) error {
+	// This ensures that we don't unintentionally default to "good" when there's no map entry.
+	// (see more details in the GetStatusAssertionStr() comment)
+	v, ok := StatusAssertionStrToVal[strings.ReplaceAll(string(in), "\"", "")]
+	if !ok {
+		// set unknown as fallback
+		v = StatusAssertionStrToVal["unknown"]
+	}
+	*sa = v
+	return nil
+}
+
+type ChainLink struct {
+	Leaf             *x509.Certificate
+	Issuer           *x509.Certificate
+	OCSPWebEndpoints *[]*url.URL
+}
+
+// OCSPPeerConfig holds the parsed OCSP peer configuration section of TLS configuration
+type OCSPPeerConfig struct {
+	Verify                 bool
+	Timeout                float64
+	ClockSkew              float64
+	WarnOnly               bool
+	UnknownIsGood          bool
+	AllowWhenCAUnreachable bool
+	TTLUnsetNextUpdate     float64
+}
+
+func NewOCSPPeerConfig() *OCSPPeerConfig {
+	return &OCSPPeerConfig{
+		Verify:                 false,
+		Timeout:                DefaultOCSPResponderTimeout.Seconds(),
+		ClockSkew:              DefaultAllowedClockSkew.Seconds(),
+		WarnOnly:               false,
+		UnknownIsGood:          false,
+		AllowWhenCAUnreachable: false,
+		TTLUnsetNextUpdate:     DefaultTTLUnsetNextUpdate.Seconds(),
+	}
+}
+
+// Log is a neutral method of passing server loggers to plugins
+type Log struct {
+	Debugf  func(format string, v ...any)
+	Noticef func(format string, v ...any)
+	Warnf   func(format string, v ...any)
+	Errorf  func(format string, v ...any)
+	Tracef  func(format string, v ...any)
+}
+
+type CertInfo struct {
+	Subject     string `json:"subject,omitempty"`
+	Issuer      string `json:"issuer,omitempty"`
+	Fingerprint string `json:"fingerprint,omitempty"`
+	Raw         []byte `json:"raw,omitempty"`
+}
+
+var OCSPPeerUsage = `
+For client, leaf spoke (remotes), and leaf hub connections, you may enable OCSP peer validation:
+
+    tls {
+        ...
+        # mTLS must be enabled (with exception of Leaf remotes)
+        verify: true
+        ...
+        # short form enables peer verify and takes option defaults
+        ocsp_peer: true
+
+        # long form includes settable options
+        ocsp_peer {
+           # Enable OCSP peer validation (default false)
+           verify: true
+
+           # OCSP responder timeout in seconds (may be fractional, default 2 seconds)
+           ca_timeout: 2
+
+           # Allowed skew between server and OCSP responder time in seconds (may be fractional, default 30 seconds)
+           allowed_clockskew: 30
+
+           # Warn-only and never reject connections (default false)
+           warn_only: false
+
+           # Treat response Unknown status as valid certificate (default false)
+           unknown_is_good: false
+
+           # Warn-only if no CA response can be obtained and no cached revocation exists (default false)
+           allow_when_ca_unreachable: false
+
+           # If response NextUpdate unset by CA, set a default cache TTL in seconds from ThisUpdate (default 1 hour)
+           cache_ttl_when_next_update_unset: 3600
+        }
+        ...
+    }
+
+Note: OCSP validation for route and gateway connections is enabled using the 'ocsp' configuration option.
+`
+
+// GenerateFingerprint returns a base64-encoded SHA256 hash of the raw certificate
+func GenerateFingerprint(cert *x509.Certificate) string {
+	data := sha256.Sum256(cert.Raw)
+	return base64.StdEncoding.EncodeToString(data[:])
+}
+
+func getWebEndpoints(uris []string) []*url.URL {
+	var urls []*url.URL
+	for _, uri := range uris {
+		endpoint, err := url.ParseRequestURI(uri)
+		if err != nil {
+			// skip invalid URLs
+			continue
+		}
+		if endpoint.Scheme != "http" && endpoint.Scheme != "https" {
+			// skip non-web URLs
+			continue
+		}
+		urls = append(urls, endpoint)
+	}
+	return urls
+}
+
+// GetSubjectDNForm returns RDN sequence concatenation of the certificate's subject to be
+// used in logs, events, etc. Should never be used for reliable cache matching or other crypto purposes.
+func GetSubjectDNForm(cert *x509.Certificate) string {
+	if cert == nil {
+		return ""
+	}
+	return strings.TrimSuffix(fmt.Sprintf("%s+", cert.Subject.ToRDNSequence()), "+")
+}
+
+// GetIssuerDNForm returns RDN sequence concatenation of the certificate's issuer to be
+// used in logs, events, etc. Should never be used for reliable cache matching or other crypto purposes.
+func GetIssuerDNForm(cert *x509.Certificate) string {
+	if cert == nil {
+		return ""
+	}
+	return strings.TrimSuffix(fmt.Sprintf("%s+", cert.Issuer.ToRDNSequence()), "+")
+}
+
+// CertOCSPEligible checks if the certificate's issuer has populated AIA with OCSP responder endpoint(s)
+// and is thus eligible for OCSP validation
+func CertOCSPEligible(link *ChainLink) bool {
+	if link == nil || link.Leaf.Raw == nil || len(link.Leaf.Raw) == 0 {
+		return false
+	}
+	if len(link.Leaf.OCSPServer) == 0 {
+		return false
+	}
+	urls := getWebEndpoints(link.Leaf.OCSPServer)
+	if len(urls) == 0 {
+		return false
+	}
+	link.OCSPWebEndpoints = &urls
+	return true
+}
+
+// GetLeafIssuerCert returns the issuer certificate of the leaf (positional) certificate in the chain
+func GetLeafIssuerCert(chain []*x509.Certificate, leafPos int) *x509.Certificate {
+	if len(chain) == 0 || leafPos < 0 {
+		return nil
+	}
+	// self-signed certificate or too-big leafPos
+	if leafPos >= len(chain)-1 {
+		return nil
+	}
+	// returns pointer to issuer cert or nil
+	return (chain)[leafPos+1]
+}
+
+// OCSPResponseCurrent checks if the OCSP response is current (i.e. not expired and not future effective)
+func OCSPResponseCurrent(ocspr *ocsp.Response, opts *OCSPPeerConfig, log *Log) bool {
+	skew := time.Duration(opts.ClockSkew * float64(time.Second))
+	if skew < 0*time.Second {
+		skew = DefaultAllowedClockSkew
+	}
+	now := time.Now().UTC()
+	// Typical effectivity check based on CA response ThisUpdate and NextUpdate semantics
+	if !ocspr.NextUpdate.IsZero() && ocspr.NextUpdate.Before(now.Add(-1*skew)) {
+		t := ocspr.NextUpdate.Format(time.RFC3339Nano)
+		nt := now.Format(time.RFC3339Nano)
+		log.Debugf(DbgResponseExpired, t, nt, skew)
+		return false
+	}
+	// CA responder can assert NextUpdate unset, in which case use config option to set a default cache TTL
+	if ocspr.NextUpdate.IsZero() {
+		ttl := time.Duration(opts.TTLUnsetNextUpdate * float64(time.Second))
+		if ttl < 0*time.Second {
+			ttl = DefaultTTLUnsetNextUpdate
+		}
+		expiryTime := ocspr.ThisUpdate.Add(ttl)
+		if expiryTime.Before(now.Add(-1 * skew)) {
+			t := expiryTime.Format(time.RFC3339Nano)
+			nt := now.Format(time.RFC3339Nano)
+			log.Debugf(DbgResponseTTLExpired, t, nt, skew)
+			return false
+		}
+	}
+	if ocspr.ThisUpdate.After(now.Add(skew)) {
+		t := ocspr.ThisUpdate.Format(time.RFC3339Nano)
+		nt := now.Format(time.RFC3339Nano)
+		log.Debugf(DbgResponseFutureDated, t, nt, skew)
+		return false
+	}
+	return true
+}
+
+// ValidDelegationCheck checks if the CA OCSP Response was signed by a valid CA Issuer delegate as per (RFC 6960, section 4.2.2.2)
+// If a valid delegate or direct-signed by CA Issuer, true returned.
+func ValidDelegationCheck(iss *x509.Certificate, ocspr *ocsp.Response) bool {
+	// This call assumes prior successful parse and signature validation of the OCSP response
+	// The Go OCSP library (as of x/crypto/ocsp v0.9) will detect and perform a 1-level delegate signature check but does not
+	// implement the additional criteria for delegation specified in RFC 6960, section 4.2.2.2.
+	if iss == nil || ocspr == nil {
+		return false
+	}
+	// not a delegation, no-op
+	if ocspr.Certificate == nil {
+		return true
+	}
+	// delegate is self-same with CA Issuer, not a delegation although response issued in that form
+	if ocspr.Certificate.Equal(iss) {
+		return true
+	}
+	// we need to verify CA Issuer stamped id-kp-OCSPSigning on delegate
+	delegatedSigner := false
+	for _, keyUseExt := range ocspr.Certificate.ExtKeyUsage {
+		if keyUseExt == x509.ExtKeyUsageOCSPSigning {
+			delegatedSigner = true
+			break
+		}
+	}
+	return delegatedSigner
+}

server/certidp/certidp_test.go

@@ -0,0 +1,56 @@
+// Copyright 2023 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package certidp
+
+import "testing"
+
+// Checks the return values of the function GetStatusAssertionStr
+func TestGetStatusAssertionStr(t *testing.T) {
+	tests := []struct {
+		name     string
+		input    int
+		expected string
+	}{
+		{
+			name:     "GoodStatus",
+			input:    0,
+			expected: "good",
+		},
+		{
+			name:     "RevokedStatus",
+			input:    1,
+			expected: "revoked",
+		},
+		{
+			name:     "UnknownStatus",
+			input:    2,
+			expected: "unknown",
+		},
+		// Invalid status assertion value.
+		{
+			name:     "InvalidStatus",
+			input:    42,
+			expected: "unknown",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got := GetStatusAssertionStr(tt.input)
+			if got != tt.expected {
+				t.Errorf("Expected GetStatusAssertionStr: %v, got %v", tt.expected, got)
+			}
+		})
+	}
+}

server/certidp/messages.go

@@ -0,0 +1,106 @@
+// Copyright 2023 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package certidp
+
+var (
+	// Returned errors
+	ErrIllegalPeerOptsConfig              = "expected map to define OCSP peer options, got [%T]"
+	ErrIllegalCacheOptsConfig             = "expected map to define OCSP peer cache options, got [%T]"
+	ErrParsingPeerOptFieldGeneric         = "error parsing tls peer config, unknown field [%q]"
+	ErrParsingPeerOptFieldTypeConversion  = "error parsing tls peer config, conversion error: %s"
+	ErrParsingCacheOptFieldTypeConversion = "error parsing OCSP peer cache config, conversion error: %s"
+	ErrUnableToPlugTLSEmptyConfig         = "unable to plug TLS verify connection, config is nil"
+	ErrMTLSRequired                       = "OCSP peer verification for client connections requires TLS verify (mTLS) to be enabled"
+	ErrUnableToPlugTLSClient              = "unable to register client OCSP verification"
+	ErrUnableToPlugTLSServer              = "unable to register server OCSP verification"
+	ErrCannotWriteCompressed              = "error writing to compression writer: %w"
+	ErrCannotReadCompressed               = "error reading compression reader: %w"
+	ErrTruncatedWrite                     = "short write on body (%d != %d)"
+	ErrCannotCloseWriter                  = "error closing compression writer: %w"
+	ErrParsingCacheOptFieldGeneric        = "error parsing OCSP peer cache config, unknown field [%q]"
+	ErrUnknownCacheType                   = "error parsing OCSP peer cache config, unknown type [%s]"
+	ErrInvalidChainlink                   = "invalid chain link"
+	ErrBadResponderHTTPStatus             = "bad OCSP responder http status: [%d]"
+	ErrNoAvailOCSPServers                 = "no available OCSP servers"
+	ErrFailedWithAllRequests              = "exhausted OCSP responders: %w"
+
+	// Direct logged errors
+	ErrLoadCacheFail          = "Unable to load OCSP peer cache: %s"
+	ErrSaveCacheFail          = "Unable to save OCSP peer cache: %s"
+	ErrBadCacheTypeConfig     = "Unimplemented OCSP peer cache type [%v]"
+	ErrResponseCompressFail   = "Unable to compress OCSP response for key [%s]: %s"
+	ErrResponseDecompressFail = "Unable to decompress OCSP response for key [%s]: %s"
+	ErrPeerEmptyNoEvent       = "Peer certificate is nil, cannot send OCSP peer reject event"
+	ErrPeerEmptyAutoReject    = "Peer certificate is nil, rejecting OCSP peer"
+
+	// Debug information
+	DbgPlugTLSForKind        = "Plugging TLS OCSP peer for [%s]"
+	DbgNumServerChains       = "Peer OCSP enabled: %d TLS server chain(s) will be evaluated"
+	DbgNumClientChains       = "Peer OCSP enabled: %d TLS client chain(s) will be evaluated"
+	DbgLinksInChain          = "Chain [%d]: %d total link(s)"
+	DbgSelfSignedValid       = "Chain [%d] is self-signed, thus peer is valid"
+	DbgValidNonOCSPChain     = "Chain [%d] has no OCSP eligible links, thus peer is valid"
+	DbgChainIsOCSPEligible   = "Chain [%d] has %d OCSP eligible link(s)"
+	DbgChainIsOCSPValid      = "Chain [%d] is OCSP valid for all eligible links, thus peer is valid"
+	DbgNoOCSPValidChains     = "No OCSP valid chains, thus peer is invalid"
+	DbgCheckingCacheForCert  = "Checking OCSP peer cache for [%s], key [%s]"
+	DbgCurrentResponseCached = "Cached OCSP response is current, status [%s]"
+	DbgExpiredResponseCached = "Cached OCSP response is expired, status [%s]"
+	DbgOCSPValidPeerLink     = "OCSP verify pass for [%s]"
+	DbgCachingResponse       = "Caching OCSP response for [%s], key [%s]"
+	DbgAchievedCompression   = "OCSP response compression ratio: [%f]"
+	DbgCacheHit              = "OCSP peer cache hit for key [%s]"
+	DbgCacheMiss             = "OCSP peer cache miss for key [%s]"
+	DbgPreservedRevocation   = "Revoked OCSP response for key [%s] preserved by cache policy"
+	DbgDeletingCacheResponse = "Deleting OCSP peer cached response for key [%s]"
+	DbgStartingCache         = "Starting OCSP peer cache"
+	DbgStoppingCache         = "Stopping OCSP peer cache"
+	DbgLoadingCache          = "Loading OCSP peer cache [%s]"
+	DbgNoCacheFound          = "No OCSP peer cache found, starting with empty cache"
+	DbgSavingCache           = "Saving OCSP peer cache [%s]"
+	DbgCacheSaved            = "Saved OCSP peer cache successfully (%d bytes)"
+	DbgMakingCARequest       = "Trying OCSP responder url [%s]"
+	DbgResponseExpired       = "OCSP response NextUpdate [%s] is before now [%s] with clockskew [%s]"
+	DbgResponseTTLExpired    = "OCSP response cache expiry [%s] is before now [%s] with clockskew [%s]"
+	DbgResponseFutureDated   = "OCSP response ThisUpdate [%s] is before now [%s] with clockskew [%s]"
+	DbgCacheSaveTimerExpired = "OCSP peer cache save timer expired"
+	DbgCacheDirtySave        = "OCSP peer cache is dirty, saving"
+
+	// Returned to peer as TLS reject reason
+	MsgTLSClientRejectConnection = "client not OCSP valid"
+	MsgTLSServerRejectConnection = "server not OCSP valid"
+
+	// Expected runtime errors (direct logged)
+	ErrCAResponderCalloutFail  = "Attempt to obtain OCSP response from CA responder for [%s] failed: %s"
+	ErrNewCAResponseNotCurrent = "New OCSP CA response obtained for [%s] but not current"
+	ErrCAResponseParseFailed   = "Could not parse OCSP CA response for [%s]: %s"
+	ErrOCSPInvalidPeerLink     = "OCSP verify fail for [%s] with CA status [%s]"
+
+	// Policy override warnings (direct logged)
+	MsgAllowWhenCAUnreachableOccurred             = "Failed to obtain OCSP CA response for [%s] but AllowWhenCAUnreachable set; no cached revocation so allowing"
+	MsgAllowWhenCAUnreachableOccurredCachedRevoke = "Failed to obtain OCSP CA response for [%s] but AllowWhenCAUnreachable set; cached revocation exists so rejecting"
+	MsgAllowWarnOnlyOccurred                      = "OCSP verify fail for [%s] but WarnOnly is true so allowing"
+
+	// Info (direct logged)
+	MsgCacheOnline  = "OCSP peer cache online, type [%s]"
+	MsgCacheOffline = "OCSP peer cache offline, type [%s]"
+
+	// OCSP cert invalid reasons (debug and event reasons)
+	MsgFailedOCSPResponseFetch       = "Failed OCSP response fetch"
+	MsgOCSPResponseNotEffective      = "OCSP response not in effectivity window"
+	MsgFailedOCSPResponseParse       = "Failed OCSP response parse"
+	MsgOCSPResponseInvalidStatus     = "Invalid OCSP response status: %s"
+	MsgOCSPResponseDelegationInvalid = "Invalid OCSP response delegation: %s"
+	MsgCachedOCSPResponseInvalid     = "Invalid cached OCSP response for [%s] with fingerprint [%s]"
+)

server/certidp/ocsp_responder.go

@@ -0,0 +1,92 @@
+// Copyright 2023-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package certidp
+
+import (
+	"encoding/base64"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"net/url"
+	"strings"
+	"time"
+
+	"golang.org/x/crypto/ocsp"
+)
+
+func FetchOCSPResponse(link *ChainLink, opts *OCSPPeerConfig, log *Log) ([]byte, error) {
+	if link == nil || link.Leaf == nil || link.Issuer == nil || opts == nil || log == nil {
+		return nil, errors.New(ErrInvalidChainlink)
+	}
+
+	timeout := time.Duration(opts.Timeout * float64(time.Second))
+	if timeout <= 0*time.Second {
+		timeout = DefaultOCSPResponderTimeout
+	}
+
+	getRequestBytes := func(u string, hc *http.Client) ([]byte, error) {
+		resp, err := hc.Get(u)
+		if err != nil {
+			return nil, err
+		}
+		defer resp.Body.Close()
+		if resp.StatusCode != http.StatusOK {
+			return nil, fmt.Errorf(ErrBadResponderHTTPStatus, resp.StatusCode)
+		}
+		return io.ReadAll(resp.Body)
+	}
+
+	// Request documentation:
+	// https://tools.ietf.org/html/rfc6960#appendix-A.1
+
+	reqDER, err := ocsp.CreateRequest(link.Leaf, link.Issuer, nil)
+	if err != nil {
+		return nil, err
+	}
+
+	reqEnc := encodeOCSPRequest(reqDER)
+
+	responders := *link.OCSPWebEndpoints
+
+	if len(responders) == 0 {
+		return nil, errors.New(ErrNoAvailOCSPServers)
+	}
+
+	var raw []byte
+	hc := &http.Client{
+		Timeout: timeout,
+	}
+	for _, u := range responders {
+		responderURL := u.String()
+		log.Debugf(DbgMakingCARequest, responderURL)
+		responderURL = strings.TrimSuffix(responderURL, "/")
+		raw, err = getRequestBytes(fmt.Sprintf("%s/%s", responderURL, reqEnc), hc)
+		if err == nil {
+			break
+		}
+	}
+	if err != nil {
+		return nil, fmt.Errorf(ErrFailedWithAllRequests, err)
+	}
+
+	return raw, nil
+}
+
+// encodeOCSPRequest encodes the OCSP request in base64 and URL-encodes it.
+// This is needed to fulfill the OCSP responder's requirements for the request format. (X.690)
+func encodeOCSPRequest(reqDER []byte) string {
+	reqEnc := base64.StdEncoding.EncodeToString(reqDER)
+	return url.QueryEscape(reqEnc)
+}

server/certidp/ocsp_responder_test.go

@@ -0,0 +1,29 @@
+package certidp
+
+import (
+	"encoding/base64"
+	"net/url"
+	"strings"
+	"testing"
+)
+
+func TestEncodeOCSPRequest(t *testing.T) {
+	data := []byte("test data for OCSP request")
+	encoded := encodeOCSPRequest(data)
+
+	if strings.ContainsAny(encoded, "+/=") {
+		t.Errorf("URL contains unescaped characters: %s", encoded)
+	}
+
+	decodedURL, err := url.QueryUnescape(encoded)
+	if err != nil {
+		t.Errorf("failed to url-decode request: %v", err)
+	}
+	decodedData, err := base64.StdEncoding.DecodeString(decodedURL)
+	if err != nil {
+		t.Errorf("failed to base64-decode request: %v", err)
+	}
+	if string(decodedData) != string(data) {
+		t.Errorf("Decoded data does not match original: got %s, want %s", decodedData, data)
+	}
+}

server/certstore/certstore.go

@@ -0,0 +1,104 @@
+// Copyright 2022-2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package certstore
+
+import (
+	"crypto"
+	"crypto/x509"
+	"io"
+	"runtime"
+	"strings"
+)
+
+type StoreType int
+
+const MATCHBYEMPTY = 0
+const STOREEMPTY = 0
+
+const (
+	windowsCurrentUser StoreType = iota + 1
+	windowsLocalMachine
+)
+
+var StoreMap = map[string]StoreType{
+	"windowscurrentuser":  windowsCurrentUser,
+	"windowslocalmachine": windowsLocalMachine,
+}
+
+var StoreOSMap = map[StoreType]string{
+	windowsCurrentUser:  "windows",
+	windowsLocalMachine: "windows",
+}
+
+type MatchByType int
+
+const (
+	matchByIssuer MatchByType = iota + 1
+	matchBySubject
+	matchByThumbprint
+)
+
+var MatchByMap = map[string]MatchByType{
+	"issuer":     matchByIssuer,
+	"subject":    matchBySubject,
+	"thumbprint": matchByThumbprint,
+}
+
+var Usage = `
+In place of cert_file and key_file you may use the windows certificate store:
+
+    tls {
+        cert_store:     "WindowsCurrentUser"
+        cert_match_by:  "Subject"
+        cert_match:     "MyServer123"
+    }
+`
+
+func ParseCertStore(certStore string) (StoreType, error) {
+	certStoreType, exists := StoreMap[strings.ToLower(certStore)]
+	if !exists {
+		return 0, ErrBadCertStore
+	}
+	validOS, exists := StoreOSMap[certStoreType]
+	if !exists || validOS != runtime.GOOS {
+		return 0, ErrOSNotCompatCertStore
+	}
+	return certStoreType, nil
+}
+
+func ParseCertMatchBy(certMatchBy string) (MatchByType, error) {
+	certMatchByType, exists := MatchByMap[strings.ToLower(certMatchBy)]
+	if !exists {
+		return 0, ErrBadMatchByType
+	}
+	return certMatchByType, nil
+}
+
+func GetLeafIssuer(leaf *x509.Certificate, vOpts x509.VerifyOptions) (issuer *x509.Certificate) {
+	chains, err := leaf.Verify(vOpts)
+	if err != nil || len(chains) == 0 {
+		issuer = nil
+	} else {
+		issuer = chains[0][1]
+	}
+	return
+}
+
+// credential provides access to a public key and is a crypto.Signer.
+type credential interface {
+	// Public returns the public key corresponding to the leaf certificate.
+	Public() crypto.PublicKey
+	// Sign signs digest with the private key.
+	Sign(rand io.Reader, digest []byte, opts crypto.SignerOpts) (signature []byte, err error)
+}

server/certstore/certstore_other.go

@@ -0,0 +1,45 @@
+// Copyright 2022-2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build !windows
+
+package certstore
+
+import (
+	"crypto"
+	"crypto/tls"
+	"io"
+)
+
+var _ = MATCHBYEMPTY
+
+// otherKey implements crypto.Signer and crypto.Decrypter to satisfy linter on platforms that don't implement certstore
+type otherKey struct{}
+
+func TLSConfig(_ StoreType, _ MatchByType, _ string, _ []string, _ bool, _ *tls.Config) error {
+	return ErrOSNotCompatCertStore
+}
+
+// Public always returns nil public key since this is a stub on non-supported platform
+func (k otherKey) Public() crypto.PublicKey {
+	return nil
+}
+
+// Sign always returns a nil signature since this is a stub on non-supported platform
+func (k otherKey) Sign(rand io.Reader, digest []byte, opts crypto.SignerOpts) (signature []byte, err error) {
+	_, _, _ = rand, digest, opts
+	return nil, nil
+}
+
+// Verify interface conformance.
+var _ credential = &otherKey{}

server/certstore/certstore_windows.go

@@ -0,0 +1,965 @@
+// Copyright 2022-2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+// Adapted, updated, and enhanced from CertToStore, https://github.com/google/certtostore/releases/tag/v1.0.2
+// Apache License, Version 2.0, Copyright 2017 Google Inc.
+
+package certstore
+
+import (
+	"bytes"
+	"crypto"
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rsa"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/binary"
+	"fmt"
+	"io"
+	"math/big"
+	"reflect"
+	"sync"
+	"syscall"
+	"unicode/utf16"
+	"unsafe"
+
+	"golang.org/x/crypto/cryptobyte"
+	"golang.org/x/crypto/cryptobyte/asn1"
+	"golang.org/x/sys/windows"
+)
+
+const (
+	// wincrypt.h constants
+	winAcquireCached         = windows.CRYPT_ACQUIRE_CACHE_FLAG
+	winAcquireSilent         = windows.CRYPT_ACQUIRE_SILENT_FLAG
+	winAcquireOnlyNCryptKey  = windows.CRYPT_ACQUIRE_ONLY_NCRYPT_KEY_FLAG
+	winEncodingX509ASN       = windows.X509_ASN_ENCODING
+	winEncodingPKCS7         = windows.PKCS_7_ASN_ENCODING
+	winCertStoreProvSystem   = windows.CERT_STORE_PROV_SYSTEM
+	winCertStoreCurrentUser  = windows.CERT_SYSTEM_STORE_CURRENT_USER
+	winCertStoreLocalMachine = windows.CERT_SYSTEM_STORE_LOCAL_MACHINE
+	winCertStoreReadOnly     = windows.CERT_STORE_READONLY_FLAG
+	winInfoIssuerFlag        = windows.CERT_INFO_ISSUER_FLAG
+	winInfoSubjectFlag       = windows.CERT_INFO_SUBJECT_FLAG
+	winCompareNameStrW       = windows.CERT_COMPARE_NAME_STR_W
+	winCompareShift          = windows.CERT_COMPARE_SHIFT
+
+	// Reference https://learn.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-certfindcertificateinstore
+	winFindIssuerStr  = windows.CERT_FIND_ISSUER_STR_W
+	winFindSubjectStr = windows.CERT_FIND_SUBJECT_STR_W
+	winFindHashStr    = windows.CERT_FIND_HASH_STR
+
+	winNcryptKeySpec = windows.CERT_NCRYPT_KEY_SPEC
+
+	winBCryptPadPKCS1   uintptr = 0x2
+	winBCryptPadPSS     uintptr = 0x8 // Modern TLS 1.2+
+	winBCryptPadPSSSalt uint32  = 32  // default 20, 32 optimal for typical SHA256 hash
+
+	winRSA1Magic = 0x31415352 // "RSA1" BCRYPT_RSAPUBLIC_MAGIC
+
+	winECS1Magic = 0x31534345 // "ECS1" BCRYPT_ECDSA_PUBLIC_P256_MAGIC
+	winECS3Magic = 0x33534345 // "ECS3" BCRYPT_ECDSA_PUBLIC_P384_MAGIC
+	winECS5Magic = 0x35534345 // "ECS5" BCRYPT_ECDSA_PUBLIC_P521_MAGIC
+
+	winECK1Magic = 0x314B4345 // "ECK1" BCRYPT_ECDH_PUBLIC_P256_MAGIC
+	winECK3Magic = 0x334B4345 // "ECK3" BCRYPT_ECDH_PUBLIC_P384_MAGIC
+	winECK5Magic = 0x354B4345 // "ECK5" BCRYPT_ECDH_PUBLIC_P521_MAGIC
+
+	winCryptENotFound = windows.CRYPT_E_NOT_FOUND
+
+	providerMSSoftware = "Microsoft Software Key Storage Provider"
+)
+
+var (
+	winBCryptRSAPublicBlob = winWide("RSAPUBLICBLOB")
+	winBCryptECCPublicBlob = winWide("ECCPUBLICBLOB")
+
+	winNCryptAlgorithmGroupProperty = winWide("Algorithm Group") // NCRYPT_ALGORITHM_GROUP_PROPERTY
+	winNCryptUniqueNameProperty     = winWide("Unique Name")     // NCRYPT_UNIQUE_NAME_PROPERTY
+	winNCryptECCCurveNameProperty   = winWide("ECCCurveName")    // NCRYPT_ECC_CURVE_NAME_PROPERTY
+
+	winCurveIDs = map[uint32]elliptic.Curve{
+		winECS1Magic: elliptic.P256(), // BCRYPT_ECDSA_PUBLIC_P256_MAGIC
+		winECS3Magic: elliptic.P384(), // BCRYPT_ECDSA_PUBLIC_P384_MAGIC
+		winECS5Magic: elliptic.P521(), // BCRYPT_ECDSA_PUBLIC_P521_MAGIC
+		winECK1Magic: elliptic.P256(), // BCRYPT_ECDH_PUBLIC_P256_MAGIC
+		winECK3Magic: elliptic.P384(), // BCRYPT_ECDH_PUBLIC_P384_MAGIC
+		winECK5Magic: elliptic.P521(), // BCRYPT_ECDH_PUBLIC_P521_MAGIC
+	}
+
+	winCurveNames = map[string]elliptic.Curve{
+		"nistP256": elliptic.P256(), // BCRYPT_ECC_CURVE_NISTP256
+		"nistP384": elliptic.P384(), // BCRYPT_ECC_CURVE_NISTP384
+		"nistP521": elliptic.P521(), // BCRYPT_ECC_CURVE_NISTP521
+	}
+
+	winAlgIDs = map[crypto.Hash]*uint16{
+		crypto.SHA1:   winWide("SHA1"),   // BCRYPT_SHA1_ALGORITHM
+		crypto.SHA256: winWide("SHA256"), // BCRYPT_SHA256_ALGORITHM
+		crypto.SHA384: winWide("SHA384"), // BCRYPT_SHA384_ALGORITHM
+		crypto.SHA512: winWide("SHA512"), // BCRYPT_SHA512_ALGORITHM
+	}
+
+	// MY is well-known system store on Windows that holds personal certificates. Read
+	// More about the CA locations here:
+	// https://learn.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/wcf/certificate-of-clientcertificate-element?redirectedfrom=MSDN
+	// https://superuser.com/questions/217719/what-are-the-windows-system-certificate-stores
+	// https://docs.microsoft.com/en-us/windows/win32/seccrypto/certificate-stores
+	// https://learn.microsoft.com/en-us/windows/win32/seccrypto/system-store-locations
+	// https://stackoverflow.com/questions/63286085/which-x509-storename-refers-to-the-certificates-stored-beneath-trusted-root-cert#:~:text=4-,StoreName.,is%20%22Intermediate%20Certification%20Authorities%22.
+	winMyStore             = winWide("MY")
+	winIntermediateCAStore = winWide("CA")
+	winRootStore           = winWide("Root")
+	winAuthRootStore       = winWide("AuthRoot")
+
+	// These DLLs must be available on all Windows hosts
+	winCrypt32 = windows.NewLazySystemDLL("crypt32.dll")
+	winNCrypt  = windows.NewLazySystemDLL("ncrypt.dll")
+
+	winCertFindCertificateInStore        = winCrypt32.NewProc("CertFindCertificateInStore")
+	winCertVerifyTimeValidity            = winCrypt32.NewProc("CertVerifyTimeValidity")
+	winCryptAcquireCertificatePrivateKey = winCrypt32.NewProc("CryptAcquireCertificatePrivateKey")
+	winNCryptExportKey                   = winNCrypt.NewProc("NCryptExportKey")
+	winNCryptOpenStorageProvider         = winNCrypt.NewProc("NCryptOpenStorageProvider")
+	winNCryptGetProperty                 = winNCrypt.NewProc("NCryptGetProperty")
+	winNCryptSignHash                    = winNCrypt.NewProc("NCryptSignHash")
+
+	winFnGetProperty = winGetProperty
+)
+
+func init() {
+	for _, d := range []*windows.LazyDLL{
+		winCrypt32, winNCrypt,
+	} {
+		if err := d.Load(); err != nil {
+			panic(err)
+		}
+	}
+	for _, p := range []*windows.LazyProc{
+		winCertFindCertificateInStore, winCryptAcquireCertificatePrivateKey,
+		winNCryptExportKey, winNCryptOpenStorageProvider,
+		winNCryptGetProperty, winNCryptSignHash,
+	} {
+		if err := p.Find(); err != nil {
+			panic(err)
+		}
+	}
+}
+
+type winPKCS1PaddingInfo struct {
+	pszAlgID *uint16
+}
+
+type winPSSPaddingInfo struct {
+	pszAlgID *uint16
+	cbSalt   uint32
+}
+
+// createCACertsPool generates a CertPool from the Windows certificate store,
+// adding all matching certificates from the caCertsMatch array to the pool.
+// All matching certificates (vs first) are added to the pool based on a user
+// request. If no certificates are found an error is returned.
+func createCACertsPool(cs *winCertStore, storeType uint32, caCertsMatch []string, skipInvalid bool) (*x509.CertPool, error) {
+	var errs []error
+	caPool := x509.NewCertPool()
+	for _, s := range caCertsMatch {
+		lfs, err := cs.caCertsBySubjectMatch(s, storeType, skipInvalid)
+		if err != nil {
+			errs = append(errs, err)
+		} else {
+			for _, lf := range lfs {
+				caPool.AddCert(lf)
+			}
+		}
+	}
+	// If every lookup failed return the errors.
+	if len(errs) == len(caCertsMatch) {
+		return nil, fmt.Errorf("unable to match any CA certificate: %v", errs)
+	}
+	return caPool, nil
+}
+
+// TLSConfig fulfills the same function as reading cert and key pair from
+// pem files but sources the Windows certificate store instead. The
+// certMatchBy and certMatch fields search the "MY" certificate location
+// for the first certificate that matches the certMatch field. The
+// caCertsMatch field is used to search the Trusted Root, Third Party Root,
+// and Intermediate Certificate Authority locations for certificates with
+// Subjects matching the provided strings. If a match is found, the
+// certificate is added to the pool that is used to verify the certificate
+// chain.
+func TLSConfig(certStore StoreType, certMatchBy MatchByType, certMatch string, caCertsMatch []string, skipInvalid bool, config *tls.Config) error {
+	var (
+		leaf     *x509.Certificate
+		leafCtx  *windows.CertContext
+		pk       *winKey
+		vOpts    = x509.VerifyOptions{}
+		chains   [][]*x509.Certificate
+		chain    []*x509.Certificate
+		rawChain [][]byte
+	)
+
+	// By StoreType, open a store
+	if certStore == windowsCurrentUser || certStore == windowsLocalMachine {
+		var scope uint32
+		cs, err := winOpenCertStore(providerMSSoftware)
+		if err != nil || cs == nil {
+			return err
+		}
+		if certStore == windowsCurrentUser {
+			scope = winCertStoreCurrentUser
+		}
+		if certStore == windowsLocalMachine {
+			scope = winCertStoreLocalMachine
+		}
+
+		// certByIssuer or certBySubject
+		if certMatchBy == matchBySubject || certMatchBy == MATCHBYEMPTY {
+			leaf, leafCtx, err = cs.certBySubject(certMatch, scope, skipInvalid)
+		} else if certMatchBy == matchByIssuer {
+			leaf, leafCtx, err = cs.certByIssuer(certMatch, scope, skipInvalid)
+		} else if certMatchBy == matchByThumbprint {
+			leaf, leafCtx, err = cs.certByThumbprint(certMatch, scope, skipInvalid)
+		} else {
+			return ErrBadMatchByType
+		}
+		if err != nil {
+			// pass through error from cert search
+			return err
+		}
+		if leaf == nil || leafCtx == nil {
+			return ErrFailedCertSearch
+		}
+		pk, err = cs.certKey(leafCtx)
+		if err != nil {
+			return err
+		}
+		if pk == nil {
+			return ErrNoPrivateKeyStoreRef
+		}
+		// Look for CA Certificates
+		if len(caCertsMatch) != 0 {
+			caPool, err := createCACertsPool(cs, scope, caCertsMatch, skipInvalid)
+			if err != nil {
+				return err
+			}
+			config.ClientCAs = caPool
+		}
+	} else {
+		return ErrBadCertStore
+	}
+
+	// Get intermediates in the cert store for the found leaf IFF there is a full chain of trust in the store
+	// otherwise just use leaf as the final chain.
+	//
+	// Using std lib Verify as a reliable way to get valid chains out of the win store for the leaf; however,
+	// using empty options since server TLS stanza could be TLS role as server identity or client identity.
+	chains, err := leaf.Verify(vOpts)
+	if err != nil || len(chains) == 0 {
+		chains = append(chains, []*x509.Certificate{leaf})
+	}
+
+	// We have at least one verified chain so pop the first chain and remove the self-signed CA cert (if present)
+	// from the end of the chain
+	chain = chains[0]
+	if len(chain) > 1 {
+		chain = chain[:len(chain)-1]
+	}
+
+	// For tls.Certificate.Certificate need a [][]byte from []*x509.Certificate
+	// Approximate capacity for efficiency
+	rawChain = make([][]byte, 0, len(chain))
+	for _, link := range chain {
+		rawChain = append(rawChain, link.Raw)
+	}
+
+	tlsCert := tls.Certificate{
+		Certificate: rawChain,
+		PrivateKey:  pk,
+		Leaf:        leaf,
+	}
+	config.Certificates = []tls.Certificate{tlsCert}
+
+	// note: pk is a windows pointer (not freed by Go) but needs to live the life of the server for Signing.
+	// The cert context (leafCtx) windows pointer must not be freed underneath the pk so also life of the server.
+	return nil
+}
+
+// winWide returns a pointer to uint16 representing the equivalent
+// to a Windows LPCWSTR.
+func winWide(s string) *uint16 {
+	w := utf16.Encode([]rune(s))
+	w = append(w, 0)
+	return &w[0]
+}
+
+// winOpenProvider gets a provider handle for subsequent calls
+func winOpenProvider(provider string) (uintptr, error) {
+	var hProv uintptr
+	pname := winWide(provider)
+	// Open the provider, the last parameter is not used
+	r, _, err := winNCryptOpenStorageProvider.Call(uintptr(unsafe.Pointer(&hProv)), uintptr(unsafe.Pointer(pname)), 0)
+	if r == 0 {
+		return hProv, nil
+	}
+	return hProv, fmt.Errorf("NCryptOpenStorageProvider returned %X: %v", r, err)
+}
+
+// winFindCert wraps the CertFindCertificateInStore library call. Note that any cert context passed
+// into prev will be freed. If no certificate was found, nil will be returned.
+func winFindCert(store windows.Handle, enc, findFlags, findType uint32, para *uint16, prev *windows.CertContext) (*windows.CertContext, error) {
+	h, _, err := winCertFindCertificateInStore.Call(
+		uintptr(store),
+		uintptr(enc),
+		uintptr(findFlags),
+		uintptr(findType),
+		uintptr(unsafe.Pointer(para)),
+		uintptr(unsafe.Pointer(prev)),
+	)
+	if h == 0 {
+		// Actual error, or simply not found?
+		if errno, ok := err.(syscall.Errno); ok && errno == syscall.Errno(winCryptENotFound) {
+			return nil, ErrFailedCertSearch
+		}
+		return nil, ErrFailedCertSearch
+	}
+	// nolint:govet
+	return (*windows.CertContext)(unsafe.Pointer(h)), nil
+}
+
+// winVerifyCertValid wraps the CertVerifyTimeValidity and simply returns true if the certificate is valid
+func winVerifyCertValid(timeToVerify *windows.Filetime, certInfo *windows.CertInfo) bool {
+	// this function does not document returning errors / setting lasterror
+	r, _, _ := winCertVerifyTimeValidity.Call(
+		uintptr(unsafe.Pointer(timeToVerify)),
+		uintptr(unsafe.Pointer(certInfo)),
+	)
+	return r == 0
+}
+
+// winCertStore is a store implementation for the Windows Certificate Store
+type winCertStore struct {
+	Prov     uintptr
+	ProvName string
+	stores   map[string]*winStoreHandle
+	mu       sync.Mutex
+}
+
+// winOpenCertStore creates a winCertStore
+func winOpenCertStore(provider string) (*winCertStore, error) {
+	cngProv, err := winOpenProvider(provider)
+	if err != nil {
+		// pass through error from winOpenProvider
+		return nil, err
+	}
+
+	wcs := &winCertStore{
+		Prov:     cngProv,
+		ProvName: provider,
+		stores:   make(map[string]*winStoreHandle),
+	}
+
+	return wcs, nil
+}
+
+// winCertContextToX509 creates an x509.Certificate from a Windows cert context.
+func winCertContextToX509(ctx *windows.CertContext) (*x509.Certificate, error) {
+	var der []byte
+	slice := (*reflect.SliceHeader)(unsafe.Pointer(&der))
+	slice.Data = uintptr(unsafe.Pointer(ctx.EncodedCert))
+	slice.Len = int(ctx.Length)
+	slice.Cap = int(ctx.Length)
+	return x509.ParseCertificate(der)
+}
+
+// certByIssuer matches and returns the first certificate found by passed issuer.
+// CertContext pointer returned allows subsequent key operations like Sign. Caller specifies
+// current user's personal certs or local machine's personal certs using storeType.
+// See CERT_FIND_ISSUER_STR description at https://learn.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-certfindcertificateinstore
+func (w *winCertStore) certByIssuer(issuer string, storeType uint32, skipInvalid bool) (*x509.Certificate, *windows.CertContext, error) {
+	return w.certSearch(winFindIssuerStr, issuer, winMyStore, storeType, skipInvalid)
+}
+
+// certBySubject matches and returns the first certificate found by passed subject field.
+// CertContext pointer returned allows subsequent key operations like Sign. Caller specifies
+// current user's personal certs or local machine's personal certs using storeType.
+// See CERT_FIND_SUBJECT_STR description at https://learn.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-certfindcertificateinstore
+func (w *winCertStore) certBySubject(subject string, storeType uint32, skipInvalid bool) (*x509.Certificate, *windows.CertContext, error) {
+	return w.certSearch(winFindSubjectStr, subject, winMyStore, storeType, skipInvalid)
+}
+
+// certByThumbprint matches and returns the first certificate found by passed SHA1 thumbprint.
+// CertContext pointer returned allows subsequent key operations like Sign. Caller specifies
+// current user's personal certs or local machine's personal certs using storeType.
+// See CERT_FIND_SUBJECT_STR description at https://learn.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-certfindcertificateinstore
+func (w *winCertStore) certByThumbprint(hash string, storeType uint32, skipInvalid bool) (*x509.Certificate, *windows.CertContext, error) {
+	return w.certSearch(winFindHashStr, hash, winMyStore, storeType, skipInvalid)
+}
+
+// caCertsBySubjectMatch matches and returns all matching certificates of the subject field.
+//
+// The following locations are searched:
+// 1) Root (Trusted Root Certification Authorities)
+// 2) AuthRoot (Third-Party Root Certification Authorities)
+// 3) CA (Intermediate Certification Authorities)
+//
+// Caller specifies current user's personal certs or local machine's personal certs using storeType.
+// See CERT_FIND_SUBJECT_STR description at https://learn.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-certfindcertificateinstore
+func (w *winCertStore) caCertsBySubjectMatch(subject string, storeType uint32, skipInvalid bool) ([]*x509.Certificate, error) {
+	var (
+		leaf            *x509.Certificate
+		searchLocations = [3]*uint16{winRootStore, winAuthRootStore, winIntermediateCAStore}
+		rv              []*x509.Certificate
+	)
+	// surprisingly, an empty string returns a result. We'll treat this as an error.
+	if subject == "" {
+		return nil, ErrBadCaCertMatchField
+	}
+	for _, sr := range searchLocations {
+		var err error
+		if leaf, _, err = w.certSearch(winFindSubjectStr, subject, sr, storeType, skipInvalid); err == nil {
+			rv = append(rv, leaf)
+		} else {
+			// Ignore the failed search from a single location. Errors we catch include
+			// ErrFailedX509Extract (resulting from a malformed certificate) and errors
+			// around invalid attributes, unsupported algorithms, etc. These are corner
+			// cases as certificates with these errors shouldn't have been allowed
+			// to be added to the store in the first place.
+			if err != ErrFailedCertSearch {
+				return nil, err
+			}
+		}
+	}
+	// Not found anywhere
+	if len(rv) == 0 {
+		return nil, ErrFailedCertSearch
+	}
+	return rv, nil
+}
+
+// certSearch is a helper function to lookup certificates based on search type and match value.
+// store is used to specify which store to perform the lookup in (system or user).
+func (w *winCertStore) certSearch(searchType uint32, matchValue string, searchRoot *uint16, store uint32, skipInvalid bool) (*x509.Certificate, *windows.CertContext, error) {
+	// store handle to "MY" store
+	h, err := w.storeHandle(store, searchRoot)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	var prev *windows.CertContext
+	var cert *x509.Certificate
+
+	i, err := windows.UTF16PtrFromString(matchValue)
+	if err != nil {
+		return nil, nil, ErrFailedCertSearch
+	}
+
+	// pass 0 as the third parameter because it is not used
+	// https://msdn.microsoft.com/en-us/library/windows/desktop/aa376064(v=vs.85).aspx
+
+	for {
+		nc, err := winFindCert(h, winEncodingX509ASN|winEncodingPKCS7, 0, searchType, i, prev)
+		if err != nil {
+			return nil, nil, err
+		}
+		if nc != nil {
+			// certificate found
+			prev = nc
+
+			var now *windows.Filetime
+			if skipInvalid && !winVerifyCertValid(now, nc.CertInfo) {
+				continue
+			}
+
+			// Extract the DER-encoded certificate from the cert context
+			xc, err := winCertContextToX509(nc)
+			if err == nil {
+				cert = xc
+				break
+			} else {
+				return nil, nil, ErrFailedX509Extract
+			}
+		} else {
+			return nil, nil, ErrFailedCertSearch
+		}
+	}
+
+	if cert == nil {
+		return nil, nil, ErrFailedX509Extract
+	}
+
+	return cert, prev, nil
+}
+
+type winStoreHandle struct {
+	handle *windows.Handle
+}
+
+func winNewStoreHandle(provider uint32, store *uint16) (*winStoreHandle, error) {
+	var s winStoreHandle
+	if s.handle != nil {
+		return &s, nil
+	}
+	st, err := windows.CertOpenStore(
+		winCertStoreProvSystem,
+		0,
+		0,
+		provider|winCertStoreReadOnly,
+		uintptr(unsafe.Pointer(store)))
+	if err != nil {
+		return nil, ErrBadCryptoStoreProvider
+	}
+	s.handle = &st
+	return &s, nil
+}
+
+// winKey implements crypto.Signer and crypto.Decrypter for key based operations.
+type winKey struct {
+	handle         uintptr
+	pub            crypto.PublicKey
+	Container      string
+	AlgorithmGroup string
+}
+
+// Public exports a public key to implement crypto.Signer
+func (k winKey) Public() crypto.PublicKey {
+	return k.pub
+}
+
+// Sign returns the signature of a hash to implement crypto.Signer
+func (k winKey) Sign(_ io.Reader, digest []byte, opts crypto.SignerOpts) ([]byte, error) {
+	switch k.AlgorithmGroup {
+	case "ECDSA", "ECDH":
+		return winSignECDSA(k.handle, digest)
+	case "RSA":
+		hf := opts.HashFunc()
+		algID, ok := winAlgIDs[hf]
+		if !ok {
+			return nil, ErrBadRSAHashAlgorithm
+		}
+		switch opts.(type) {
+		case *rsa.PSSOptions:
+			return winSignRSAPSSPadding(k.handle, digest, algID)
+		default:
+			return winSignRSAPKCS1Padding(k.handle, digest, algID)
+		}
+	default:
+		return nil, ErrBadSigningAlgorithm
+	}
+}
+
+func winSignECDSA(kh uintptr, digest []byte) ([]byte, error) {
+	var size uint32
+	// Obtain the size of the signature
+	r, _, _ := winNCryptSignHash.Call(
+		kh,
+		0,
+		uintptr(unsafe.Pointer(&digest[0])),
+		uintptr(len(digest)),
+		0,
+		0,
+		uintptr(unsafe.Pointer(&size)),
+		0)
+	if r != 0 {
+		return nil, ErrStoreECDSASigningError
+	}
+
+	// Obtain the signature data
+	buf := make([]byte, size)
+	r, _, _ = winNCryptSignHash.Call(
+		kh,
+		0,
+		uintptr(unsafe.Pointer(&digest[0])),
+		uintptr(len(digest)),
+		uintptr(unsafe.Pointer(&buf[0])),
+		uintptr(size),
+		uintptr(unsafe.Pointer(&size)),
+		0)
+	if r != 0 {
+		return nil, ErrStoreECDSASigningError
+	}
+	if len(buf) != int(size) {
+		return nil, ErrStoreECDSASigningError
+	}
+
+	return winPackECDSASigValue(bytes.NewReader(buf[:size]), int(size/2))
+}
+
+func winPackECDSASigValue(r io.Reader, digestLength int) ([]byte, error) {
+	sigR := make([]byte, digestLength)
+	if _, err := io.ReadFull(r, sigR); err != nil {
+		return nil, ErrStoreECDSASigningError
+	}
+
+	sigS := make([]byte, digestLength)
+	if _, err := io.ReadFull(r, sigS); err != nil {
+		return nil, ErrStoreECDSASigningError
+	}
+
+	var b cryptobyte.Builder
+	b.AddASN1(asn1.SEQUENCE, func(b *cryptobyte.Builder) {
+		b.AddASN1BigInt(new(big.Int).SetBytes(sigR))
+		b.AddASN1BigInt(new(big.Int).SetBytes(sigS))
+	})
+	return b.Bytes()
+}
+
+func winSignRSAPKCS1Padding(kh uintptr, digest []byte, algID *uint16) ([]byte, error) {
+	// PKCS#1 v1.5 padding for some TLS 1.2
+	padInfo := winPKCS1PaddingInfo{pszAlgID: algID}
+	var size uint32
+	// Obtain the size of the signature
+	r, _, _ := winNCryptSignHash.Call(
+		kh,
+		uintptr(unsafe.Pointer(&padInfo)),
+		uintptr(unsafe.Pointer(&digest[0])),
+		uintptr(len(digest)),
+		0,
+		0,
+		uintptr(unsafe.Pointer(&size)),
+		winBCryptPadPKCS1)
+	if r != 0 {
+		return nil, ErrStoreRSASigningError
+	}
+
+	// Obtain the signature data
+	sig := make([]byte, size)
+	r, _, _ = winNCryptSignHash.Call(
+		kh,
+		uintptr(unsafe.Pointer(&padInfo)),
+		uintptr(unsafe.Pointer(&digest[0])),
+		uintptr(len(digest)),
+		uintptr(unsafe.Pointer(&sig[0])),
+		uintptr(size),
+		uintptr(unsafe.Pointer(&size)),
+		winBCryptPadPKCS1)
+	if r != 0 {
+		return nil, ErrStoreRSASigningError
+	}
+
+	return sig[:size], nil
+}
+
+func winSignRSAPSSPadding(kh uintptr, digest []byte, algID *uint16) ([]byte, error) {
+	// PSS padding for TLS 1.3 and some TLS 1.2
+	padInfo := winPSSPaddingInfo{pszAlgID: algID, cbSalt: winBCryptPadPSSSalt}
+
+	var size uint32
+	// Obtain the size of the signature
+	r, _, _ := winNCryptSignHash.Call(
+		kh,
+		uintptr(unsafe.Pointer(&padInfo)),
+		uintptr(unsafe.Pointer(&digest[0])),
+		uintptr(len(digest)),
+		0,
+		0,
+		uintptr(unsafe.Pointer(&size)),
+		winBCryptPadPSS)
+	if r != 0 {
+		return nil, ErrStoreRSASigningError
+	}
+
+	// Obtain the signature data
+	sig := make([]byte, size)
+	r, _, _ = winNCryptSignHash.Call(
+		kh,
+		uintptr(unsafe.Pointer(&padInfo)),
+		uintptr(unsafe.Pointer(&digest[0])),
+		uintptr(len(digest)),
+		uintptr(unsafe.Pointer(&sig[0])),
+		uintptr(size),
+		uintptr(unsafe.Pointer(&size)),
+		winBCryptPadPSS)
+	if r != 0 {
+		return nil, ErrStoreRSASigningError
+	}
+
+	return sig[:size], nil
+}
+
+// certKey wraps CryptAcquireCertificatePrivateKey. It obtains the CNG private
+// key of a known certificate and returns a pointer to a winKey which implements
+// both crypto.Signer. When a nil cert context is passed
+// a nil key is intentionally returned, to model the expected behavior of a
+// non-existent cert having no private key.
+// https://docs.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-cryptacquirecertificateprivatekey
+func (w *winCertStore) certKey(cert *windows.CertContext) (*winKey, error) {
+	// Return early if a nil cert was passed.
+	if cert == nil {
+		return nil, nil
+	}
+	var (
+		kh       uintptr
+		spec     uint32
+		mustFree int
+	)
+	r, _, _ := winCryptAcquireCertificatePrivateKey.Call(
+		uintptr(unsafe.Pointer(cert)),
+		winAcquireCached|winAcquireSilent|winAcquireOnlyNCryptKey,
+		0, // Reserved, must be null.
+		uintptr(unsafe.Pointer(&kh)),
+		uintptr(unsafe.Pointer(&spec)),
+		uintptr(unsafe.Pointer(&mustFree)),
+	)
+	// If the function succeeds, the return value is nonzero (TRUE).
+	if r == 0 {
+		return nil, ErrNoPrivateKeyStoreRef
+	}
+	if mustFree != 0 {
+		return nil, ErrNoPrivateKeyStoreRef
+	}
+	if spec != winNcryptKeySpec {
+		return nil, ErrNoPrivateKeyStoreRef
+	}
+
+	return winKeyMetadata(kh)
+}
+
+func winKeyMetadata(kh uintptr) (*winKey, error) {
+	// uc is used to populate the unique container name attribute of the private key
+	uc, err := winGetPropertyStr(kh, winNCryptUniqueNameProperty)
+	if err != nil {
+		// unable to determine key unique name
+		return nil, ErrExtractingPrivateKeyMetadata
+	}
+
+	alg, err := winGetPropertyStr(kh, winNCryptAlgorithmGroupProperty)
+	if err != nil {
+		// unable to determine key algorithm
+		return nil, ErrExtractingPrivateKeyMetadata
+	}
+
+	var pub crypto.PublicKey
+
+	switch alg {
+	case "ECDSA", "ECDH":
+		buf, err := winExport(kh, winBCryptECCPublicBlob)
+		if err != nil {
+			// failed to export ECC public key
+			return nil, ErrExtractingECCPublicKey
+		}
+		pub, err = unmarshalECC(buf, kh)
+		if err != nil {
+			return nil, ErrExtractingECCPublicKey
+		}
+	case "RSA":
+		buf, err := winExport(kh, winBCryptRSAPublicBlob)
+		if err != nil {
+			return nil, ErrExtractingRSAPublicKey
+		}
+		pub, err = winUnmarshalRSA(buf)
+		if err != nil {
+			return nil, ErrExtractingRSAPublicKey
+		}
+	default:
+		return nil, ErrBadPublicKeyAlgorithm
+	}
+
+	return &winKey{handle: kh, pub: pub, Container: uc, AlgorithmGroup: alg}, nil
+}
+
+func winGetProperty(kh uintptr, property *uint16) ([]byte, error) {
+	var strSize uint32
+	r, _, _ := winNCryptGetProperty.Call(
+		kh,
+		uintptr(unsafe.Pointer(property)),
+		0,
+		0,
+		uintptr(unsafe.Pointer(&strSize)),
+		0,
+		0)
+	if r != 0 {
+		return nil, ErrExtractPropertyFromKey
+	}
+
+	buf := make([]byte, strSize)
+	r, _, _ = winNCryptGetProperty.Call(
+		kh,
+		uintptr(unsafe.Pointer(property)),
+		uintptr(unsafe.Pointer(&buf[0])),
+		uintptr(strSize),
+		uintptr(unsafe.Pointer(&strSize)),
+		0,
+		0)
+	if r != 0 {
+		return nil, ErrExtractPropertyFromKey
+	}
+
+	return buf, nil
+}
+
+func winGetPropertyStr(kh uintptr, property *uint16) (string, error) {
+	buf, err := winFnGetProperty(kh, property)
+	if err != nil {
+		return "", ErrExtractPropertyFromKey
+	}
+	uc := bytes.ReplaceAll(buf, []byte{0x00}, []byte(""))
+	return string(uc), nil
+}
+
+func winExport(kh uintptr, blobType *uint16) ([]byte, error) {
+	var size uint32
+	// When obtaining the size of a public key, most parameters are not required
+	r, _, _ := winNCryptExportKey.Call(
+		kh,
+		0,
+		uintptr(unsafe.Pointer(blobType)),
+		0,
+		0,
+		0,
+		uintptr(unsafe.Pointer(&size)),
+		0)
+	if r != 0 {
+		return nil, ErrExtractingPublicKey
+	}
+
+	// Place the exported key in buf now that we know the size required
+	buf := make([]byte, size)
+	r, _, _ = winNCryptExportKey.Call(
+		kh,
+		0,
+		uintptr(unsafe.Pointer(blobType)),
+		0,
+		uintptr(unsafe.Pointer(&buf[0])),
+		uintptr(size),
+		uintptr(unsafe.Pointer(&size)),
+		0)
+	if r != 0 {
+		return nil, ErrExtractingPublicKey
+	}
+	return buf, nil
+}
+
+func unmarshalECC(buf []byte, kh uintptr) (*ecdsa.PublicKey, error) {
+	// BCRYPT_ECCKEY_BLOB from bcrypt.h
+	header := struct {
+		Magic uint32
+		Key   uint32
+	}{}
+
+	r := bytes.NewReader(buf)
+	if err := binary.Read(r, binary.LittleEndian, &header); err != nil {
+		return nil, ErrExtractingECCPublicKey
+	}
+
+	curve, ok := winCurveIDs[header.Magic]
+	if !ok {
+		// Fix for b/185945636, where despite specifying the curve, nCrypt returns
+		// an incorrect response with BCRYPT_ECDSA_PUBLIC_GENERIC_MAGIC.
+		var err error
+		curve, err = winCurveName(kh)
+		if err != nil {
+			// unsupported header magic or cannot match the curve by name
+			return nil, err
+		}
+	}
+
+	keyX := make([]byte, header.Key)
+	if n, err := r.Read(keyX); n != int(header.Key) || err != nil {
+		// failed to read key X
+		return nil, ErrExtractingECCPublicKey
+	}
+
+	keyY := make([]byte, header.Key)
+	if n, err := r.Read(keyY); n != int(header.Key) || err != nil {
+		// failed to read key Y
+		return nil, ErrExtractingECCPublicKey
+	}
+
+	pub := &ecdsa.PublicKey{
+		Curve: curve,
+		X:     new(big.Int).SetBytes(keyX),
+		Y:     new(big.Int).SetBytes(keyY),
+	}
+	return pub, nil
+}
+
+// winCurveName reads the curve name property and returns the corresponding curve.
+func winCurveName(kh uintptr) (elliptic.Curve, error) {
+	cn, err := winGetPropertyStr(kh, winNCryptECCCurveNameProperty)
+	if err != nil {
+		// unable to determine the curve property name
+		return nil, ErrExtractPropertyFromKey
+	}
+	curve, ok := winCurveNames[cn]
+	if !ok {
+		// unknown curve name
+		return nil, ErrBadECCCurveName
+	}
+	return curve, nil
+}
+
+func winUnmarshalRSA(buf []byte) (*rsa.PublicKey, error) {
+	// BCRYPT_RSA_BLOB from bcrypt.h
+	header := struct {
+		Magic         uint32
+		BitLength     uint32
+		PublicExpSize uint32
+		ModulusSize   uint32
+		UnusedPrime1  uint32
+		UnusedPrime2  uint32
+	}{}
+
+	r := bytes.NewReader(buf)
+	if err := binary.Read(r, binary.LittleEndian, &header); err != nil {
+		return nil, ErrExtractingRSAPublicKey
+	}
+
+	if header.Magic != winRSA1Magic {
+		// invalid header magic
+		return nil, ErrExtractingRSAPublicKey
+	}
+
+	if header.PublicExpSize > 8 {
+		// unsupported public exponent size
+		return nil, ErrExtractingRSAPublicKey
+	}
+
+	exp := make([]byte, 8)
+	if n, err := r.Read(exp[8-header.PublicExpSize:]); n != int(header.PublicExpSize) || err != nil {
+		// failed to read public exponent
+		return nil, ErrExtractingRSAPublicKey
+	}
+
+	mod := make([]byte, header.ModulusSize)
+	if n, err := r.Read(mod); n != int(header.ModulusSize) || err != nil {
+		// failed to read modulus
+		return nil, ErrExtractingRSAPublicKey
+	}
+
+	pub := &rsa.PublicKey{
+		N: new(big.Int).SetBytes(mod),
+		E: int(binary.BigEndian.Uint64(exp)),
+	}
+	return pub, nil
+}
+
+// storeHandle returns a handle to a given cert store, opening the handle as needed.
+func (w *winCertStore) storeHandle(provider uint32, store *uint16) (windows.Handle, error) {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+
+	key := fmt.Sprintf("%d%s", provider, windows.UTF16PtrToString(store))
+	var err error
+	if w.stores[key] == nil {
+		w.stores[key], err = winNewStoreHandle(provider, store)
+		if err != nil {
+			return 0, ErrBadCryptoStoreProvider
+		}
+	}
+	return *w.stores[key].handle, nil
+}
+
+// Verify interface conformance.
+var _ credential = &winKey{}

server/certstore/errors.go

@@ -0,0 +1,79 @@
+package certstore
+
+import (
+	"errors"
+)
+
+var (
+	// ErrBadCryptoStoreProvider represents inablity to establish link with a certificate store
+	ErrBadCryptoStoreProvider = errors.New("unable to open certificate store or store not available")
+
+	// ErrBadRSAHashAlgorithm represents a bad or unsupported RSA hash algorithm
+	ErrBadRSAHashAlgorithm = errors.New("unsupported RSA hash algorithm")
+
+	// ErrBadSigningAlgorithm represents a bad or unsupported signing algorithm
+	ErrBadSigningAlgorithm = errors.New("unsupported signing algorithm")
+
+	// ErrStoreRSASigningError represents an error returned from store during RSA signature
+	ErrStoreRSASigningError = errors.New("unable to obtain RSA signature from store")
+
+	// ErrStoreECDSASigningError represents an error returned from store during ECDSA signature
+	ErrStoreECDSASigningError = errors.New("unable to obtain ECDSA signature from store")
+
+	// ErrNoPrivateKeyStoreRef represents an error getting a handle to a private key in store
+	ErrNoPrivateKeyStoreRef = errors.New("unable to obtain private key handle from store")
+
+	// ErrExtractingPrivateKeyMetadata represents a family of errors extracting metadata about the private key in store
+	ErrExtractingPrivateKeyMetadata = errors.New("unable to extract private key metadata")
+
+	// ErrExtractingECCPublicKey represents an error exporting ECC-type public key from store
+	ErrExtractingECCPublicKey = errors.New("unable to extract ECC public key from store")
+
+	// ErrExtractingRSAPublicKey represents an error exporting RSA-type public key from store
+	ErrExtractingRSAPublicKey = errors.New("unable to extract RSA public key from store")
+
+	// ErrExtractingPublicKey represents a general error exporting public key from store
+	ErrExtractingPublicKey = errors.New("unable to extract public key from store")
+
+	// ErrBadPublicKeyAlgorithm represents a bad or unsupported public key algorithm
+	ErrBadPublicKeyAlgorithm = errors.New("unsupported public key algorithm")
+
+	// ErrExtractPropertyFromKey represents a general failure to extract a metadata property field
+	ErrExtractPropertyFromKey = errors.New("unable to extract property from key")
+
+	// ErrBadECCCurveName represents an ECC signature curve name that is bad or unsupported
+	ErrBadECCCurveName = errors.New("unsupported ECC curve name")
+
+	// ErrFailedCertSearch represents not able to find certificate in store
+	ErrFailedCertSearch = errors.New("unable to find certificate in store")
+
+	// ErrFailedX509Extract represents not being able to extract x509 certificate from found cert in store
+	ErrFailedX509Extract = errors.New("unable to extract x509 from certificate")
+
+	// ErrBadMatchByType represents unknown CERT_MATCH_BY passed
+	ErrBadMatchByType = errors.New("cert match by type not implemented")
+
+	// ErrBadCertStore represents unknown CERT_STORE passed
+	ErrBadCertStore = errors.New("cert store type not implemented")
+
+	// ErrConflictCertFileAndStore represents ambiguous configuration of both file and store
+	ErrConflictCertFileAndStore = errors.New("'cert_file' and 'cert_store' may not both be configured")
+
+	// ErrBadCertStoreField represents malformed cert_store option
+	ErrBadCertStoreField = errors.New("expected 'cert_store' to be a valid non-empty string")
+
+	// ErrBadCertMatchByField represents malformed cert_match_by option
+	ErrBadCertMatchByField = errors.New("expected 'cert_match_by' to be a valid non-empty string")
+
+	// ErrBadCertMatchField represents malformed cert_match option
+	ErrBadCertMatchField = errors.New("expected 'cert_match' to be a valid non-empty string")
+
+	// ErrBadCaCertMatchField represents malformed cert_match option
+	ErrBadCaCertMatchField = errors.New("expected 'ca_certs_match' to be a valid non-empty string array")
+
+	// ErrBadCertMatchSkipInvalidField represents malformed cert_match_skip_invalid option
+	ErrBadCertMatchSkipInvalidField = errors.New("expected 'cert_match_skip_invalid' to be a boolean")
+
+	// ErrOSNotCompatCertStore represents cert_store passed that exists but is not valid on current OS
+	ErrOSNotCompatCertStore = errors.New("cert_store not compatible with current operating system")
+)

server/certstore_windows_test.go

@@ -0,0 +1,350 @@
+// Copyright 2022-2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build windows
+
+package server
+
+import (
+	"crypto/tls"
+	"fmt"
+	"net/url"
+	"os"
+	"os/exec"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/nats-io/nats.go"
+)
+
+func runPowershellScript(scriptFile string, args []string) error {
+	psExec, _ := exec.LookPath("powershell.exe")
+
+	execArgs := []string{psExec, "-command", fmt.Sprintf("& '%s'", scriptFile)}
+	if len(args) > 0 {
+		execArgs = append(execArgs, args...)
+	}
+
+	cmdImport := &exec.Cmd{
+		Path:   psExec,
+		Args:   execArgs,
+		Stdout: os.Stdout,
+		Stderr: os.Stderr,
+	}
+	return cmdImport.Run()
+}
+
+func runConfiguredLeaf(t *testing.T, hubPort int, certStore string, matchBy string, match string, caMatch string, expectedLeafCount int) {
+
+	// Fire up the leaf
+	u, err := url.Parse(fmt.Sprintf("nats://localhost:%d", hubPort))
+	if err != nil {
+		t.Fatalf("Error parsing url: %v", err)
+	}
+
+	configStr := fmt.Sprintf(`
+		port: -1
+		leaf {
+			remotes [
+				{
+					url: "%s"
+					tls {
+						cert_store: "%s"
+						cert_match_by: "%s"
+						cert_match: "%s"
+						ca_certs_match: %s
+
+						# Test settings that succeed should be equivalent to:
+						# cert_file: "../test/configs/certs/tlsauth/client.pem"
+						# key_file: "../test/configs/certs/tlsauth/client-key.pem"
+						# ca_file: "../test/configs/certs/tlsauth/ca.pem"
+						timeout: 5
+					}
+				}
+			]
+		}
+	`, u.String(), certStore, matchBy, match, caMatch)
+
+	leafConfig := createConfFile(t, []byte(configStr))
+	defer removeFile(t, leafConfig)
+	leafServer, _ := RunServerWithConfig(leafConfig)
+	defer leafServer.Shutdown()
+
+	// After client verify, hub will match by SAN email, SAN dns, and Subject (in that order)
+	// Our test client specifies Subject only so we should match on that...
+
+	// A little settle time
+	time.Sleep(1 * time.Second)
+	checkLeafNodeConnectedCount(t, leafServer, expectedLeafCount)
+}
+
+// TestLeafTLSWindowsCertStore tests the topology of two NATS Servers connected as leaf and hub with authentication of
+// leaf to hub via mTLS with leaf's certificate and signing key provisioned in the Windows certificate store.
+func TestLeafTLSWindowsCertStore(t *testing.T) {
+
+	// Client Identity (client.pem)
+	// Issuer: O = NATS CA, OU = NATS.io, CN = localhost
+	// Subject: OU = NATS.io, CN = example.com
+
+	// Make sure windows cert store is reset to avoid conflict with other tests
+	err := runPowershellScript("../test/configs/certs/tlsauth/certstore/delete-cert-from-store.ps1", nil)
+	if err != nil {
+		t.Fatalf("expected powershell cert delete to succeed: %s", err.Error())
+	}
+
+	// Provision Windows cert store with client cert and secret
+	err = runPowershellScript("../test/configs/certs/tlsauth/certstore/import-p12-client.ps1", nil)
+	if err != nil {
+		t.Fatalf("expected powershell provision to succeed: %s", err.Error())
+	}
+
+	err = runPowershellScript("../test/configs/certs/tlsauth/certstore/import-p12-ca.ps1", nil)
+	if err != nil {
+		t.Fatalf("expected powershell provision CA to succeed: %s", err.Error())
+	}
+
+	// Fire up the hub
+	hubConfig := createConfFile(t, []byte(`
+		port: -1
+		leaf {
+			listen: "127.0.0.1:-1"
+			tls {
+				ca_file: "../test/configs/certs/tlsauth/ca.pem"
+				cert_file: "../test/configs/certs/tlsauth/server.pem"
+				key_file:  "../test/configs/certs/tlsauth/server-key.pem"
+				timeout: 5
+				verify_and_map: true
+			}
+		}
+
+		accounts: {
+			AcctA: {
+			  users: [ {user: "OU = NATS.io, CN = example.com"} ]
+			},
+			AcctB: {
+			  users: [ {user: UserB1} ]
+			},
+			SYS: {
+				users: [ {user: System} ]
+			}
+		}
+		system_account: "SYS"
+	`))
+	defer removeFile(t, hubConfig)
+	hubServer, hubOptions := RunServerWithConfig(hubConfig)
+	defer hubServer.Shutdown()
+
+	testCases := []struct {
+		certStore         string
+		certMatchBy       string
+		certMatch         string
+		caCertsMatch      string
+		expectedLeafCount int
+	}{
+		// Test subject and issuer
+		{"WindowsCurrentUser", "Subject", "example.com", "\"NATS CA\"", 1},
+		{"WindowsCurrentUser", "Issuer", "NATS CA", "\"NATS CA\"", 1},
+		{"WindowsCurrentUser", "Issuer", "Frodo Baggins, Inc.", "\"NATS CA\"", 0},
+		{"WindowsCurrentUser", "Thumbprint", "7e44f478114a2e29b98b00beb1b3687d8dc0e481", "\"NATS CA\"", 0},
+		// Test CAs, NATS CA is valid, others are missing
+		{"WindowsCurrentUser", "Subject", "example.com", "[\"NATS CA\"]", 1},
+		{"WindowsCurrentUser", "Subject", "example.com", "[\"GlobalSign\"]", 0},
+		{"WindowsCurrentUser", "Subject", "example.com", "[\"Missing NATS Cert\"]", 0},
+		{"WindowsCurrentUser", "Subject", "example.com", "[\"NATS CA\", \"Missing NATS Cert1\"]", 1},
+		{"WindowsCurrentUser", "Subject", "example.com", "[\"Missing Cert2\",\"NATS CA\"]", 1},
+		{"WindowsCurrentUser", "Subject", "example.com", "[\"Missing, Cert3\",\"Missing NATS Cert4\"]", 0},
+	}
+	for _, tc := range testCases {
+		testName := fmt.Sprintf("%s by %s match %s", tc.certStore, tc.certMatchBy, tc.certMatch)
+		t.Run(fmt.Sprintf(testName, tc.certStore, tc.certMatchBy, tc.certMatch, tc.caCertsMatch), func(t *testing.T) {
+			defer func() {
+				if r := recover(); r != nil {
+					if tc.expectedLeafCount != 0 {
+						t.Fatalf("did not expect panic: %s", testName)
+					} else {
+						if !strings.Contains(fmt.Sprintf("%v", r), "Error processing configuration file") {
+							t.Fatalf("did not expect unknown panic: %s", testName)
+						}
+					}
+				}
+			}()
+			runConfiguredLeaf(t, hubOptions.LeafNode.Port,
+				tc.certStore, tc.certMatchBy, tc.certMatch,
+				tc.caCertsMatch, tc.expectedLeafCount)
+		})
+	}
+}
+
+// TestServerTLSWindowsCertStore tests the topology of a NATS server requiring TLS and gettings it own server
+// cert identity (as used when accepting NATS client connections and negotiating TLS) from Windows certificate store.
+func TestServerTLSWindowsCertStore(t *testing.T) {
+
+	// Server Identity (server.pem)
+	// Issuer: O = NATS CA, OU = NATS.io, CN = localhost
+	// Subject: OU = NATS.io Operators, CN = localhost
+
+	// Make sure windows cert store is reset to avoid conflict with other tests
+	err := runPowershellScript("../test/configs/certs/tlsauth/certstore/delete-cert-from-store.ps1", nil)
+	if err != nil {
+		t.Fatalf("expected powershell cert delete to succeed: %s", err.Error())
+	}
+
+	// Provision Windows cert store with server cert and secret
+	err = runPowershellScript("../test/configs/certs/tlsauth/certstore/import-p12-server.ps1", nil)
+	if err != nil {
+		t.Fatalf("expected powershell provision to succeed: %s", err.Error())
+	}
+
+	err = runPowershellScript("../test/configs/certs/tlsauth/certstore/import-p12-ca.ps1", nil)
+	if err != nil {
+		t.Fatalf("expected powershell provision CA to succeed: %s", err.Error())
+	}
+
+	// Fire up the server
+	srvConfig := createConfFile(t, []byte(`
+	listen: "localhost:-1"
+	tls {
+		cert_store: "WindowsCurrentUser"
+		cert_match_by: "Subject"
+		cert_match: "NATS.io Operators"
+		ca_certs_match: ["NATS CA"]
+		timeout: 5
+	}
+	`))
+	defer removeFile(t, srvConfig)
+	srvServer, _ := RunServerWithConfig(srvConfig)
+	if srvServer == nil {
+		t.Fatalf("expected to be able start server with cert store configuration")
+	}
+	defer srvServer.Shutdown()
+
+	testCases := []struct {
+		clientCA string
+		expect   bool
+	}{
+		{"../test/configs/certs/tlsauth/ca.pem", true},
+		{"../test/configs/certs/tlsauth/client.pem", false},
+	}
+	for _, tc := range testCases {
+		t.Run(fmt.Sprintf("Client CA: %s", tc.clientCA), func(t *testing.T) {
+			nc, _ := nats.Connect(srvServer.clientConnectURLs[0], nats.RootCAs(tc.clientCA))
+			err := nc.Publish("foo", []byte("hello TLS server-authenticated server"))
+			if (err != nil) == tc.expect {
+				t.Fatalf("expected publish result %v to TLS authenticated server", tc.expect)
+			}
+			nc.Close()
+
+			for i := 0; i < 5; i++ {
+				nc, _ = nats.Connect(srvServer.clientConnectURLs[0], nats.RootCAs(tc.clientCA))
+				err = nc.Publish("foo", []byte("hello TLS server-authenticated server"))
+				if (err != nil) == tc.expect {
+					t.Fatalf("expected repeated connection result %v to TLS authenticated server", tc.expect)
+				}
+				nc.Close()
+			}
+		})
+	}
+}
+
+// TestServerIgnoreExpiredCerts tests if the server skips expired certificates in configuration, and finds non-expired ones
+func TestServerIgnoreExpiredCerts(t *testing.T) {
+
+	// Server Identities: expired.pem; not-expired.pem
+	// Issuer: OU = NATS.io, CN = localhost
+	// Subject: OU = NATS.io Operators, CN = localhost
+
+	testCases := []struct {
+		certFile string
+		expect   bool
+	}{
+		{"expired.p12", false},
+		{"not-expired.p12", true},
+	}
+	for _, tc := range testCases {
+		t.Run(fmt.Sprintf("Server certificate: %s", tc.certFile), func(t *testing.T) {
+			// Make sure windows cert store is reset to avoid conflict with other tests
+			err := runPowershellScript("../test/configs/certs/tlsauth/certstore/delete-cert-from-store.ps1", nil)
+			if err != nil {
+				t.Fatalf("expected powershell cert delete to succeed: %s", err.Error())
+			}
+
+			// Provision Windows cert store with server cert and secret
+			err = runPowershellScript("../test/configs/certs/tlsauth/certstore/import-p12-server.ps1", []string{tc.certFile})
+			if err != nil {
+				t.Fatalf("expected powershell provision to succeed: %s", err.Error())
+			}
+			// Fire up the server
+			srvConfig := createConfFile(t, []byte(`
+			listen: "localhost:-1"
+			tls {
+				cert_store: "WindowsCurrentUser"
+				cert_match_by: "Subject"
+				cert_match: "NATS.io Operators"
+				cert_match_skip_invalid: true
+				timeout: 5
+			}
+			`))
+			defer removeFile(t, srvConfig)
+			cfg, _ := ProcessConfigFile(srvConfig)
+			if (cfg != nil) == tc.expect {
+				return
+			}
+			if tc.expect == false {
+				t.Fatalf("expected server start to fail with expired certificate")
+			} else {
+				t.Fatalf("expected server to start with non expired certificate")
+			}
+		})
+	}
+}
+
+func TestWindowsTLS12ECDSA(t *testing.T) {
+	err := runPowershellScript("../test/configs/certs/tlsauth/certstore/import-p12-server.ps1", []string{"ecdsa_server.pfx"})
+	if err != nil {
+		t.Fatalf("expected powershell provision to succeed: %v", err)
+	}
+
+	config := createConfFile(t, []byte(`
+	listen: "localhost:-1"
+	tls {
+		cert_store: "WindowsCurrentUser"
+		cert_match_by: "Thumbprint"
+		cert_match: "4F8AF21756E5DBBD54619BBB6F3CC5D455ED4468"
+		cert_match_skip_invalid: true
+		timeout: 5
+	}
+	`))
+	defer removeFile(t, config)
+
+	srv, _ := RunServerWithConfig(config)
+	if srv == nil {
+		t.Fatalf("expected to be able start server with cert store configuration")
+	}
+	defer srv.Shutdown()
+
+	for name, version := range map[string]uint16{
+		"TLS 1.3": tls.VersionTLS13,
+		"TLS 1.2": tls.VersionTLS12,
+	} {
+		t.Run(name, func(t *testing.T) {
+			tc := &tls.Config{MaxVersion: version, MinVersion: version, InsecureSkipVerify: true}
+
+			if _, err = nats.Connect(srv.clientConnectURLs[0], nats.Secure(tc)); err != nil {
+				t.Fatalf("connection with %s: %v", name, err)
+			}
+
+			t.Logf("successful connection with %s", name)
+		})
+	}
+}

server/ciphersuites.go

@@ -0,0 +1,79 @@
+// Copyright 2016-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"crypto/fips140"
+	"crypto/tls"
+)
+
+func init() {
+	for _, cs := range tls.CipherSuites() {
+		cipherMap[cs.Name] = cs
+		cipherMapByID[cs.ID] = cs
+	}
+	for _, cs := range tls.InsecureCipherSuites() {
+		cipherMap[cs.Name] = cs
+		cipherMapByID[cs.ID] = cs
+	}
+}
+
+var cipherMap = map[string]*tls.CipherSuite{}
+var cipherMapByID = map[uint16]*tls.CipherSuite{}
+
+// Where we maintain available curve preferences
+var curvePreferenceMap = map[string]tls.CurveID{
+	"SecP384r1MLKEM1024": tls.SecP384r1MLKEM1024,
+	"SecP256r1MLKEM768":  tls.SecP256r1MLKEM768,
+	"X25519MLKEM768":     tls.X25519MLKEM768,
+	"X25519":             tls.X25519,
+	"CurveP256":          tls.CurveP256,
+	"CurveP384":          tls.CurveP384,
+	"CurveP521":          tls.CurveP521,
+}
+
+func defaultCipherSuites() []uint16 {
+	ciphers := tls.CipherSuites()
+	defaults := make([]uint16, 0, len(ciphers))
+	for _, cs := range ciphers {
+		defaults = append(defaults, cs.ID)
+	}
+	return defaults
+}
+
+// reorder to default to the highest level of security.  See:
+// https://blog.bracebin.com/achieving-perfect-ssl-labs-score-with-go
+func defaultCurvePreferences() []tls.CurveID {
+	if fips140.Enabled() {
+		// X25519 is not FIPS-approved by itself, but it is when combined with MLKEM768.
+		// SecP256r1MLKEM768 and SecP384r1MLKEM1024 are both FIPS-approved hybrids.
+		return []tls.CurveID{
+			tls.X25519MLKEM768,     // post-quantum
+			tls.SecP384r1MLKEM1024, // post-quantum
+			tls.SecP256r1MLKEM768,  // post-quantum
+			tls.CurveP256,
+			tls.CurveP384,
+			tls.CurveP521,
+		}
+	}
+	return []tls.CurveID{
+		tls.X25519MLKEM768,     // post-quantum
+		tls.SecP384r1MLKEM1024, // post-quantum
+		tls.SecP256r1MLKEM768,  // post-quantum
+		tls.X25519,             // faster than P256, arguably more secure
+		tls.CurveP256,
+		tls.CurveP384,
+		tls.CurveP521,
+	}
+}

server/client_proxyproto.go

@@ -0,0 +1,415 @@
+// Copyright 2025-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"encoding/binary"
+	"errors"
+	"fmt"
+	"io"
+	"net"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// PROXY protocol v2 constants
+const (
+	// Protocol signature (12 bytes)
+	proxyProtoV2Sig = "\x0D\x0A\x0D\x0A\x00\x0D\x0A\x51\x55\x49\x54\x0A"
+
+	// Version and command byte format: version(4 bits) | command(4 bits)
+	proxyProtoV2VerMask = 0xF0
+	proxyProtoV2Ver     = 0x20 // Version 2
+
+	// Commands
+	proxyProtoCmdMask  = 0x0F
+	proxyProtoCmdLocal = 0x00 // LOCAL command (health check, use original connection)
+	proxyProtoCmdProxy = 0x01 // PROXY command (proxied connection)
+
+	// Address family and protocol byte format: family(4 bits) | protocol(4 bits)
+	proxyProtoFamilyMask    = 0xF0
+	proxyProtoFamilyUnspec  = 0x00 // Unspecified
+	proxyProtoFamilyInet    = 0x10 // IPv4
+	proxyProtoFamilyInet6   = 0x20 // IPv6
+	proxyProtoFamilyUnix    = 0x30 // Unix socket
+	proxyProtoProtoMask     = 0x0F
+	proxyProtoProtoUnspec   = 0x00 // Unspecified
+	proxyProtoProtoStream   = 0x01 // TCP/STREAM
+	proxyProtoProtoDatagram = 0x02 // UDP/DGRAM
+
+	// Address sizes
+	proxyProtoAddrSizeIPv4 = 12 // 4 (src IP) + 4 (dst IP) + 2 (src port) + 2 (dst port)
+	proxyProtoAddrSizeIPv6 = 36 // 16 (src IP) + 16 (dst IP) + 2 (src port) + 2 (dst port)
+
+	// Header sizes
+	proxyProtoV2HeaderSize = 16 // Fixed header: 12 (sig) + 1 (ver/cmd) + 1 (fam/proto) + 2 (addr len)
+
+	// Timeout for reading PROXY protocol header
+	proxyProtoReadTimeout = 5 * time.Second
+)
+
+// PROXY protocol v1 constants
+const (
+	proxyProtoV1Prefix     = "PROXY "
+	proxyProtoV1MaxLineLen = 107 // Maximum line length including CRLF
+	proxyProtoV1TCP4       = "TCP4"
+	proxyProtoV1TCP6       = "TCP6"
+	proxyProtoV1Unknown    = "UNKNOWN"
+)
+
+var (
+	// Errors
+	errProxyProtoInvalid      = errors.New("invalid PROXY protocol header")
+	errProxyProtoUnsupported  = errors.New("unsupported PROXY protocol feature")
+	errProxyProtoTimeout      = errors.New("timeout reading PROXY protocol header")
+	errProxyProtoUnrecognized = errors.New("unrecognized PROXY protocol format")
+)
+
+// proxyProtoAddr contains the address information extracted from PROXY protocol header
+type proxyProtoAddr struct {
+	srcIP   net.IP
+	srcPort uint16
+	dstIP   net.IP
+	dstPort uint16
+}
+
+// String implements net.Addr interface
+func (p *proxyProtoAddr) String() string {
+	return net.JoinHostPort(p.srcIP.String(), fmt.Sprintf("%d", p.srcPort))
+}
+
+// Network implements net.Addr interface
+func (p *proxyProtoAddr) Network() string {
+	if p.srcIP.To4() != nil {
+		return "tcp4"
+	}
+	return "tcp6"
+}
+
+// proxyConn wraps a net.Conn to override RemoteAddr() with the address
+// extracted from the PROXY protocol header
+type proxyConn struct {
+	net.Conn
+	remoteAddr net.Addr
+}
+
+// RemoteAddr returns the original client address extracted from PROXY protocol
+func (pc *proxyConn) RemoteAddr() net.Addr {
+	return pc.remoteAddr
+}
+
+// detectProxyProtoVersion reads the first bytes and determines protocol version.
+// Returns 1 for v1, 2 for v2, or error.
+// The first 6 bytes read are returned so they can be used by the parser.
+func detectProxyProtoVersion(conn net.Conn) (version int, header []byte, err error) {
+	// Read first 6 bytes to check for "PROXY " or v2 signature
+	header = make([]byte, 6)
+	if _, err = io.ReadFull(conn, header); err != nil {
+		return 0, nil, fmt.Errorf("failed to read protocol version: %w", err)
+	}
+	switch bytesToString(header) {
+	case proxyProtoV1Prefix:
+		return 1, header, nil
+	case proxyProtoV2Sig[:6]:
+		return 2, header, nil
+	default:
+		// Return the consumed bytes so the caller can replay them into the
+		// next protocol layer instead of discarding them.
+		return 0, header, errProxyProtoUnrecognized
+	}
+}
+
+// readProxyProtoV1Header parses PROXY protocol v1 text format.
+// Expects the "PROXY " prefix (6 bytes) to have already been consumed.
+// Returns any bytes that were read past the trailing CRLF so the caller can
+// replay them into the next protocol layer.
+func readProxyProtoV1Header(conn net.Conn) (*proxyProtoAddr, []byte, error) {
+	// Read rest of line (max 107 bytes total, already read 6)
+	maxRemaining := proxyProtoV1MaxLineLen - 6
+
+	// Read up to maxRemaining bytes at once (more efficient than byte-by-byte)
+	buf := make([]byte, maxRemaining)
+	var line []byte
+	var remaining []byte
+
+	for len(line) < maxRemaining {
+		// Read available data
+		n, err := conn.Read(buf[len(line):])
+		if err != nil {
+			return nil, nil, fmt.Errorf("failed to read v1 line: %w", err)
+		}
+
+		line = buf[:len(line)+n]
+
+		// Look for CRLF in what we've read so far
+		for i := 0; i < len(line)-1; i++ {
+			if line[i] == '\r' && line[i+1] == '\n' {
+				// Found CRLF - keep any over-read bytes for the client parser.
+				remaining = append(remaining, line[i+2:]...)
+				line = line[:i]
+				goto foundCRLF
+			}
+		}
+	}
+
+	// Exceeded max length without finding CRLF
+	return nil, nil, fmt.Errorf("%w: v1 line too long", errProxyProtoInvalid)
+
+foundCRLF:
+	// Get parts from the protocol
+	parts := strings.Fields(string(line))
+
+	// Validate format
+	if len(parts) < 1 {
+		return nil, nil, fmt.Errorf("%w: invalid v1 format", errProxyProtoInvalid)
+	}
+
+	// Handle UNKNOWN (health check, like v2 LOCAL)
+	if parts[0] == proxyProtoV1Unknown {
+		return nil, remaining, nil
+	}
+
+	// Must have exactly 5 parts: protocol, src-ip, dst-ip, src-port, dst-port
+	if len(parts) != 5 {
+		return nil, nil, fmt.Errorf("%w: invalid v1 format", errProxyProtoInvalid)
+	}
+
+	protocol := parts[0]
+	srcIP := net.ParseIP(parts[1])
+	dstIP := net.ParseIP(parts[2])
+
+	if srcIP == nil || dstIP == nil {
+		return nil, nil, fmt.Errorf("%w: invalid address", errProxyProtoInvalid)
+	}
+
+	// Parse ports
+	srcPort, err := strconv.ParseUint(parts[3], 10, 16)
+	if err != nil {
+		return nil, nil, fmt.Errorf("invalid source port: %w", err)
+	}
+
+	dstPort, err := strconv.ParseUint(parts[4], 10, 16)
+	if err != nil {
+		return nil, nil, fmt.Errorf("invalid dest port: %w", err)
+	}
+
+	// Validate protocol matches IP version. The textual form determines the
+	// family: TCP4 requires dotted-quad addresses, TCP6 requires IPv6
+	// addresses. IPv4-mapped IPv6 addresses (e.g. "::ffff:192.0.2.1") are
+	// valid for TCP6 since dual-stack proxies can emit those for IPv4
+	// clients on IPv6 sockets, matching the v2 parser behavior.
+	srcIsV6 := strings.Contains(parts[1], ":")
+	dstIsV6 := strings.Contains(parts[2], ":")
+	if protocol == proxyProtoV1TCP4 && (srcIsV6 || dstIsV6) {
+		return nil, nil, fmt.Errorf("%w: TCP4 with IPv6 address", errProxyProtoInvalid)
+	}
+	if protocol == proxyProtoV1TCP6 && (!srcIsV6 || !dstIsV6) {
+		return nil, nil, fmt.Errorf("%w: TCP6 with IPv4 address", errProxyProtoInvalid)
+	}
+	if protocol != proxyProtoV1TCP4 && protocol != proxyProtoV1TCP6 {
+		return nil, nil, fmt.Errorf("%w: invalid protocol %s", errProxyProtoInvalid, protocol)
+	}
+
+	return &proxyProtoAddr{
+		srcIP:   srcIP,
+		srcPort: uint16(srcPort),
+		dstIP:   dstIP,
+		dstPort: uint16(dstPort),
+	}, remaining, nil
+}
+
+// readProxyProtoHeader reads and parses PROXY protocol (v1 or v2) from the connection.
+// Automatically detects version and routes to appropriate parser.
+// If the command is LOCAL/UNKNOWN (health check), it returns nil for addr and no error.
+// If the command is PROXY, it returns the parsed address information.
+// It also returns any bytes that were read past the v1 header terminator so the
+// caller can replay them into the normal client parser.
+// The connection must be fresh (no data read yet).
+func readProxyProtoHeader(conn net.Conn) (*proxyProtoAddr, []byte, error) {
+	// Set read deadline to prevent hanging on slow/malicious clients
+	if err := conn.SetReadDeadline(time.Now().Add(proxyProtoReadTimeout)); err != nil {
+		return nil, nil, err
+	}
+	defer conn.SetReadDeadline(time.Time{})
+
+	// Detect version.
+	// On errProxyProtoUnrecognized, firstBytes holds the bytes that were
+	// consumed so the caller can replay them.
+	version, firstBytes, err := detectProxyProtoVersion(conn)
+	if err != nil {
+		return nil, firstBytes, err
+	}
+
+	switch version {
+	case 1:
+		// v1 parser expects "PROXY " prefix already consumed
+		return readProxyProtoV1Header(conn)
+	case 2:
+		// Read rest of v2 signature (bytes 6-11, total 6 more bytes)
+		remaining := make([]byte, 6)
+		if _, err := io.ReadFull(conn, remaining); err != nil {
+			return nil, nil, fmt.Errorf("failed to read v2 signature: %w", err)
+		}
+
+		// Verify full signature
+		fullSig := string(firstBytes) + string(remaining)
+		if fullSig != proxyProtoV2Sig {
+			return nil, nil, fmt.Errorf("%w: invalid signature", errProxyProtoInvalid)
+		}
+
+		// Read rest of header: ver/cmd, fam/proto, addr-len (4 bytes)
+		header := make([]byte, 4)
+		if _, err := io.ReadFull(conn, header); err != nil {
+			return nil, nil, fmt.Errorf("failed to read v2 header: %w", err)
+		}
+
+		// Continue with parsing
+		addr, err := parseProxyProtoV2Header(conn, header)
+		return addr, nil, err
+	default:
+		return nil, nil, fmt.Errorf("unsupported PROXY protocol version: %d", version)
+	}
+}
+
+// readProxyProtoV2Header is kept for backward compatibility and direct testing.
+// It reads and parses a PROXY protocol v2 header from the connection.
+// If the command is LOCAL (health check), it returns nil for addr and no error.
+// If the command is PROXY, it returns the parsed address information.
+// The connection must be fresh (no data read yet).
+func readProxyProtoV2Header(conn net.Conn) (*proxyProtoAddr, error) {
+	// Set read deadline to prevent hanging on slow/malicious clients
+	if err := conn.SetReadDeadline(time.Now().Add(proxyProtoReadTimeout)); err != nil {
+		return nil, err
+	}
+	defer conn.SetReadDeadline(time.Time{})
+
+	// Read fixed header (16 bytes)
+	header := make([]byte, proxyProtoV2HeaderSize)
+	if _, err := io.ReadFull(conn, header); err != nil {
+		if ne, ok := err.(net.Error); ok && ne.Timeout() {
+			return nil, errProxyProtoTimeout
+		}
+		return nil, fmt.Errorf("failed to read PROXY protocol header: %w", err)
+	}
+
+	// Validate signature (first 12 bytes)
+	if string(header[:12]) != proxyProtoV2Sig {
+		return nil, fmt.Errorf("%w: invalid signature", errProxyProtoInvalid)
+	}
+
+	// Continue with parsing after signature
+	return parseProxyProtoV2Header(conn, header[12:16])
+}
+
+// parseProxyProtoV2Header parses v2 protocol after signature has been validated.
+// header contains the 4 bytes: ver/cmd, fam/proto, addr-len (2 bytes).
+func parseProxyProtoV2Header(conn net.Conn, header []byte) (*proxyProtoAddr, error) {
+	// Parse version and command
+	verCmd := header[0]
+	version := verCmd & proxyProtoV2VerMask
+	command := verCmd & proxyProtoCmdMask
+
+	if version != proxyProtoV2Ver {
+		return nil, fmt.Errorf("%w: invalid version 0x%02x", errProxyProtoInvalid, version)
+	}
+
+	// Parse address family and protocol
+	famProto := header[1]
+	family := famProto & proxyProtoFamilyMask
+	protocol := famProto & proxyProtoProtoMask
+
+	// Parse address length (big-endian uint16)
+	addrLen := binary.BigEndian.Uint16(header[2:4])
+
+	// Handle LOCAL command (health check)
+	if command == proxyProtoCmdLocal {
+		// For LOCAL, we should skip the address data if any
+		if addrLen > 0 {
+			// Discard the address data
+			if _, err := io.CopyN(io.Discard, conn, int64(addrLen)); err != nil {
+				return nil, fmt.Errorf("failed to discard LOCAL command address data: %w", err)
+			}
+		}
+		return nil, nil // nil addr indicates LOCAL command
+	}
+
+	// Handle PROXY command
+	if command != proxyProtoCmdProxy {
+		return nil, fmt.Errorf("unknown PROXY protocol command: 0x%02x", command)
+	}
+
+	// Validate protocol (we only support STREAM/TCP)
+	if protocol != proxyProtoProtoStream {
+		return nil, fmt.Errorf("%w: only STREAM protocol supported", errProxyProtoUnsupported)
+	}
+
+	// Parse address data based on family
+	var addr *proxyProtoAddr
+	var err error
+	switch family {
+	case proxyProtoFamilyInet:
+		addr, err = parseIPv4Addr(conn, addrLen)
+	case proxyProtoFamilyInet6:
+		addr, err = parseIPv6Addr(conn, addrLen)
+	case proxyProtoFamilyUnspec:
+		// UNSPEC family with PROXY command is valid but rare
+		// Just skip the address data
+		if addrLen > 0 {
+			if _, err := io.CopyN(io.Discard, conn, int64(addrLen)); err != nil {
+				return nil, fmt.Errorf("failed to discard UNSPEC address address data: %w", err)
+			}
+		}
+		return nil, nil
+	default:
+		return nil, fmt.Errorf("%w: unsupported address family 0x%02x", errProxyProtoUnsupported, family)
+	}
+	return addr, err
+}
+
+// parseIPv4Addr parses IPv4 address data from PROXY protocol header
+func parseIPv4Addr(conn net.Conn, addrLen uint16) (*proxyProtoAddr, error) {
+	// IPv4: 4 (src IP) + 4 (dst IP) + 2 (src port) + 2 (dst port) = 12 bytes minimum
+	if addrLen < proxyProtoAddrSizeIPv4 {
+		return nil, fmt.Errorf("IPv4 address data too short: %d bytes", addrLen)
+	}
+	addrData := make([]byte, addrLen)
+	if _, err := io.ReadFull(conn, addrData); err != nil {
+		return nil, fmt.Errorf("failed to read IPv4 address data: %w", err)
+	}
+	return &proxyProtoAddr{
+		srcIP:   net.IP(addrData[0:4]),
+		dstIP:   net.IP(addrData[4:8]),
+		srcPort: binary.BigEndian.Uint16(addrData[8:10]),
+		dstPort: binary.BigEndian.Uint16(addrData[10:12]),
+	}, nil
+}
+
+// parseIPv6Addr parses IPv6 address data from PROXY protocol header
+func parseIPv6Addr(conn net.Conn, addrLen uint16) (*proxyProtoAddr, error) {
+	// IPv6: 16 (src IP) + 16 (dst IP) + 2 (src port) + 2 (dst port) = 36 bytes minimum
+	if addrLen < proxyProtoAddrSizeIPv6 {
+		return nil, fmt.Errorf("IPv6 address data too short: %d bytes", addrLen)
+	}
+	addrData := make([]byte, addrLen)
+	if _, err := io.ReadFull(conn, addrData); err != nil {
+		return nil, fmt.Errorf("failed to read IPv6 address data: %w", err)
+	}
+	return &proxyProtoAddr{
+		srcIP:   net.IP(addrData[0:16]),
+		dstIP:   net.IP(addrData[16:32]),
+		srcPort: binary.BigEndian.Uint16(addrData[32:34]),
+		dstPort: binary.BigEndian.Uint16(addrData[34:36]),
+	}, nil
+}

server/client_proxyproto_test.go

@@ -0,0 +1,925 @@
+// Copyright 2025-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"bufio"
+	"bytes"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/binary"
+	"fmt"
+	"io"
+	"net"
+	"os"
+	"strings"
+	"testing"
+	"time"
+)
+
+// mockConn is a mock net.Conn for testing
+type mockConn struct {
+	net.Conn
+	readBuf  *bytes.Buffer
+	writeBuf *bytes.Buffer
+	closed   bool
+	deadline time.Time
+}
+
+func newMockConn(data []byte) *mockConn {
+	return &mockConn{
+		readBuf:  bytes.NewBuffer(data),
+		writeBuf: &bytes.Buffer{},
+	}
+}
+
+func (m *mockConn) Read(b []byte) (int, error) {
+	if m.closed {
+		return 0, fmt.Errorf("connection closed")
+	}
+	if !m.deadline.IsZero() && time.Now().After(m.deadline) {
+		return 0, &net.OpError{Op: "read", Err: fmt.Errorf("timeout")}
+	}
+	return m.readBuf.Read(b)
+}
+
+func (m *mockConn) Write(b []byte) (int, error) {
+	if m.closed {
+		return 0, fmt.Errorf("connection closed")
+	}
+	return m.writeBuf.Write(b)
+}
+
+func (m *mockConn) Close() error {
+	m.closed = true
+	return nil
+}
+
+func (m *mockConn) LocalAddr() net.Addr {
+	return &net.TCPAddr{IP: net.ParseIP("127.0.0.1"), Port: 4222}
+}
+
+func (m *mockConn) RemoteAddr() net.Addr {
+	return &net.TCPAddr{IP: net.ParseIP("192.168.1.100"), Port: 54321}
+}
+
+func (m *mockConn) SetDeadline(t time.Time) error {
+	m.deadline = t
+	return nil
+}
+
+func (m *mockConn) SetReadDeadline(t time.Time) error {
+	m.deadline = t
+	return nil
+}
+
+func (m *mockConn) SetWriteDeadline(t time.Time) error {
+	return nil
+}
+
+// buildProxyV2Header builds a valid PROXY protocol v2 header
+func buildProxyV2Header(t *testing.T, srcIP, dstIP string, srcPort, dstPort uint16, family byte) []byte {
+	t.Helper()
+
+	buf := &bytes.Buffer{}
+	buf.WriteString(proxyProtoV2Sig)                    // Write signature
+	buf.WriteByte(proxyProtoV2Ver | proxyProtoCmdProxy) // Write version and command (version 2, PROXY command)
+	buf.WriteByte(family | proxyProtoProtoStream)       // Write family and protocol
+
+	src := net.ParseIP(srcIP)
+	dst := net.ParseIP(dstIP)
+	var addrData []byte
+	switch family {
+	case proxyProtoFamilyInet:
+		// IPv4: 12 bytes
+		addrData = make([]byte, proxyProtoAddrSizeIPv4)
+		copy(addrData[0:4], src.To4())
+		copy(addrData[4:8], dst.To4())
+		binary.BigEndian.PutUint16(addrData[8:10], srcPort)
+		binary.BigEndian.PutUint16(addrData[10:12], dstPort)
+	case proxyProtoFamilyInet6:
+		// IPv6: 36 bytes
+		addrData = make([]byte, proxyProtoAddrSizeIPv6)
+		copy(addrData[0:16], src.To16())
+		copy(addrData[16:32], dst.To16())
+		binary.BigEndian.PutUint16(addrData[32:34], srcPort)
+		binary.BigEndian.PutUint16(addrData[34:36], dstPort)
+	default:
+		t.Fatalf("unsupported address family: %d", family)
+	}
+
+	addrLen := make([]byte, 2)
+	binary.BigEndian.PutUint16(addrLen, uint16(len(addrData)))
+	buf.Write(addrLen)
+	buf.Write(addrData)
+
+	return buf.Bytes()
+}
+
+// buildProxyV2LocalHeader builds a PROXY protocol v2 LOCAL command header
+func buildProxyV2LocalHeader() []byte {
+	buf := &bytes.Buffer{}
+	buf.WriteString(proxyProtoV2Sig)                              // // Write signature
+	buf.WriteByte(proxyProtoV2Ver | proxyProtoCmdLocal)           // // Write version and command (version 2, LOCAL command)
+	buf.WriteByte(proxyProtoFamilyUnspec | proxyProtoProtoUnspec) // // Write family and protocol (UNSPEC)
+	buf.WriteByte(0)
+	buf.WriteByte(0)
+	return buf.Bytes()
+}
+
+// buildProxyV1Header builds a valid PROXY protocol v1 header (text format)
+func buildProxyV1Header(t *testing.T, protocol, srcIP, dstIP string, srcPort, dstPort uint16) []byte {
+	t.Helper()
+
+	if protocol != "TCP4" && protocol != "TCP6" && protocol != "UNKNOWN" {
+		t.Fatalf("invalid protocol: %s", protocol)
+	}
+
+	var line string
+	if protocol == "UNKNOWN" {
+		line = "PROXY UNKNOWN\r\n"
+	} else {
+		line = fmt.Sprintf("PROXY %s %s %s %d %d\r\n", protocol, srcIP, dstIP, srcPort, dstPort)
+	}
+
+	return []byte(line)
+}
+
+func TestClientProxyProtoV2ParseIPv4(t *testing.T) {
+	header := buildProxyV2Header(t, "192.168.1.50", "10.0.0.1", 12345, 4222, proxyProtoFamilyInet)
+	conn := newMockConn(header)
+
+	addr, err := readProxyProtoV2Header(conn)
+	require_NoError(t, err)
+	require_NotNil(t, addr)
+
+	require_Equal(t, addr.srcIP.String(), "192.168.1.50")
+	require_Equal(t, addr.srcPort, 12345)
+
+	require_Equal(t, addr.dstIP.String(), "10.0.0.1")
+	require_Equal(t, addr.dstPort, 4222)
+
+	// Test String() and Network() methods
+	require_Equal(t, addr.String(), "192.168.1.50:12345")
+	require_Equal(t, addr.Network(), "tcp4")
+}
+
+func TestClientProxyProtoV2ParseIPv6(t *testing.T) {
+	header := buildProxyV2Header(t, "2001:db8::1", "2001:db8::2", 54321, 4222, proxyProtoFamilyInet6)
+	conn := newMockConn(header)
+
+	addr, err := readProxyProtoV2Header(conn)
+	require_NoError(t, err)
+	require_NotNil(t, addr)
+
+	require_Equal(t, addr.srcIP.String(), "2001:db8::1")
+	require_Equal(t, addr.srcPort, 54321)
+
+	require_Equal(t, addr.dstIP.String(), "2001:db8::2")
+	require_Equal(t, addr.dstPort, 4222)
+
+	// Test Network() method for IPv6
+	require_Equal(t, addr.String(), "[2001:db8::1]:54321")
+	require_Equal(t, addr.Network(), "tcp6")
+}
+
+func TestClientProxyProtoV2ParseLocalCommand(t *testing.T) {
+	header := buildProxyV2LocalHeader()
+	conn := newMockConn(header)
+
+	addr, err := readProxyProtoV2Header(conn)
+	require_NoError(t, err)
+	require_True(t, addr == nil)
+}
+
+func TestClientProxyProtoV2InvalidSignature(t *testing.T) {
+	// Create invalid signature
+	header := []byte("INVALID_SIG_")
+	header = append(header, []byte{0x20, 0x11, 0x00, 0x0C}...)
+	conn := newMockConn(header)
+
+	_, err := readProxyProtoV2Header(conn)
+	require_Error(t, err, errProxyProtoInvalid)
+}
+
+func TestClientProxyProtoV2InvalidVersion(t *testing.T) {
+	buf := &bytes.Buffer{}
+	buf.WriteString(proxyProtoV2Sig)
+	buf.WriteByte(0x10 | proxyProtoCmdProxy) // Version 1 instead of 2
+	buf.WriteByte(proxyProtoFamilyInet | proxyProtoProtoStream)
+	buf.WriteByte(0)
+	buf.WriteByte(0)
+
+	conn := newMockConn(buf.Bytes())
+
+	_, err := readProxyProtoV2Header(conn)
+	require_Error(t, err, errProxyProtoInvalid)
+}
+
+func TestClientProxyProtoV2UnsupportedFamily(t *testing.T) {
+	buf := &bytes.Buffer{}
+	buf.WriteString(proxyProtoV2Sig)
+	buf.WriteByte(proxyProtoV2Ver | proxyProtoCmdProxy)
+	buf.WriteByte(proxyProtoFamilyUnix | proxyProtoProtoStream) // Unix socket family
+	buf.WriteByte(0)
+	buf.WriteByte(0)
+
+	conn := newMockConn(buf.Bytes())
+
+	_, err := readProxyProtoV2Header(conn)
+	require_Error(t, err, errProxyProtoUnsupported)
+}
+
+func TestClientProxyProtoV2UnsupportedProtocol(t *testing.T) {
+	buf := &bytes.Buffer{}
+	buf.WriteString(proxyProtoV2Sig)
+	buf.WriteByte(proxyProtoV2Ver | proxyProtoCmdProxy)
+	buf.WriteByte(proxyProtoFamilyInet | proxyProtoProtoDatagram) // UDP instead of TCP
+	buf.WriteByte(0)
+	buf.WriteByte(12)
+
+	conn := newMockConn(buf.Bytes())
+
+	_, err := readProxyProtoV2Header(conn)
+	require_Error(t, err, errProxyProtoUnsupported)
+}
+
+func TestClientProxyProtoV2TruncatedHeader(t *testing.T) {
+	header := buildProxyV2Header(t, "192.168.1.50", "10.0.0.1", 12345, 4222, proxyProtoFamilyInet)
+	// Only send first 10 bytes (incomplete header)
+	conn := newMockConn(header[:10])
+
+	_, err := readProxyProtoV2Header(conn)
+	require_Error(t, err, io.ErrUnexpectedEOF)
+}
+
+func TestClientProxyProtoV2ShortAddressData(t *testing.T) {
+	buf := &bytes.Buffer{}
+	buf.WriteString(proxyProtoV2Sig)
+	buf.WriteByte(proxyProtoV2Ver | proxyProtoCmdProxy)
+	buf.WriteByte(proxyProtoFamilyInet | proxyProtoProtoStream)
+	// Set address length to 12 but don't provide data
+	buf.WriteByte(0)
+	buf.WriteByte(12)
+	// Only provide 5 bytes instead of 12
+	buf.Write([]byte{1, 2, 3, 4, 5})
+
+	conn := newMockConn(buf.Bytes())
+
+	_, err := readProxyProtoV2Header(conn)
+	require_Error(t, err, io.ErrUnexpectedEOF)
+}
+
+func TestProxyConnRemoteAddr(t *testing.T) {
+	// Create a real TCP connection for testing
+	originalAddr := &net.TCPAddr{IP: net.ParseIP("192.168.1.100"), Port: 54321}
+
+	// Create proxy address
+	proxyAddr := &proxyProtoAddr{
+		srcIP:   net.ParseIP("10.0.0.50"),
+		srcPort: 12345,
+		dstIP:   net.ParseIP("10.0.0.1"),
+		dstPort: 4222,
+	}
+
+	mockConn := newMockConn(nil)
+	wrapped := &proxyConn{
+		Conn:       mockConn,
+		remoteAddr: proxyAddr,
+	}
+
+	// Verify RemoteAddr returns the proxied address
+	addr := wrapped.RemoteAddr()
+	require_Equal(t, addr.String(), "10.0.0.50:12345")
+	require_Equal(t, mockConn.RemoteAddr().String(), originalAddr.String())
+}
+
+func TestClientProxyProtoV2EndToEnd(t *testing.T) {
+	// Start a test server with PROXY protocol enabled
+	opts := DefaultOptions()
+	opts.Port = -1 // Random port
+	opts.ProxyProtocol = true
+
+	s := RunServer(opts)
+	defer s.Shutdown()
+
+	// Get the server's listening port
+	addr := s.Addr().String()
+
+	// Connect to the server
+	conn, err := net.Dial("tcp", addr)
+	require_NoError(t, err)
+	defer conn.Close()
+
+	// Send PROXY protocol header
+	clientIP := "203.0.113.50"
+	clientPort := uint16(54321)
+	header := buildProxyV2Header(t, clientIP, "127.0.0.1", clientPort, 4222, proxyProtoFamilyInet)
+
+	_, err = conn.Write(header)
+	require_NoError(t, err)
+
+	// Send CONNECT message
+	connectMsg := "CONNECT {\"verbose\":false,\"pedantic\":false,\"protocol\":1}\r\n"
+	_, err = conn.Write([]byte(connectMsg))
+	require_NoError(t, err)
+
+	// Read INFO and +OK
+	buf := make([]byte, 4096)
+	n, err := conn.Read(buf)
+	require_NoError(t, err)
+
+	response := string(buf[:n])
+	require_True(t, strings.Contains(response, "INFO"))
+
+	// Give server time to process
+	time.Sleep(100 * time.Millisecond)
+
+	// Check server's client list to verify the IP was extracted correctly
+	s.mu.Lock()
+	clients := s.clients
+	s.mu.Unlock()
+	require_True(t, len(clients) != 0)
+
+	// Find our client
+	var foundClient *client
+	for _, c := range clients {
+		c.mu.Lock()
+		if c.host == clientIP && c.port == clientPort {
+			foundClient = c
+		}
+		c.mu.Unlock()
+		if foundClient != nil {
+			break
+		}
+	}
+	require_NotNil(t, foundClient)
+}
+
+func TestClientProxyProtoV2LocalCommandEndToEnd(t *testing.T) {
+	// Start a test server with PROXY protocol enabled
+	opts := DefaultOptions()
+	opts.Port = -1 // Random port
+	opts.ProxyProtocol = true
+
+	s := RunServer(opts)
+	defer s.Shutdown()
+
+	// Get the server's listening port
+	addr := s.Addr().String()
+
+	// Connect to the server
+	conn, err := net.Dial("tcp", addr)
+	require_NoError(t, err)
+	defer conn.Close()
+
+	// Send PROXY protocol LOCAL header (health check)
+	header := buildProxyV2LocalHeader()
+
+	_, err = conn.Write(header)
+	require_NoError(t, err)
+
+	// Send CONNECT message
+	connectMsg := "CONNECT {\"verbose\":false,\"pedantic\":false,\"protocol\":1}\r\n"
+	_, err = conn.Write([]byte(connectMsg))
+	require_NoError(t, err)
+
+	// Read INFO and +OK
+	buf := make([]byte, 4096)
+	n, err := conn.Read(buf)
+	require_NoError(t, err)
+
+	response := string(buf[:n])
+	require_True(t, strings.Contains(response, "INFO"))
+
+	// Connection should work normally with LOCAL command
+	time.Sleep(100 * time.Millisecond)
+
+	// Verify at least one client is connected
+	s.mu.Lock()
+	numClients := len(s.clients)
+	s.mu.Unlock()
+	require_NotEqual(t, numClients, 0)
+}
+
+// ============================================================================
+// PROXY Protocol v1 Tests
+// ============================================================================
+
+func TestClientProxyProtoV1ParseTCP4(t *testing.T) {
+	header := buildProxyV1Header(t, "TCP4", "192.168.1.50", "10.0.0.1", 12345, 4222)
+	conn := newMockConn(header)
+
+	addr, _, err := readProxyProtoHeader(conn)
+	require_NoError(t, err)
+	require_NotNil(t, addr)
+
+	require_Equal(t, addr.srcIP.String(), "192.168.1.50")
+	require_Equal(t, addr.srcPort, 12345)
+
+	require_Equal(t, addr.dstIP.String(), "10.0.0.1")
+	require_Equal(t, addr.dstPort, 4222)
+}
+
+func TestClientProxyProtoV1ParseTCP6(t *testing.T) {
+	header := buildProxyV1Header(t, "TCP6", "2001:db8::1", "2001:db8::2", 54321, 4222)
+	conn := newMockConn(header)
+
+	addr, _, err := readProxyProtoHeader(conn)
+	require_NoError(t, err)
+	require_NotNil(t, addr)
+
+	require_Equal(t, addr.srcIP.String(), "2001:db8::1")
+	require_Equal(t, addr.srcPort, 54321)
+
+	require_Equal(t, addr.dstIP.String(), "2001:db8::2")
+	require_Equal(t, addr.dstPort, 4222)
+}
+
+func TestClientProxyProtoV1ParseTCP6IPv4Mapped(t *testing.T) {
+	// Dual-stack proxies can emit IPv4-mapped IPv6 addresses for TCP6
+	// (an IPv4 client connecting to an IPv6 socket). These are valid IPv6
+	// textual addresses and must be accepted, matching the v2 parser.
+	header := buildProxyV1Header(t, "TCP6", "::ffff:192.0.2.1", "::ffff:10.0.0.1", 1234, 4222)
+	conn := newMockConn(header)
+
+	addr, _, err := readProxyProtoHeader(conn)
+	require_NoError(t, err)
+	require_NotNil(t, addr)
+
+	require_Equal(t, addr.srcIP.String(), "192.0.2.1")
+	require_Equal(t, addr.srcPort, 1234)
+
+	require_Equal(t, addr.dstIP.String(), "10.0.0.1")
+	require_Equal(t, addr.dstPort, 4222)
+}
+
+func TestClientProxyProtoV1MismatchedDestProtocol(t *testing.T) {
+	// TCP4 with IPv6 destination address
+	header := buildProxyV1Header(t, "TCP4", "192.168.1.1", "2001:db8::2", 12345, 443)
+	conn := newMockConn(header)
+
+	_, _, err := readProxyProtoHeader(conn)
+	require_Error(t, err, errProxyProtoInvalid)
+
+	// TCP6 with IPv4 destination address
+	header2 := buildProxyV1Header(t, "TCP6", "2001:db8::1", "10.0.0.1", 12345, 443)
+	conn2 := newMockConn(header2)
+
+	_, _, err = readProxyProtoHeader(conn2)
+	require_Error(t, err, errProxyProtoInvalid)
+}
+
+func TestClientProxyProtoV1ParseUnknown(t *testing.T) {
+	header := buildProxyV1Header(t, "UNKNOWN", "", "", 0, 0)
+	conn := newMockConn(header)
+
+	addr, _, err := readProxyProtoHeader(conn)
+	require_NoError(t, err)
+	require_True(t, addr == nil)
+}
+
+func TestClientProxyProtoV1PreservesCoalescedClientBytes(t *testing.T) {
+	header := buildProxyV1Header(t, "TCP4", "192.168.1.50", "10.0.0.1", 12345, 4222)
+	connect := []byte("CONNECT {\"verbose\":false,\"pedantic\":false,\"protocol\":1}\r\n")
+	conn := newMockConn(append(header, connect...))
+
+	addr, remaining, err := readProxyProtoHeader(conn)
+	require_NoError(t, err)
+	require_NotNil(t, addr)
+	require_Equal(t, string(remaining), string(connect))
+
+	rest, err := io.ReadAll(conn)
+	require_NoError(t, err)
+	require_Equal(t, len(rest), 0)
+}
+
+func TestClientProxyProtoV1InvalidFormat(t *testing.T) {
+	// Missing fields
+	header := []byte("PROXY TCP4 192.168.1.1\r\n")
+	conn := newMockConn(header)
+
+	_, _, err := readProxyProtoHeader(conn)
+	require_Error(t, err, errProxyProtoInvalid)
+}
+
+func TestClientProxyProtoV1LineTooLong(t *testing.T) {
+	// Create a line longer than 107 bytes
+	longIP := strings.Repeat("1234567890", 12) // 120 chars
+	header := fmt.Appendf(nil, "PROXY TCP4 %s 10.0.0.1 12345 443\r\n", longIP)
+	conn := newMockConn(header)
+
+	_, _, err := readProxyProtoHeader(conn)
+	require_Error(t, err, errProxyProtoInvalid)
+}
+
+func TestClientProxyProtoV1InvalidIP(t *testing.T) {
+	header := []byte("PROXY TCP4 not.an.ip.addr 10.0.0.1 12345 443\r\n")
+	conn := newMockConn(header)
+
+	_, _, err := readProxyProtoHeader(conn)
+	require_Error(t, err, errProxyProtoInvalid)
+}
+
+func TestClientProxyProtoV1MismatchedProtocol(t *testing.T) {
+	// TCP4 with IPv6 address
+	header := buildProxyV1Header(t, "TCP4", "2001:db8::1", "2001:db8::2", 12345, 443)
+	conn := newMockConn(header)
+
+	_, _, err := readProxyProtoHeader(conn)
+	require_Error(t, err, errProxyProtoInvalid)
+
+	// TCP6 with IPv4 address
+	header2 := buildProxyV1Header(t, "TCP6", "192.168.1.1", "10.0.0.1", 12345, 443)
+	conn2 := newMockConn(header2)
+
+	_, _, err = readProxyProtoHeader(conn2)
+	require_Error(t, err, errProxyProtoInvalid)
+}
+
+func TestClientProxyProtoV1InvalidPort(t *testing.T) {
+	header := []byte("PROXY TCP4 192.168.1.1 10.0.0.1 99999 443\r\n")
+	conn := newMockConn(header)
+
+	_, _, err := readProxyProtoHeader(conn)
+	require_True(t, err != nil)
+}
+
+func TestClientProxyProtoV1EndToEnd(t *testing.T) {
+	// Start a test server with PROXY protocol enabled
+	opts := DefaultOptions()
+	opts.Port = -1 // Random port
+	opts.ProxyProtocol = true
+
+	s := RunServer(opts)
+	defer s.Shutdown()
+
+	// Get the server's listening port
+	addr := s.Addr().String()
+
+	// Connect to the server
+	conn, err := net.Dial("tcp", addr)
+	require_NoError(t, err)
+	defer conn.Close()
+
+	// Send the PROXY protocol header and first client commands in one write so
+	// the parser must preserve bytes read past the end of the v1 header.
+	clientIP, clientPort := "203.0.113.50", uint16(54321)
+	header := buildProxyV1Header(t, "TCP4", clientIP, "127.0.0.1", clientPort, 4222)
+	payload := append(header, []byte("CONNECT {\"verbose\":true,\"pedantic\":false,\"protocol\":1}\r\nPING\r\n")...)
+
+	_, err = conn.Write(payload)
+	require_NoError(t, err)
+
+	require_NoError(t, conn.SetReadDeadline(time.Now().Add(2*time.Second)))
+	cr := bufio.NewReader(conn)
+
+	line, err := cr.ReadString('\n')
+	require_NoError(t, err)
+	require_True(t, strings.HasPrefix(line, "INFO "))
+
+	line, err = cr.ReadString('\n')
+	require_NoError(t, err)
+	require_True(t, strings.HasPrefix(line, "+OK"))
+
+	expectPong(t, cr)
+
+	// Give server time to process
+	time.Sleep(100 * time.Millisecond)
+
+	// Check server's client list to verify the IP was extracted correctly
+	s.mu.Lock()
+	clients := s.clients
+	var foundClient *client
+	for _, c := range clients {
+		c.mu.Lock()
+		if c.host == clientIP && c.port == clientPort {
+			foundClient = c
+		}
+		c.mu.Unlock()
+		if foundClient != nil {
+			break
+		}
+	}
+	s.mu.Unlock()
+	require_NotNil(t, foundClient)
+}
+
+// ============================================================================
+// Mixed Protocol Version Tests
+// ============================================================================
+
+func TestClientProxyProtoVersionDetection(t *testing.T) {
+	// Test v1 detection
+	v1Header := buildProxyV1Header(t, "TCP4", "192.168.1.1", "10.0.0.1", 12345, 443)
+	conn1 := newMockConn(v1Header)
+
+	addr1, _, err := readProxyProtoHeader(conn1)
+	require_NoError(t, err)
+	require_NotNil(t, addr1)
+	require_Equal(t, addr1.srcIP.String(), "192.168.1.1")
+
+	// Test v2 detection
+	v2Header := buildProxyV2Header(t, "192.168.1.2", "10.0.0.1", 54321, 443, proxyProtoFamilyInet)
+	conn2 := newMockConn(v2Header)
+
+	addr2, _, err := readProxyProtoHeader(conn2)
+	require_NoError(t, err)
+	require_NotNil(t, addr2)
+	require_Equal(t, addr2.srcIP.String(), "192.168.1.2")
+}
+
+func TestClientProxyProtoUnrecognizedVersion(t *testing.T) {
+	// Invalid header that doesn't match v1 or v2
+	header := []byte("HELLO WORLD\r\n")
+	conn := newMockConn(header)
+
+	_, _, err := readProxyProtoHeader(conn)
+	require_Error(t, err, errProxyProtoUnrecognized)
+}
+
+// ============================================================================
+// PROXY Protocol + TLS Tests (regression for issue #7571)
+// ============================================================================
+
+// runProxyProtoTLSServer starts a server with proxy_protocol and a required
+// TLS listener (no allow_non_tls, no tls_first).
+func runProxyProtoTLSServer(t *testing.T) *Server {
+	t.Helper()
+	tc := &TLSConfigOpts{
+		CertFile: "../test/configs/certs/server-cert.pem",
+		KeyFile:  "../test/configs/certs/server-key.pem",
+		CaFile:   "../test/configs/certs/ca.pem",
+	}
+	tlsConfig, err := GenTLSConfig(tc)
+	require_NoError(t, err)
+
+	opts := DefaultOptions()
+	opts.Port = -1
+	opts.ProxyProtocol = true
+	opts.TLSConfig = tlsConfig
+	opts.TLSTimeout = 2
+	return RunServer(opts)
+}
+
+// proxyProtoTLSClientConfig builds a tls.Config trusting the test CA. The
+// server cert is issued for "localhost" / 127.0.0.1, so ServerName must match.
+func proxyProtoTLSClientConfig(t *testing.T) *tls.Config {
+	t.Helper()
+	caPEM, err := os.ReadFile("../test/configs/certs/ca.pem")
+	require_NoError(t, err)
+	pool := x509.NewCertPool()
+	require_True(t, pool.AppendCertsFromPEM(caPEM))
+	return &tls.Config{
+		RootCAs:    pool,
+		ServerName: "localhost",
+	}
+}
+
+// findProxyProtoClient finds a connected client whose host/port matches what
+// was advertised in the PROXY header.
+func findProxyProtoClient(t *testing.T, s *Server, clientIP string, clientPort uint16) *client {
+	t.Helper()
+	s.mu.Lock()
+	clients := make([]*client, 0, len(s.clients))
+	for _, c := range s.clients {
+		clients = append(clients, c)
+	}
+	s.mu.Unlock()
+
+	for _, c := range clients {
+		c.mu.Lock()
+		host, port := c.host, c.port
+		c.mu.Unlock()
+		if host == clientIP && port == clientPort {
+			return c
+		}
+	}
+	return nil
+}
+
+// connectProxyProtoTLS mirrors what an L4 proxy and a TLS-enabled NATS client
+// do over a single TCP connection: prepend the PROXY header, read the initial
+// plaintext INFO, then upgrade to TLS. Returns the buffered reader and the
+// TLS-wrapped connection for subsequent protocol writes.
+func connectProxyProtoTLS(t *testing.T, addr string, header []byte) (*bufio.Reader, *tls.Conn) {
+	t.Helper()
+	rawConn, err := net.Dial("tcp", addr)
+	require_NoError(t, err)
+	require_NoError(t, rawConn.SetDeadline(time.Now().Add(5*time.Second)))
+
+	_, err = rawConn.Write(header)
+	require_NoError(t, err)
+
+	// Server emits INFO over plaintext before the TLS upgrade. Read it now
+	// so the bytes don't end up as input to the TLS handshake.
+	plainReader := bufio.NewReader(rawConn)
+	infoLine, err := plainReader.ReadString('\n')
+	require_NoError(t, err)
+	require_True(t, strings.HasPrefix(infoLine, "INFO "))
+	require_Equal(t, plainReader.Buffered(), 0)
+
+	tlsConn := tls.Client(rawConn, proxyProtoTLSClientConfig(t))
+	require_NoError(t, tlsConn.Handshake())
+	return bufio.NewReader(tlsConn), tlsConn
+}
+
+// TestClientProxyProtoV1WithRequiredTLS verifies that with proxy_protocol and
+// required TLS, a client may send the PROXY v1 header followed by the TLS
+// upgrade on a single TCP connection. Regression for issue #7571.
+func TestClientProxyProtoV1WithRequiredTLS(t *testing.T) {
+	s := runProxyProtoTLSServer(t)
+	defer s.Shutdown()
+
+	clientIP, clientPort := "203.0.113.50", uint16(54321)
+	header := buildProxyV1Header(t, "TCP4", clientIP, "127.0.0.1", clientPort, 4222)
+	cr, tlsConn := connectProxyProtoTLS(t, s.Addr().String(), header)
+	defer tlsConn.Close()
+
+	_, err := tlsConn.Write([]byte("CONNECT {\"verbose\":true,\"pedantic\":false,\"protocol\":1}\r\nPING\r\n"))
+	require_NoError(t, err)
+
+	line, err := cr.ReadString('\n')
+	require_NoError(t, err)
+	require_True(t, strings.HasPrefix(line, "+OK"))
+	expectPong(t, cr)
+
+	checkFor(t, time.Second, 10*time.Millisecond, func() error {
+		if findProxyProtoClient(t, s, clientIP, clientPort) == nil {
+			return fmt.Errorf("no client with PROXY-supplied host/port yet")
+		}
+		return nil
+	})
+}
+
+// TestClientProxyProtoV2WithRequiredTLS is the v2 counterpart to
+// TestClientProxyProtoV1WithRequiredTLS.
+func TestClientProxyProtoV2WithRequiredTLS(t *testing.T) {
+	s := runProxyProtoTLSServer(t)
+	defer s.Shutdown()
+
+	clientIP, clientPort := "203.0.113.51", uint16(54322)
+	header := buildProxyV2Header(t, clientIP, "127.0.0.1", clientPort, 4222, proxyProtoFamilyInet)
+	cr, tlsConn := connectProxyProtoTLS(t, s.Addr().String(), header)
+	defer tlsConn.Close()
+
+	_, err := tlsConn.Write([]byte("CONNECT {\"verbose\":true,\"pedantic\":false,\"protocol\":1}\r\nPING\r\n"))
+	require_NoError(t, err)
+
+	line, err := cr.ReadString('\n')
+	require_NoError(t, err)
+	require_True(t, strings.HasPrefix(line, "+OK"))
+	expectPong(t, cr)
+
+	checkFor(t, time.Second, 10*time.Millisecond, func() error {
+		if findProxyProtoClient(t, s, clientIP, clientPort) == nil {
+			return fmt.Errorf("no client with PROXY-supplied host/port yet")
+		}
+		return nil
+	})
+}
+
+func TestClientProxyProtoNonProxiedTLSFirstFallback(t *testing.T) {
+	tc := &TLSConfigOpts{
+		CertFile: "../test/configs/certs/server-cert.pem",
+		KeyFile:  "../test/configs/certs/server-key.pem",
+		CaFile:   "../test/configs/certs/ca.pem",
+	}
+	tlsConfig, err := GenTLSConfig(tc)
+	require_NoError(t, err)
+
+	opts := DefaultOptions()
+	opts.Port = -1
+	opts.ProxyProtocol = true
+	opts.TLSConfig = tlsConfig
+	opts.TLSTimeout = 2
+	opts.TLSHandshakeFirst = true
+	opts.TLSHandshakeFirstFallback = 250 * time.Millisecond
+	s := RunServer(opts)
+	defer s.Shutdown()
+
+	rawConn, err := net.Dial("tcp", s.Addr().String())
+	require_NoError(t, err)
+	defer rawConn.Close()
+	require_NoError(t, rawConn.SetDeadline(time.Now().Add(5*time.Second)))
+
+	// Direct TLS-first client: start the handshake right away (within the
+	// fallback delay), no PROXY header.
+	tlsConn := tls.Client(rawConn, proxyProtoTLSClientConfig(t))
+	require_NoError(t, tlsConn.Handshake())
+
+	cr := bufio.NewReader(tlsConn)
+	infoLine, err := cr.ReadString('\n')
+	require_NoError(t, err)
+	require_True(t, strings.HasPrefix(infoLine, "INFO "))
+
+	_, err = tlsConn.Write([]byte("CONNECT {\"verbose\":true,\"pedantic\":false,\"protocol\":1}\r\nPING\r\n"))
+	require_NoError(t, err)
+
+	line, err := cr.ReadString('\n')
+	require_NoError(t, err)
+	require_True(t, strings.HasPrefix(line, "+OK"))
+	expectPong(t, cr)
+}
+
+// runProxyProtoAllowNonTLSServer starts a server with proxy_protocol and a
+// TLS listener that also allows non-TLS clients (allow_non_tls).
+func runProxyProtoAllowNonTLSServer(t *testing.T) *Server {
+	t.Helper()
+	tc := &TLSConfigOpts{
+		CertFile: "../test/configs/certs/server-cert.pem",
+		KeyFile:  "../test/configs/certs/server-key.pem",
+		CaFile:   "../test/configs/certs/ca.pem",
+	}
+	tlsConfig, err := GenTLSConfig(tc)
+	require_NoError(t, err)
+
+	opts := DefaultOptions()
+	opts.Port = -1
+	opts.ProxyProtocol = true
+	opts.TLSConfig = tlsConfig
+	opts.TLSTimeout = 2
+	opts.AllowNonTLS = true
+	return RunServer(opts)
+}
+
+// testClientProxyProtoAllowNonTLS verifies that with proxy_protocol and
+// allow_non_tls, the TLS-vs-plaintext detection is performed on the first
+// byte of the client's actual traffic, not on the first byte of the PROXY
+// header. Both a TLS client and a plaintext client behind the proxy must
+// be able to connect on the same listener.
+func testClientProxyProtoAllowNonTLS(t *testing.T, header []byte, clientIP string, clientPort uint16) {
+	t.Helper()
+	s := runProxyProtoAllowNonTLSServer(t)
+	defer s.Shutdown()
+
+	// TLS client behind the proxy: the sniff must detect the TLS
+	// ClientHello that follows the PROXY header.
+	cr, tlsConn := connectProxyProtoTLS(t, s.Addr().String(), header)
+	defer tlsConn.Close()
+
+	_, err := tlsConn.Write([]byte("CONNECT {\"verbose\":true,\"pedantic\":false,\"protocol\":1}\r\nPING\r\n"))
+	require_NoError(t, err)
+
+	line, err := cr.ReadString('\n')
+	require_NoError(t, err)
+	require_True(t, strings.HasPrefix(line, "+OK"))
+	expectPong(t, cr)
+
+	checkFor(t, time.Second, 10*time.Millisecond, func() error {
+		if findProxyProtoClient(t, s, clientIP, clientPort) == nil {
+			return fmt.Errorf("no client with PROXY-supplied host/port yet")
+		}
+		return nil
+	})
+
+	// Plaintext client behind the proxy must still be allowed.
+	rawConn, err := net.Dial("tcp", s.Addr().String())
+	require_NoError(t, err)
+	defer rawConn.Close()
+	require_NoError(t, rawConn.SetDeadline(time.Now().Add(5*time.Second)))
+
+	_, err = rawConn.Write(header)
+	require_NoError(t, err)
+
+	plainReader := bufio.NewReader(rawConn)
+	infoLine, err := plainReader.ReadString('\n')
+	require_NoError(t, err)
+	require_True(t, strings.HasPrefix(infoLine, "INFO "))
+
+	_, err = rawConn.Write([]byte("CONNECT {\"verbose\":true,\"pedantic\":false,\"protocol\":1}\r\nPING\r\n"))
+	require_NoError(t, err)
+
+	line, err = plainReader.ReadString('\n')
+	require_NoError(t, err)
+	require_True(t, strings.HasPrefix(line, "+OK"))
+	expectPong(t, plainReader)
+}
+
+// TestClientProxyProtoV1WithAllowNonTLS verifies that a proxied TLS client is
+// detected as TLS (and a proxied plaintext client as non-TLS) when the
+// listener has both tls and allow_non_tls configured, using a PROXY v1 header.
+func TestClientProxyProtoV1WithAllowNonTLS(t *testing.T) {
+	clientIP, clientPort := "203.0.113.60", uint16(54331)
+	header := buildProxyV1Header(t, "TCP4", clientIP, "127.0.0.1", clientPort, 4222)
+	testClientProxyProtoAllowNonTLS(t, header, clientIP, clientPort)
+}
+
+// TestClientProxyProtoV2WithAllowNonTLS is the v2 counterpart to
+// TestClientProxyProtoV1WithAllowNonTLS.
+func TestClientProxyProtoV2WithAllowNonTLS(t *testing.T) {
+	clientIP, clientPort := "203.0.113.61", uint16(54332)
+	header := buildProxyV2Header(t, clientIP, "127.0.0.1", clientPort, 4222, proxyProtoFamilyInet)
+	testClientProxyProtoAllowNonTLS(t, header, clientIP, clientPort)
+}

server/closed_conns_test.go

@@ -0,0 +1,270 @@
+// Copyright 2018-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"fmt"
+	"net"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/nats-io/nats.go"
+)
+
+func checkClosedConns(t *testing.T, s *Server, num int, wait time.Duration) {
+	t.Helper()
+	checkFor(t, wait, 5*time.Millisecond, func() error {
+		if nc := s.closed.len(); nc != num {
+			return fmt.Errorf("Closed conns expected to be %v, got %v", num, nc)
+		}
+		return nil
+	})
+}
+
+func checkTotalClosedConns(t *testing.T, s *Server, num uint64, wait time.Duration) {
+	t.Helper()
+	checkFor(t, wait, 5*time.Millisecond, func() error {
+		if nc := s.closed.totalConns(); nc != num {
+			return fmt.Errorf("Total closed conns expected to be %v, got %v", num, nc)
+		}
+		return nil
+	})
+}
+
+func TestClosedConnsAccounting(t *testing.T) {
+	opts := DefaultOptions()
+	opts.MaxClosedClients = 10
+	opts.NoSystemAccount = true
+
+	s := RunServer(opts)
+	defer s.Shutdown()
+
+	wait := time.Second
+
+	nc, err := nats.Connect(fmt.Sprintf("nats://%s:%d", opts.Host, opts.Port))
+	if err != nil {
+		t.Fatalf("Error on connect: %v", err)
+	}
+	id, _ := nc.GetClientID()
+	nc.Close()
+
+	checkClosedConns(t, s, 1, wait)
+
+	conns := s.closed.closedClients()
+	if lc := len(conns); lc != 1 {
+		t.Fatalf("len(conns) expected to be %d, got %d\n", 1, lc)
+	}
+	if conns[0].Cid != id {
+		t.Fatalf("Expected CID to be %d, got %d\n", id, conns[0].Cid)
+	}
+
+	// Now create 21 more
+	for i := 0; i < 21; i++ {
+		nc, err = nats.Connect(fmt.Sprintf("nats://%s:%d", opts.Host, opts.Port))
+		if err != nil {
+			t.Fatalf("Error on connect: %v", err)
+		}
+		nc.Close()
+		checkTotalClosedConns(t, s, uint64(i+2), wait)
+	}
+
+	checkClosedConns(t, s, opts.MaxClosedClients, wait)
+	checkTotalClosedConns(t, s, 22, wait)
+
+	conns = s.closed.closedClients()
+	if lc := len(conns); lc != opts.MaxClosedClients {
+		t.Fatalf("len(conns) expected to be %d, got %d\n",
+			opts.MaxClosedClients, lc)
+	}
+
+	// Set it to the start after overflow.
+	cid := uint64(22 - opts.MaxClosedClients)
+	for _, ci := range conns {
+		cid++
+		if ci.Cid != cid {
+			t.Fatalf("Expected cid of %d, got %d\n", cid, ci.Cid)
+		}
+	}
+}
+
+func TestClosedConnsSubsAccounting(t *testing.T) {
+	opts := DefaultOptions()
+	s := RunServer(opts)
+	defer s.Shutdown()
+
+	url := fmt.Sprintf("nats://%s:%d", opts.Host, opts.Port)
+
+	nc, err := nats.Connect(url)
+	if err != nil {
+		t.Fatalf("Error on subscribe: %v", err)
+	}
+	defer nc.Close()
+
+	// Now create some subscriptions
+	numSubs := 10
+	for i := 0; i < numSubs; i++ {
+		subj := fmt.Sprintf("foo.%d", i)
+		nc.Subscribe(subj, func(m *nats.Msg) {})
+	}
+	nc.Flush()
+	nc.Close()
+
+	checkClosedConns(t, s, 1, 20*time.Millisecond)
+	conns := s.closed.closedClients()
+	if lc := len(conns); lc != 1 {
+		t.Fatalf("len(conns) expected to be 1, got %d\n", lc)
+	}
+	ci := conns[0]
+
+	if len(ci.subs) != numSubs {
+		t.Fatalf("Expected number of Subs to be %d, got %d\n", numSubs, len(ci.subs))
+	}
+}
+
+func checkReason(t *testing.T, reason string, expected ClosedState) {
+	if !strings.Contains(reason, expected.String()) {
+		t.Fatalf("Expected closed connection with `%s` state, got `%s`\n",
+			expected, reason)
+	}
+}
+
+func TestClosedAuthorizationTimeout(t *testing.T) {
+	serverOptions := DefaultOptions()
+	serverOptions.Authorization = "my_token"
+	serverOptions.AuthTimeout = 0.4
+	s := RunServer(serverOptions)
+	defer s.Shutdown()
+
+	conn, err := net.Dial("tcp", net.JoinHostPort(serverOptions.Host, fmt.Sprintf("%d", serverOptions.Port)))
+	if err != nil {
+		t.Fatalf("Error dialing server: %v\n", err)
+	}
+	defer conn.Close()
+
+	checkClosedConns(t, s, 1, 2*time.Second)
+	conns := s.closed.closedClients()
+	if lc := len(conns); lc != 1 {
+		t.Fatalf("len(conns) expected to be %d, got %d\n", 1, lc)
+	}
+	checkReason(t, conns[0].Reason, AuthenticationTimeout)
+}
+
+func TestClosedAuthorizationViolation(t *testing.T) {
+	serverOptions := DefaultOptions()
+	serverOptions.Authorization = "my_token"
+	s := RunServer(serverOptions)
+	defer s.Shutdown()
+
+	opts := s.getOpts()
+	url := fmt.Sprintf("nats://%s:%d", opts.Host, opts.Port)
+
+	nc, err := nats.Connect(url)
+	if err == nil {
+		nc.Close()
+		t.Fatal("Expected failure for connection")
+	}
+
+	checkClosedConns(t, s, 1, 2*time.Second)
+	conns := s.closed.closedClients()
+	if lc := len(conns); lc != 1 {
+		t.Fatalf("len(conns) expected to be %d, got %d\n", 1, lc)
+	}
+	checkReason(t, conns[0].Reason, AuthenticationViolation)
+}
+
+func TestClosedUPAuthorizationViolation(t *testing.T) {
+	serverOptions := DefaultOptions()
+	serverOptions.Username = "my_user"
+	serverOptions.Password = "my_secret"
+	s := RunServer(serverOptions)
+	defer s.Shutdown()
+
+	opts := s.getOpts()
+	url := fmt.Sprintf("nats://%s:%d", opts.Host, opts.Port)
+
+	nc, err := nats.Connect(url)
+	if err == nil {
+		nc.Close()
+		t.Fatal("Expected failure for connection")
+	}
+
+	url2 := fmt.Sprintf("nats://my_user:wrong_pass@%s:%d", opts.Host, opts.Port)
+	nc, err = nats.Connect(url2)
+	if err == nil {
+		nc.Close()
+		t.Fatal("Expected failure for connection")
+	}
+
+	checkClosedConns(t, s, 2, 2*time.Second)
+	conns := s.closed.closedClients()
+	if lc := len(conns); lc != 2 {
+		t.Fatalf("len(conns) expected to be %d, got %d\n", 2, lc)
+	}
+	checkReason(t, conns[0].Reason, AuthenticationViolation)
+	checkReason(t, conns[1].Reason, AuthenticationViolation)
+}
+
+func TestClosedMaxPayload(t *testing.T) {
+	serverOptions := DefaultOptions()
+	serverOptions.MaxPayload = 100
+
+	s := RunServer(serverOptions)
+	defer s.Shutdown()
+
+	opts := s.getOpts()
+	endpoint := net.JoinHostPort(opts.Host, fmt.Sprintf("%d", opts.Port))
+
+	conn, err := net.DialTimeout("tcp", endpoint, time.Second)
+	if err != nil {
+		t.Fatalf("Could not make a raw connection to the server: %v", err)
+	}
+	defer conn.Close()
+
+	// This should trigger it.
+	pub := "PUB foo.bar 1024\r\n"
+	conn.Write([]byte(pub))
+
+	checkClosedConns(t, s, 1, 2*time.Second)
+	conns := s.closed.closedClients()
+	if lc := len(conns); lc != 1 {
+		t.Fatalf("len(conns) expected to be %d, got %d\n", 1, lc)
+	}
+	checkReason(t, conns[0].Reason, MaxPayloadExceeded)
+}
+
+func TestClosedTLSHandshake(t *testing.T) {
+	opts, err := ProcessConfigFile("./configs/tls.conf")
+	if err != nil {
+		t.Fatalf("Error processing config file: %v", err)
+	}
+	opts.TLSVerify = true
+	opts.NoLog = true
+	opts.NoSigs = true
+	s := RunServer(opts)
+	defer s.Shutdown()
+
+	nc, err := nats.Connect(fmt.Sprintf("tls://%s:%d", opts.Host, opts.Port))
+	if err == nil {
+		nc.Close()
+		t.Fatal("Expected failure for connection")
+	}
+
+	checkClosedConns(t, s, 1, 2*time.Second)
+	conns := s.closed.closedClients()
+	if lc := len(conns); lc != 1 {
+		t.Fatalf("len(conns) expected to be %d, got %d\n", 1, lc)
+	}
+	checkReason(t, conns[0].Reason, TLSHandshakeError)
+}

server/configs/certs/cert.new.pem

@@ -0,0 +1,22 @@
+-----BEGIN CERTIFICATE-----
+MIIDnjCCAoagAwIBAgIJAM/HacKKaH7zMA0GCSqGSIb3DQEBCwUAMIGDMQswCQYD
+VQQGEwJVUzELMAkGA1UECAwCQ0ExFDASBgNVBAcMC0xvcyBBbmdlbGVzMSQwIgYD
+VQQKDBtTeW5hZGlhIENvbW11bmljYXRpb25zIEluYy4xKzApBgNVBAMMIm5hdHMu
+aW8vZW1haWxBZGRyZXNzPWRlcmVrQG5hdHMuaW8wHhcNMTkxMDE3MTIyODIxWhcN
+MjkxMDE0MTIyODIxWjCBgzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRQwEgYD
+VQQHDAtMb3MgQW5nZWxlczEkMCIGA1UECgwbU3luYWRpYSBDb21tdW5pY2F0aW9u
+cyBJbmMuMSswKQYDVQQDDCJuYXRzLmlvL2VtYWlsQWRkcmVzcz1kZXJla0BuYXRz
+LmlvMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvBIAVUr4gCEBRVbA
+HSweCLExOrMZmII4AdvgIqT+svjBkJd+vdkbd5b/SC1HQx1E14kiRJ/JrZIZoMqi
++7pK3kFM63Fkhkg8rWOxn0tQznSymKTpha5NdDWxnB0dXlXFCQG1e/cuDalR7UhF
+LPHiuK42gAvhivBcymDPV0hTYt4rHb71SQ1DwfCYzcLkDvDFA/W7kronaEhRyWn6
+uvZvHkdvScoubdzoW/kNBH4JYZw5svLzGz3z20rUGeLttF4ge5SCAz9unZk96HUO
+EFmUDvFmxdnTXrYINjraNvgu7fPFmVzupWrWPA/7U+cxOvm3qBMdqxFxNr7bq+Md
+UPKi3QIDAQABoxMwETAPBgNVHREECDAGhwR/AAABMA0GCSqGSIb3DQEBCwUAA4IB
+AQCCgFKP/bMi5cvasJOzXKLpwOneW+oItL2t/5NxPumIBMDo5NnShzsFLfGmujYw
+fjcMharynkkbz/oeDAm8h1mySAJrnqtabiWgaW8zbNJfJOrAq4Jvs9COMAzjJclL
++h9GWtvVylDnsNtd18n1gA5OYv0A6YjuSrWINL8Sp5QvTF/5tT8jFrDOIjZl7m50
+lX4R70N9GLt2jIlKro+qdsi6qUZccuJmoQxUpG1iQcRNFHtWfDPr5KFEXaO6IoYt
+D1kYWmo/A3WQm7nbXeZw/zaSGSS0t6/hKZwm+gPCL6TEdDrjhjxpCZQTwaMd6jj2
+bvT2OA0ZpzUWyxnaX6u+cDfM
+-----END CERTIFICATE-----

server/configs/certs/key.new.pem

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC8EgBVSviAIQFF
+VsAdLB4IsTE6sxmYgjgB2+AipP6y+MGQl3692Rt3lv9ILUdDHUTXiSJEn8mtkhmg
+yqL7ukreQUzrcWSGSDytY7GfS1DOdLKYpOmFrk10NbGcHR1eVcUJAbV79y4NqVHt
+SEUs8eK4rjaAC+GK8FzKYM9XSFNi3isdvvVJDUPB8JjNwuQO8MUD9buSuidoSFHJ
+afq69m8eR29Jyi5t3Ohb+Q0EfglhnDmy8vMbPfPbStQZ4u20XiB7lIIDP26dmT3o
+dQ4QWZQO8WbF2dNetgg2Oto2+C7t88WZXO6latY8D/tT5zE6+beoEx2rEXE2vtur
+4x1Q8qLdAgMBAAECggEBAI0zJrTi2Le6D+4zN4Gvgc0c280qcmkiO1KJ9jmMjv1n
+7kvUsf9vZUPgRkG2XO5ypyD7gJLtNMnwCvXBraQ5NcSwWkPampKG4ad8Vfs23LBk
+xUH9bqZDOzuopHSFF2ugEZK1icBM0HLJUQ1JWUZpRMNLaPex8+AQnloDXSg20QRE
+Z53CDYZVEVaLAYIkfgZ6QgBSbJ0ob/a+ZJkChWs3f6cpNHige4tR2RQ0aJFwvlwu
+eC8G3RDNzUMhPf4TSf/XUlHXngyHKa7MS3feEtuKIQ4gb8hjESDcZznJUGG9qyMT
+7445mapc60t8z46o9iVeVJQmQxmjFUDkLcZ9ckcWfzECgYEA4OiolNUiTnXbvP0H
+3rxAnhx9GlyHiRaIZvgsAHH/6aMbJIXnH89cAR38PL79OaQpWZqK8Z1vrCHC8q7s
+Wb1RehgnwPxtTl4RTZ0wj3Ri5BIq32nzgla0w/5RAmAw9HrYXjp4TR4Quc+bHLcr
+ZM2gA8nZ4PG73rYvd/Mu3lCQiS8CgYEA1hGoaduGG7eHTMCct/kCupFfYXgEuUDh
+V/nO1SYJbIkMuSONNaQy1oUc5o0BB/JTcslZWqtuGa9snKPVd3/rJz75YGt1QR6u
+MukCYjbWGAWXfDh8G8aGSr3k1k3JmKNE7fI7L5DuvKCGoGjzzC+kclyoij8jwXk0
+zghaYPvM+bMCgYEA3/fwgR3p5vZJF19mqfEP7CP0lP7V3bd5qAi1UNA1h4VsrydF
+LRFCzr38hMWwx+jpYJicitU78s9AIon9RbRY4dwSIoV9mE/mrUK+q+y72eEZnpgU
+7ZPIuXCVXWdK+PsoYlWZnTo2b8ME9UiWxvBZy8wD05UGgFcu2CVsY+kYtfMCgYAs
+Vn/xXPyL5Rlq9kH/gN3l0pJU18zyqdOCq0UBtN0i08gE2K44vAejkvKHdhEOmkxa
+bAXL19H4E/OFBhICrEYCXPK928PvdvFRrh1GRmFVnGLh4bki752FAYvSL05gBQET
+36YOlhA8lWsM8m/8jKmc3kAyUh2PxxD+05AUolK0LQKBgHqPgMj5rfiR7MoVIaXr
+K8h7g+HejppmCxzENc98hvIQIO2aWeC4Hi/EMe+XYSN/Gl6d5+aw80JYRgIv6dFV
+hDEEt8lEI4sFhKD0oG1dQo+BdDNXOLbpHq+RQRPHa8J/pmSjI5Joqb2mCaAcm/KJ
+nYE20ox06YMSb+ZrjP2p4qw3
+-----END PRIVATE KEY-----

server/configs/certs/key.pem

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQDc8YaxliKNsa8M
+idwQsDXS0MI4z+ZP1UMW9U/Vt2qTienYs3L6KqezRfwEH2AoE7XyvQdzePV3aN1g
+A0L3mAKiS2FN3DhxR6Y8D/JNb5YA6Pl0flaMKnWktma/SMOf7o7eS7Zt6ZmIvtBg
+p1ZtRSZSH64KSqC467eGByHnXm25Z7Uf/2rO8K1jsmRwgDEJoJL5Z4va4pypX5Kq
+7OxWk8Za50Kuk6DtdRfIcnUjoDadsYzGHzF7kp1/31elQ2aUPp9e6X3aWu++gJv5
+brBMsWeQ+ETmjCMPqxoXDM2AJqwP73irELU7POUR/q3gooqy16BvTp7TaLvAtKdP
+SK7kl1QVAgMBAAECggEAKVT1gZs4c5Fd0daoWGbeasWqz730s9KNbmoNuUlKorcW
+C/fl9m6sWJkrAApujpDBZNt/3VYvdVskrqVtjaveMkYvucUMugRRUMMa7OmHAjyT
+DfvkbOw0fc+xgO83yV9zUpqPUhh1JGOtz6b+QArDlHFguYQkdPODrYsACKxdkmU/
+A+ZJkC2Sdn0L7ARmROxvoelxV3HNbC+p8lqrIVarpni8oeDF2JxQfyJETl3Df099
+/KA/IQvXytEQBJpguh0j7sK81cI40TKO3kNWsCe46S2nkCuEeEums7BqAspmdNMS
+gBW0rpFA9Ep5AyT4+A2XhWFQAwf/PI26JUVDhbClAQKBgQD/XlONHf5tNiOEtW88
+Whd9ha7oLgoBhr4gFOcPM4MUStRDftRXY1ty0IGseV0Rj1xQRUp31Pl6Kep/udy1
+r/6ML/IQNaht1VJOf2+2v7oZFoLRGLvR3qSRFNVtfLWp1Dg5OPgyi00OpFnxz/6o
+iuWlh1EWwSm9vPl91HcWPPcwBQKBgQDdfWfDfy8q9jExj8tVNh5++haeemprkZxk
+QuduIkahlbQ3V+SLaSWQrwzcB4PWL+45fnIWwK3Vmm2POBv+OWfKGjJniSd4Zrrf
+H+lSg4+hpivvoaYf1TsCTHy/af/5pYIkEUmbPM0EqwubyoW6oMAgGIAMyH6XBLaY
+a4vWZo6g0QKBgFXDXhgUrLAM8JzPOk5wi1cSoI1FeQLON+gaXQdT63/TKbqJS9MV
+gU7sC8Da+ZC+LuiefMYF9ss6bJD84Mz8EGcQayFag/hvHjdSwTgE6AEo+EI1Jk7z
+kR1Qe+VLbs9cgI1nPqPq+LQkKjj1+7aq/zk6WtdjhBs+7iN+SYhkbTfpAoGAeCbv
+gz4beFLVnO1EgJU7Nea1HoOJ95CmJj2lDjnJ1x4/BUbI1FfV6QcNEs+A1VBrCwVt
+HqqnopiDlo35oY/CngBYF5JvtwEDnsbQ69IyuJ5Md1JZrCsgN78GbVAzbFo8nxRB
+udTh4wZm5byXjwZlMwQXctfQ1FvaMhUlZsl71gECgYA82osmLHxZZOiQrYJ7z302
+CmPohwcTtUASqeA0kQOX/OdFJ1IxWIFpzq2ffMipLTfzk7OeFS2hqvNo/NE8NHlR
+/79IycYwlO99T3e2yQhYKW5ZMT3Q8YtyLLPtBzPPtO/Tc8yKhjX8wc9w9sxq9kqg
+i5Z+6xfO+l/NBT/JHF4yEw==
+-----END PRIVATE KEY-----

server/configs/certs/server.pem

@@ -0,0 +1,22 @@
+-----BEGIN CERTIFICATE-----
+MIIDnjCCAoagAwIBAgIJAOXpOWk41Pd7MA0GCSqGSIb3DQEBCwUAMIGDMQswCQYD
+VQQGEwJVUzELMAkGA1UECAwCQ0ExFDASBgNVBAcMC0xvcyBBbmdlbGVzMSQwIgYD
+VQQKDBtTeW5hZGlhIENvbW11bmljYXRpb25zIEluYy4xKzApBgNVBAMMIm5hdHMu
+aW8vZW1haWxBZGRyZXNzPWRlcmVrQG5hdHMuaW8wHhcNMTkxMDE3MTIzMjQ5WhcN
+MjkxMDE0MTIzMjQ5WjCBgzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRQwEgYD
+VQQHDAtMb3MgQW5nZWxlczEkMCIGA1UECgwbU3luYWRpYSBDb21tdW5pY2F0aW9u
+cyBJbmMuMSswKQYDVQQDDCJuYXRzLmlvL2VtYWlsQWRkcmVzcz1kZXJla0BuYXRz
+LmlvMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3PGGsZYijbGvDInc
+ELA10tDCOM/mT9VDFvVP1bdqk4np2LNy+iqns0X8BB9gKBO18r0Hc3j1d2jdYANC
+95gCokthTdw4cUemPA/yTW+WAOj5dH5WjCp1pLZmv0jDn+6O3ku2bemZiL7QYKdW
+bUUmUh+uCkqguOu3hgch515tuWe1H/9qzvCtY7JkcIAxCaCS+WeL2uKcqV+Squzs
+VpPGWudCrpOg7XUXyHJ1I6A2nbGMxh8xe5Kdf99XpUNmlD6fXul92lrvvoCb+W6w
+TLFnkPhE5owjD6saFwzNgCasD+94qxC1OzzlEf6t4KKKstegb06e02i7wLSnT0iu
+5JdUFQIDAQABoxMwETAPBgNVHREECDAGhwR/AAABMA0GCSqGSIb3DQEBCwUAA4IB
+AQDIMNgMqgGdXwZTIDxz4iq76iHdSHxZK4YwJr/4xRSq1uEddxmUfAQzo+gBboA5
+c0XxYxc0xViuueThnqdLhHmEyCs8uGFRLaQAI5Bq1PdMQP12m5fCWRAyKrCxdtli
+zm8ByDq7mWpkfMTd/rJGR4wCR9qI9Y5Bp6p4FBKZ3pzEanFXMV9IHhkm1BGh9tbe
+l6GQyBptEpfTiRwNCC/ympeiL3G8hfDCPkcLed5sQ+OhPe5iWMVPncZh/qehnUJK
+B5CIXcagcROFutsDYPCurKcfQOsfqulu0q95h7FQUOsIeU7jIlcLxIii19qjlTZh
+sFjsul5G7qqMEgIUsx3U985v
+-----END CERTIFICATE-----

server/configs/certs/tls/benchmark-ca-cert.pem

@@ -0,0 +1,23 @@
+-----BEGIN CERTIFICATE-----
+MIID2zCCAsOgAwIBAgIUZj0PngA93uUSShcRndTQju/J88YwDQYJKoZIhvcNAQEL
+BQAwfDETMBEGA1UEAwwKbmF0cy5pby5DQTEiMCAGA1UECwwZUGVyZm9ybWFuY2VB
+bmRSZWxpYWJpbGl0eTEQMA4GA1UECgwHU3luYWRpYTEQMA4GA1UEBwwHVG9yb250
+bzEQMA4GA1UECAwHT250YXJpbzELMAkGA1UEBhMCQ0EwIBcNMjMwODE0MTUyNzU3
+WhgPMjEyMzA3MjExNTI3NTdaMHwxEzARBgNVBAMMCm5hdHMuaW8uQ0ExIjAgBgNV
+BAsMGVBlcmZvcm1hbmNlQW5kUmVsaWFiaWxpdHkxEDAOBgNVBAoMB1N5bmFkaWEx
+EDAOBgNVBAcMB1Rvcm9udG8xEDAOBgNVBAgMB09udGFyaW8xCzAJBgNVBAYTAkNB
+MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2cCyJL+DExUyZto2eFLm
+MBRSkQLxM9pOWB9O8TecHlPcc/SPGq/x9lpguJ/IiaUj+VffVWy236KW2JL5Xj83
+PZwhXi1yZzxlIBsKAgAUeNfWuTAc0K0Qm9pR5Wjv5eNcT0mw6JX0SPgUQAl9BSwU
+WvtMOTxOt0hBjHmZaEamp7nLmwogpvgPsrubD6U4O/vUQm3JTsbp2rFQxXPpkG19
+69PGsT37r0/w9Zv0xNAcB/zCWdNBXCTA2ACV2IpJedWm8Jrjcn3Kp4Fv3TKTsCZl
+eWtfxCdljndk88+NFK7cEw7b9Bs5R5Zhu20C+Ne8vmMWhYbVBFYws5/jGzPBkVTD
+7wIDAQABo1MwUTAdBgNVHQ4EFgQUEqfeAemfeIp4MM4C7H1bJS+mra4wHwYDVR0j
+BBgwFoAUEqfeAemfeIp4MM4C7H1bJS+mra4wDwYDVR0TAQH/BAUwAwEB/zANBgkq
+hkiG9w0BAQsFAAOCAQEAiamiPxOlZ0pwOJvv0ylDreHVk2kast67YlhAOcZoSMvi
+e2jbKL98U3+ZznGj21AKqEaOkO7UmKoJ/3QlrjgElXzcMUrUrJ1WNowlbXPlAhyL
+KhNthLKUr72Tv6wv5GZdAR6DaAwq3iYTbpnLq4oCnFHiXgDWgWyJDLsTGulWve/K
+GGM2JMcnacNgNC18uki440Wcfp0vGj9HhO6I/u63oGewZnIK87GQMQCt3JLFyiUc
+hrn9nWoixFWcJfCjBcMlwZXMIAlDdelU1/hWtSknKCs57GvZuACcicAYiYIkWCkd
+p1pF4G0Ic6irAnLTqhdGwL4+5pjNd1Ih0Gezn9hJLg==
+-----END CERTIFICATE-----

server/configs/certs/tls/benchmark-ca-key.pem

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQDZwLIkv4MTFTJm
+2jZ4UuYwFFKRAvEz2k5YH07xN5weU9xz9I8ar/H2WmC4n8iJpSP5V99VbLbfopbY
+kvlePzc9nCFeLXJnPGUgGwoCABR419a5MBzQrRCb2lHlaO/l41xPSbDolfRI+BRA
+CX0FLBRa+0w5PE63SEGMeZloRqanucubCiCm+A+yu5sPpTg7+9RCbclOxunasVDF
+c+mQbX3r08axPfuvT/D1m/TE0BwH/MJZ00FcJMDYAJXYikl51abwmuNyfcqngW/d
+MpOwJmV5a1/EJ2WOd2Tzz40UrtwTDtv0GzlHlmG7bQL417y+YxaFhtUEVjCzn+Mb
+M8GRVMPvAgMBAAECggEASvFWfm5JMtqjP6HPiGbjoV2FMzJjiFiUiSCxXzSn7wom
+v+PGEsXGTWa6jiAz+SeUc38KNtDVOa+wIfani4fPP82J8GtMyfoPxdZ4gcq8QQDr
+/k1wRWOi6Tjg4cdVdXXkMcencs0VR73V3lpFpG+Qy+VcTQCUCOF96dZ59VkHh4a4
+CHX6PegOWwr0TSaCUacwhua+rPmCar/btAYv7Wp7+c+Zf7Rn2WYTV7ol4sYXR4ZQ
+Sy/ROijeFTMkYNpaW/KOO2/pn3OJA8ycYH8UWZpsenPfIajC0Eka7BoEQpw6M8HR
+wRWrKwssBEs0psFiq9s8J+6resPgXfU/9pf+mTkTqQKBgQD8kktUqN+vYc6t22DE
+tSkg8y8OsGh9VTYfp7x5hu9qEC9t4mAKjqA/rRLiTXze/wInntreRTjjMb/NlqMy
+PvI0Z+dM1UuqcF1axgKrIYsgnLJWuunOhaj5K3LhiNcHznlCtN9601dbccwLlQhL
+5jdjnOuJ0i+Nh9v5oiu37SfldwKBgQDctWdbF4hJrBPnS6CcojuQj6ha10wYYe46
+ZVcxKe5hFBs1q975YCHnEntyCDvXGfOTRgbKeZbwNhMvAc7Pp6eGMR/9SpiRwTt4
+567hUz56WXVmp4gSvxoNuYRlWiMI8rZkyKJ8KFipvHgRa8nuamh0QBB4ShEJiVk8
+fhaUiZeTSQKBgF0hAD/OKPR9Jv06J9tARVMN+Cr9ZvnXwqY3biqNU5gTMbndv7YE
+0xfHlG/3THTZKI09aMyOT6SOQn/m7HPpe9tQ+Jt/BnBpEDMZUgCR1MAIp0WNlAp/
+hEej+q8oiskpG9M56DFc3hgsxKT8pdt+nqvPP5ZI9xnDn5vTbTVbb9uPAoGAFRvU
+csXhZwpqLOjyx4hMohrbQzTsNjjHjBY9LJqSDf7aS1vQy5ECLRN7cwCOmJgGz8MW
+yy6t3POPCiPmH74tK4xvPs5voSEWCw49j5dillkP/W1wejqEx2NC4l6okyaDg0gd
+IjrJoBJCeYgRnBfZPaUS7i5HSt40BrEYf8RZFuECgYAjSnvY8nYFRudsylYOu3TL
+AcGbAdpDfL2H4G9z7qEQC6t1LqqGNcfMan/qS4J/n5GCOedVWHcfCraROOMil52v
+3ZDXyyjGEO08XgKnoa2ZL/z2a6s077+hAAnbcywyi2Qz8Yfi6GnwYfU1u8SN0APd
+T71HPNsWkU4zkxmP4S0Olg==
+-----END PRIVATE KEY-----

server/configs/certs/tls/benchmark-server-cert-ed25519.pem

@@ -0,0 +1,19 @@
+-----BEGIN CERTIFICATE-----
+MIIDHTCCAgWgAwIBAgIUc31LCktokAIQGqLsSC2BlsLCTsYwDQYJKoZIhvcNAQEL
+BQAwfDETMBEGA1UEAwwKbmF0cy5pby5DQTEiMCAGA1UECwwZUGVyZm9ybWFuY2VB
+bmRSZWxpYWJpbGl0eTEQMA4GA1UECgwHU3luYWRpYTEQMA4GA1UEBwwHVG9yb250
+bzEQMA4GA1UECAwHT250YXJpbzELMAkGA1UEBhMCQ0EwIBcNMjMwODE0MTUyNzU3
+WhgPMjEyMzA3MjExNTI3NTdaMHkxEDAOBgNVBAMMB25hdHMuaW8xIjAgBgNVBAsM
+GVBlcmZvcm1hbmNlQW5kUmVsaWFiaWxpdHkxEDAOBgNVBAoMB1N5bmFkaWExEDAO
+BgNVBAcMB1Rvcm9udG8xEDAOBgNVBAgMB09udGFyaW8xCzAJBgNVBAYTAkNBMCow
+BQYDK2VwAyEAyyc9y9iZgWWSsPRahbeGxF6XN3VOFPZBvD/HQps6jr6jgZEwgY4w
+CwYDVR0PBAQDAgQwMBMGA1UdJQQMMAoGCCsGAQUFBwMBMCoGA1UdEQQjMCGCDnJl
+dWJlbi5uYXRzLmlvgg9yZXViZW4ubmF0cy5jb20wHQYDVR0OBBYEFBwkwMU8xuQO
+FN1Ck5o2qQ4Dz87ZMB8GA1UdIwQYMBaAFBKn3gHpn3iKeDDOAux9WyUvpq2uMA0G
+CSqGSIb3DQEBCwUAA4IBAQALjCynuxEobk1MYQAFhkrfAD29H6yRpOcKigHCZjTJ
+Dnpupip1xvaFPPvhi4nxtuWcXgKpWEfd1jOPaiNV6lrefahitZpzcflD7wNOxqvx
+Hau2U3lFnjnGaC0ppp66x26cQznp6YcTdxrJ1QF4vkOejxqNvaTzmiwzSPIIYm7+
+iKVWT+Z86WKof3vAdsX/f148YH1YSPk0ykiBzlbLScbyWebbaydrAIpU01IkSvMo
+qDYu+Fba0tpONLe1BUklc608riwQjw9HiJJ2zJIAOBAUev5+48RP91/K111Ix1bl
+fGPT8/1TJbyGG2jeJwyLoSIu72aDnnIBfqGkVunRTmeg
+-----END CERTIFICATE-----

server/configs/certs/tls/benchmark-server-cert-rsa-1024.pem

@@ -0,0 +1,22 @@
+-----BEGIN CERTIFICATE-----
+MIIDkzCCAnugAwIBAgIUc31LCktokAIQGqLsSC2BlsLCTsMwDQYJKoZIhvcNAQEL
+BQAwfDETMBEGA1UEAwwKbmF0cy5pby5DQTEiMCAGA1UECwwZUGVyZm9ybWFuY2VB
+bmRSZWxpYWJpbGl0eTEQMA4GA1UECgwHU3luYWRpYTEQMA4GA1UEBwwHVG9yb250
+bzEQMA4GA1UECAwHT250YXJpbzELMAkGA1UEBhMCQ0EwIBcNMjMwODE0MTUyNzU3
+WhgPMjEyMzA3MjExNTI3NTdaMHkxEDAOBgNVBAMMB25hdHMuaW8xIjAgBgNVBAsM
+GVBlcmZvcm1hbmNlQW5kUmVsaWFiaWxpdHkxEDAOBgNVBAoMB1N5bmFkaWExEDAO
+BgNVBAcMB1Rvcm9udG8xEDAOBgNVBAgMB09udGFyaW8xCzAJBgNVBAYTAkNBMIGf
+MA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCyHHaVHinB3jBsicR4hp7uopz0u3+O
+kUicIUSQXDcWiPzdvE+7YZ/s4+Ud4aw4g9q0wHzkZSaMg8nil4tCKmTrUKolVTVj
+CCCBmtqq3LwzNLapyoDJRyXsWqHt5TWYSxaf/UQT6sWOgqHOLrbd4J8F0sjxEniB
+GDHR1ZXpJCBaIQIDAQABo4GRMIGOMAsGA1UdDwQEAwIEMDATBgNVHSUEDDAKBggr
+BgEFBQcDATAqBgNVHREEIzAhgg5yZXViZW4ubmF0cy5pb4IPcmV1YmVuLm5hdHMu
+Y29tMB0GA1UdDgQWBBQk5kWOUcUNn7FppddLANe3droUlzAfBgNVHSMEGDAWgBQS
+p94B6Z94ingwzgLsfVslL6atrjANBgkqhkiG9w0BAQsFAAOCAQEA2Njy2f1PUZRf
+G1/oZ0El7J8L6Ql1HmEC7tOTzbORg7U9uMHKqIFL/IXXAdAlE/EjFEA2riPO8cu/
+bvL2A4CapYzt2kDD9PPYfVtniRr7mv0EVntPwEvfiySMAEeZuW/M2liPfgPpQkhL
+fzwPeCOfqM8AjpyDab8NEGX5Bbf421oQorlENpm4PKQCXoUN5cWpBwuwWxj7yndj
+256MevLDKKe/ALSLQEo/2Jgpnmp7Qol0GtomCzsLgZ+ASuVtCsGTFmaRrsqVPspJ
+oOl6qby5gYwN9TR8zfRYL1m1sbYROz+5+ofEoiTnaOoOSjiBIoYoMeSC/jvJQTPT
+VdD8QeQ6Og==
+-----END CERTIFICATE-----

server/configs/certs/tls/benchmark-server-cert-rsa-2048.pem

@@ -0,0 +1,24 @@
+-----BEGIN CERTIFICATE-----
+MIIEFzCCAv+gAwIBAgIUc31LCktokAIQGqLsSC2BlsLCTsQwDQYJKoZIhvcNAQEL
+BQAwfDETMBEGA1UEAwwKbmF0cy5pby5DQTEiMCAGA1UECwwZUGVyZm9ybWFuY2VB
+bmRSZWxpYWJpbGl0eTEQMA4GA1UECgwHU3luYWRpYTEQMA4GA1UEBwwHVG9yb250
+bzEQMA4GA1UECAwHT250YXJpbzELMAkGA1UEBhMCQ0EwIBcNMjMwODE0MTUyNzU3
+WhgPMjEyMzA3MjExNTI3NTdaMHkxEDAOBgNVBAMMB25hdHMuaW8xIjAgBgNVBAsM
+GVBlcmZvcm1hbmNlQW5kUmVsaWFiaWxpdHkxEDAOBgNVBAoMB1N5bmFkaWExEDAO
+BgNVBAcMB1Rvcm9udG8xEDAOBgNVBAgMB09udGFyaW8xCzAJBgNVBAYTAkNBMIIB
+IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAyx3O+Z6u8Y1SiuHu3szWbLvL
+WrZpSpEiZkll+wk5205S1FRcQLccfr4ubdtjOBdi+RzILCtkflUI01Dbqu6cV7/2
+yfLthxBeNDiXMhjyOFkYLwwE4w7CdTwWWsmW31oUH1rYXIDPoeb7WPF7w3NwaUJu
+ZXnqM98LRgWDTmh+nsqDDW/bz1fYIdxcO9az6iBOnJ2AGWI2ur5GzWc4+gNMOZiZ
+Xj657g0MbyVM4Gzyc4Au22hShZ/YorLP8NAiwNJamlrCFzlnZN/ePjuQPcI6glnb
+oO9IAGfPdAOJptfayuPAZgUngzewB38yY0Q/rKG1GJKSkQ8X6/lXiWaRPZJjYwID
+AQABo4GRMIGOMAsGA1UdDwQEAwIEMDATBgNVHSUEDDAKBggrBgEFBQcDATAqBgNV
+HREEIzAhgg5yZXViZW4ubmF0cy5pb4IPcmV1YmVuLm5hdHMuY29tMB0GA1UdDgQW
+BBRtanJZScdSlsPsi58lBcpdj+bV/zAfBgNVHSMEGDAWgBQSp94B6Z94ingwzgLs
+fVslL6atrjANBgkqhkiG9w0BAQsFAAOCAQEAV4TZ3b8cYO7ZeRyoCQtCBAab9gNe
+kbQpWqICvkVQOk5Anq3opwAWk2FuIRs5KoT7ssckHpXwTwWLs+KuIVo+Fet19IH6
+BQfck1jwhzM04MA6zLO/F2j548XlrJy3IzViPM/VxwMMTt5YSoogrz/3TzzJPIe0
+eQomf5HbpVgrf08pMVkdaI7PCd7N/CxeWiD5zEWqBu9FqofO188Kb/umx0VwgBju
+dX46MKO5TyUc91UrG3M35/r4Z7fd52SWWWFQiI7UBOl2L27samjHlJsKjyFoBF3Z
+alvnoUVzo7zwAYmhEdPYDNVceF4KtAFpGipoQPRMg83G87LgYBA4Sa6uKw==
+-----END CERTIFICATE-----

server/configs/certs/tls/benchmark-server-cert-rsa-4096.pem

@@ -0,0 +1,30 @@
+-----BEGIN CERTIFICATE-----
+MIIFFzCCA/+gAwIBAgIUc31LCktokAIQGqLsSC2BlsLCTsUwDQYJKoZIhvcNAQEL
+BQAwfDETMBEGA1UEAwwKbmF0cy5pby5DQTEiMCAGA1UECwwZUGVyZm9ybWFuY2VB
+bmRSZWxpYWJpbGl0eTEQMA4GA1UECgwHU3luYWRpYTEQMA4GA1UEBwwHVG9yb250
+bzEQMA4GA1UECAwHT250YXJpbzELMAkGA1UEBhMCQ0EwIBcNMjMwODE0MTUyNzU3
+WhgPMjEyMzA3MjExNTI3NTdaMHkxEDAOBgNVBAMMB25hdHMuaW8xIjAgBgNVBAsM
+GVBlcmZvcm1hbmNlQW5kUmVsaWFiaWxpdHkxEDAOBgNVBAoMB1N5bmFkaWExEDAO
+BgNVBAcMB1Rvcm9udG8xEDAOBgNVBAgMB09udGFyaW8xCzAJBgNVBAYTAkNBMIIC
+IjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAj/rw0WpCnizUSs06NxSsjuYb
+6sWtYQjLd6O6SCoQwrPSY2Zv0u28RywBJkSiIbv12kWxrhqv+kDqhmt7tfwEdHAE
+gOl6E2P/CFDnyAJ4it5qgFRHcEItGp1Ap4ZrZ3OTl41DDe28giflHb7+VAzxUV6r
+zkHmrqkaJd41YrJoKyZ3u+/FHs6CudO+jGXC2ubH1i71ARXHn6tkmOTwFf1N49wB
+Hr144xBmPyvH/A/elgXBjR28/7x64MO/qBZsr/lbTaS4YwN1+rod/HRc2GIiJxpS
+tB1bh/dXZCfa0QyhTzCNLG/j7IzetDrPBGzpw2WjufhkSuaxoMDWlDkkfqmQxtHJ
+5L+PqIiPT69tzEbfuS9Ogz7DW10CcpSdXW13sWCdSNCGEvPqLFka36q9B82V0GHz
+tmx8VqdfWSu4qMyVmTsxxzLTTxpQU4X1Q2RnT/1igbsOM620LuvrvnXlh5Rdg7tp
+T++QJ/b4xCCg62tv4VwORe27xHYXMeDn4aoRdyoI45/+ZK6yqwNAOrKKpse/M8uz
+mJK2i8pfEFmitIKoNYn3MR2dFrCqifZkFf9rX9A/1Ym+WKAPLmWdGp13fvTdzxQG
+Y44f9tBL2RWsoGX++01XEwIiWz7kqObC0L8fz3EdIPaULX7MZiQrxzzhRCcJhyn/
+aOrJfLYj0GAmIaHElHkCAwEAAaOBkTCBjjALBgNVHQ8EBAMCBDAwEwYDVR0lBAww
+CgYIKwYBBQUHAwEwKgYDVR0RBCMwIYIOcmV1YmVuLm5hdHMuaW+CD3JldWJlbi5u
+YXRzLmNvbTAdBgNVHQ4EFgQUbJSM8LNWmc9IgLe0X53yE3c3h1cwHwYDVR0jBBgw
+FoAUEqfeAemfeIp4MM4C7H1bJS+mra4wDQYJKoZIhvcNAQELBQADggEBAEGmLvEE
++MTE1bHMbl/5QG+/xusmervIuxkfAfId0H+8TWB75y+yhVZpEdM7knfl+lexmtGQ
+Gr4HNGTZhAZ3NYFaBw7nfeqO48He7gHUKfJA/zv7FREF3Fy+Qe/hydDJQzBzZfaU
+64XqhY6jOurpZhTAoOXfjpYzZaLi7+rdpTAbfxHCCAC8SxZD3++Q97ZeoT6en47O
+8SQQ7FIzWxs15k88oYalw51vZujxX7dz4l+LxsLXtlYW7ZM1163cgU7lF/jQqDcN
+z8X8jk1AjQY7AuFPuOzQ1hLXcZySm8rUG5pPgHrZ1QKmkFFWRaeCiO2hU794wI5C
+vIGR8lIhkNwEJnQ=
+-----END CERTIFICATE-----

server/configs/certs/tls/benchmark-server-key-ed25519.pem

@@ -0,0 +1,3 @@
+-----BEGIN PRIVATE KEY-----
+MC4CAQAwBQYDK2VwBCIEIJRCtUNxUuutNs9j8OtcwFw1xkbs+zxjHhpAqVuqDNo5
+-----END PRIVATE KEY-----

server/configs/certs/tls/benchmark-server-key-rsa-1024.pem

@@ -0,0 +1,16 @@
+-----BEGIN PRIVATE KEY-----
+MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBALIcdpUeKcHeMGyJ
+xHiGnu6inPS7f46RSJwhRJBcNxaI/N28T7thn+zj5R3hrDiD2rTAfORlJoyDyeKX
+i0IqZOtQqiVVNWMIIIGa2qrcvDM0tqnKgMlHJexaoe3lNZhLFp/9RBPqxY6Coc4u
+tt3gnwXSyPESeIEYMdHVlekkIFohAgMBAAECgYAwt8RfyV5WnvXT2mMZLIlwcJ5J
++rdLQcYAnsDoU7DlwxaXeBi/AlcCLtvOrpmy464A3t3KgzhmGu4vwo/ey0XK+nTQ
+tzORP/PXTaVC8DzJ8PnJmUaB7l+H7a88OSPLbjgnbpw4SyvDpKUHiiw0EDYC7L6Y
+1vvCOlnprptXbE5eeQJBAOpjwdBVWkVtmStjsxbxZsUTI7XKxS2VZinRLH0l5/hI
+hIHRxwy9oRbeNrf5815lGolTUD0mq+N0dJRlMop1yKsCQQDCiGDkH/pQqhB8ibmD
+0XNw0EzxJmPFACO/x49VCfCPE5p1FQhpyIl6JkyAFNN7Xs4HX8jMHTuvNgJVti61
+O0BjAkEAj0wr2vXDubyWrztF61nszcG0zFjKkeLL0fcLLvv0xQt4z3F0MyrgCH4U
+kAflLSm8voZMAQbagbXZ7DuuWY5G/wJAWyKnOdidXZL+3ElthwrmKVD86vEQRqe1
+F9C3HqDkeTM25mkvItfXSEmPB2Y6WY7luOCv4qhDYOdNmrgaE7+pfwJAcbV5ZVJW
+OZvH1ofsJVvUA8J58tzv1+KPb96pI3YRAu8xbMC0mzezPsYjg2wjaRgJ2C+7On27
+BaArNo75B20AkA==
+-----END PRIVATE KEY-----

server/configs/certs/tls/benchmark-server-key-rsa-2048.pem

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDLHc75nq7xjVKK
+4e7ezNZsu8tatmlKkSJmSWX7CTnbTlLUVFxAtxx+vi5t22M4F2L5HMgsK2R+VQjT
+UNuq7pxXv/bJ8u2HEF40OJcyGPI4WRgvDATjDsJ1PBZayZbfWhQfWthcgM+h5vtY
+8XvDc3BpQm5leeoz3wtGBYNOaH6eyoMNb9vPV9gh3Fw71rPqIE6cnYAZYja6vkbN
+Zzj6A0w5mJlePrnuDQxvJUzgbPJzgC7baFKFn9iiss/w0CLA0lqaWsIXOWdk394+
+O5A9wjqCWdug70gAZ890A4mm19rK48BmBSeDN7AHfzJjRD+sobUYkpKRDxfr+VeJ
+ZpE9kmNjAgMBAAECggEADicdEWKfsQAWZMv6X3bpZ/kr4b29F2+GdJcfrn7Fk8Tg
+25+nL/EyYJhy1r/HKZTjlhUN05oQbgcRztue+smWhjy/fvHY4CThU4Uv79MyKX/3
+wet1+bZBEXcm3ZuXUifOKCMl2Ug2b4MPN3LYG1XTWtpAo/x7N7MOb4oZzKBWVk0J
+GEfxvJVZyVe3libIN2WWZFQJ1620AxaWP2jZ83PhRR/TJIBiq+pI/lBIyt+nu8Z3
+3vzB020R2uvENOhnaDjzNVfnJxSvlAAQkN47zo2Tf4UnB0792Y7DEEjFKX4XCFVd
+zxmjmw7VcRcnyruDhCoRC6mNraaAHuMqPwuBoC6GtQKBgQD/7CV8XKzcPPfFJY9e
+aHPzgXJwK+5u3jq7tNYUksVfv0s2lLQnRdbqAhHzxNYLQjdVd7J6t55h2z8scYaP
+oB7TTwszhKZS2sQ/lcpfOFoFN6KjN0iOnXVFucGoQ36gexNqPw894NFKWX/RHrnZ
+UfL/OnOUPpra3w+WMxjUYQ5ivQKBgQDLLZDIpM8fSRKeqZzjv/eoKnoDhqmzPipj
+bvNXAkIr/nWfUHL9YRnpxX7PW8DqFWIYoM8b0uOKO4vLJnQUMyc2jPq1rPXEBrjk
+w+xKWCipKdPrqwttiLKme+ZArRT5CJ9qcqxSX0yNp69xwXtyHe+zY18F/a4+70wT
+5wJwYmhQnwKBgHpICTk8xtOMxg6K/c/sNMr65QU32Htc789Ufp3h6zDupC92KgZB
+1oiFaLKDMIq8ntfVk5ATQDgdnDfOHq9toIzyzbVWAmrAYNjI56NLt6eah7lY5vBN
+yAUC1sdhSJXBeOthKhU04IuX6/yto7t07piJA0SoDTHbNwVbcNe5cDg5AoGBALjR
+jxVlDd+4mc5oHYXy1rZLRUg10+JvlyFyCLrKHCVmx9oO1Tr1fBvxggPfw+FraBtd
+FGiL8l2JAwXdydOiIHZ30Ys3dSxGrSOzsRqDjSEsIlEK+088/L2CkRWeHCjYliK/
+g08+zyVANtC0nrVU0/mLWCHb/AfVp4+nIMnYSmmjAoGAEMyBqq2AyUmx1xAmhw36
+LqgKy+vgHEAFRFPD8IttHFLOlUdXlvxoDq4xW2a7bJsJrs9ZrluRFKVh7QnSckmP
+Jt/Plg+XYB3B2exD5Xyh9xNYNVW/Aqvg+NuiWeCGK/o7mUfGWd9qWrD2aw51m+X3
+Svtkgck1kulqPoUFG1b3R4k=
+-----END PRIVATE KEY-----

server/configs/certs/tls/benchmark-server-key-rsa-4096.pem

@@ -0,0 +1,52 @@
+-----BEGIN PRIVATE KEY-----
+MIIJQwIBADANBgkqhkiG9w0BAQEFAASCCS0wggkpAgEAAoICAQCP+vDRakKeLNRK
+zTo3FKyO5hvqxa1hCMt3o7pIKhDCs9JjZm/S7bxHLAEmRKIhu/XaRbGuGq/6QOqG
+a3u1/AR0cASA6XoTY/8IUOfIAniK3mqAVEdwQi0anUCnhmtnc5OXjUMN7byCJ+Ud
+vv5UDPFRXqvOQeauqRol3jVismgrJne778UezoK5076MZcLa5sfWLvUBFcefq2SY
+5PAV/U3j3AEevXjjEGY/K8f8D96WBcGNHbz/vHrgw7+oFmyv+VtNpLhjA3X6uh38
+dFzYYiInGlK0HVuH91dkJ9rRDKFPMI0sb+PsjN60Os8EbOnDZaO5+GRK5rGgwNaU
+OSR+qZDG0cnkv4+oiI9Pr23MRt+5L06DPsNbXQJylJ1dbXexYJ1I0IYS8+osWRrf
+qr0HzZXQYfO2bHxWp19ZK7iozJWZOzHHMtNPGlBThfVDZGdP/WKBuw4zrbQu6+u+
+deWHlF2Du2lP75An9vjEIKDra2/hXA5F7bvEdhcx4OfhqhF3Kgjjn/5krrKrA0A6
+soqmx78zy7OYkraLyl8QWaK0gqg1ifcxHZ0WsKqJ9mQV/2tf0D/Vib5YoA8uZZ0a
+nXd+9N3PFAZjjh/20EvZFaygZf77TVcTAiJbPuSo5sLQvx/PcR0g9pQtfsxmJCvH
+POFEJwmHKf9o6sl8tiPQYCYhocSUeQIDAQABAoICAAEYUQ4KqeyJiJN0DvIIdc0q
+v1dwVEKlaOS6nvSkYJcWe9kL2E8JRfy1lIxTCneethPdoqhhZWljZn/fX+QRAVir
+BGxq5NCHxWgDCNble+oJydNlhgXldEcxnyJXBvM/trA49Q5X+pkeeWMvRsBiuSVx
+BqCi3CtZDQzw18TtzessF6OF2IuE2bYqP9anVyLH4rOvN5JKn9zH69PaLorq76rL
+YMp4JEFNKIs+R5RUFjwxBC6Q+r+9fM1qTQdtJOZMC293pfusylAoll315ZczXIah
+xYi/ThOmjtMrwWyEan1PcCIVzMI0X7o3q0eMQNU0ApmBY9k0+sYEnsJ8Um8l1icb
+WE/KBDzQk2mSoIzEdo99gYiuLAhi5WTEcQzeovZKHKkhRxLn3GUK/ZznpTuI0Voh
+1Qbfw05wf62yh8/nc747n5ULPgf8/nUH5G8jUeSis9gMGtFloqIxUEBphzIHwj7W
+FUIKoKpGkHL4Jkg9wXIe7lg5Qp+cdCrkWKKvQc57EbWxsM1Pjktw6X5ctNQclWG0
+WQJUw+BJhq6InKAV4aDW4/QlCOjvGtDDkYvwghE+ZgIzm9BARM4+DbeoPTmAkwwz
+6NTM8DYWvdcYVpgchEwphS6OFkOD1Dc6Wg2OUUtLjvl4NVf/NDYN8wIWNwmMm4aj
++Is12NwoUqZ44E4pSfohAoIBAQDLloatDuObuLzInBqER2tByPNnDV9AVTR2rm6w
+TgBmslFHBCf/uSzq0ivh1oZmFfDhcqdrtheTL0BWXbFxR/GelDm9N4d/LqOGRyWV
+Qmy5dGp2wE6gsZGZuvsbVst6eFO+zgDLvDMLrVqP+577g/QbrSR8gPz39WCgLpvf
+RcNpM2rHeKkm09IdUe0G+YEnW3fD7rxtMS+j8J57yuuxLO+SemGJuJQqGHYOukxF
+KMZxX2wMs1YMNwKEXY0/9M+LAtW9J3azkRp5xd+E2kzXGoWcl2nS/3+J6nh1+HWq
+SZE7SDWSimzut/aPTOM7YxLVqtANWHzVBBNpTbOMSrSZTZQJAoIBAQC1C/afOROb
+GQMb1LOt95AxvGyYo62mOkcPxhxQGojVIi+Yz8Q1ZWTuT5qyyWSofR3bE9mQwQGW
+KQ/znJKiwA48BS0osnmQi3Bi6eVD73NfwxMkL6zrGFxf336DNQi9mbidJvOaHNsT
+wXGPqRQXDcLa3s3WqerDzKJ2gxG22rtkoC0uCw+J9NlkJTTI9bJ9rjDZIh6mE/M0
+3ye19IgkBdbMv7FSjGVpovdqWZ2HELDYAXJftzuMPUO1GNYbyMHAHnWH0a1TQRM8
+ELHzmPRaFBeegmpBVaenlxKoi72iwqrVFrEr5FLAKxLq/9RwizJ8FzySIpH/5+hU
+Ky4mG93lmHjxAoIBAQDLOXs+jTpPW923M3yUxuYeSQYPvJ10jplMT1tWysZDvYS8
+qz1yW9qmnR4I1ihbB1Po+JZ/QsnNtsE2dViHiBV9AuGQLDopjtjVVXgCwsfdaIRN
+/jF+30JEfw3igIWlvy95rBHHThp2cZmRWM+eql2msvNVBT2AF4VY4K3f9rfV7+mL
+LLtNcuyvL/S3naB7NSccgte89/hiYfMSB8G2nvCW+2saGxJr4vcWRImWD9nnmiU1
+mF8w2ki88NXrHel/Dlll9FrdbN9M52T0LSW/I050vgB5C2q4tUGCIX7zeXRsBOzV
+VzDeKu0Ipuu9gGxwtY3xhH839FWcLGAqjvgwf+xhAoIBAQCuvWs9ZoNr0QpVFEiQ
+Aj9kIa7W7DOwGtN3gAjXr6Sdwa8a2H1R5Bk0ghSXtxW2IXxtdI0qz35OhjdlM5u8
+BY43k+9wNkJqporEjWfA2B4NMWUKKhHFnu+ZgUbEMK3NAc9TrsKz3mH8gVqwA8rm
+LVwCj8UwCTQT4zBzHjI8wITZrFeu9vH6fx5LMDXwOGQcNcHj8LCQLvUv9KqJTgkQ
+a6pUWDg3qlY/TRFrzi7iq9NjyJGxnFKXGpJ8+gm9K1kFquBZRKD7l/WOpbZ7nQdK
+4dWiIdGYWanFcWSK1MUlkKn9nTdHW8oau/g4ZM+QCGmjp3HIwiEUU6rDgiG6mm7j
+KPShAoIBAG1sFn8X841038Z/sp4JzYypQjj7g9UOTkghHcJHIZzRXLiTqW3lKxbt
+Gz93DjWRxXD404hem8dOUf22VgfjB4z1mSrV5SWtLjLf5wD8gl0eUNKP2lZFLggz
+O6nHCLLzlKs2RH5pDo3c8qyjLRIfCXy0YGnr+9RErVJG+TVB/MOSgHagmkdVOYCH
+phw4EiwJ+rPFy/xm5D6+BuOyt7hw7boQsw3EHpZTyQHWObcIggjlopTX2VSG8Dx+
+/iQRTuVRVyNAhYwCuNtSh27zawWr+A40acFJsJpvkFkbZBEH1IqoCteUaiEVU1qm
+51lKgt3ZVAXuecJ1U/0u4HtC0QdEBGE=
+-----END PRIVATE KEY-----

server/configs/one.creds

@@ -0,0 +1,13 @@
+-----BEGIN NATS USER JWT-----
+eyJ0eXAiOiJKV1QiLCJhbGciOiJlZDI1NTE5LW5rZXkifQ.eyJqdGkiOiJHRUNQVEpISE1TM01DTUtMVFBHWUdBTzQ1R1E2TjZRUFlXUTRHUExBRUIzM1ZDUkpOUlZRIiwiaWF0IjoxNjE2MjQ3MjMyLCJpc3MiOiJBQlZSWktKNlo3TklNUElZSlJDSEVZRlJVTzdFTk42TldPS1FERkxGREZWUFNNMzZVUFgyVUNQUCIsIm5hbWUiOiJvbmUiLCJzdWIiOiJVRENJQkdHR0hDSkJRUE9PNFNDSkpCSFpEUjM3TlFHR0NJV01ORzJEREFLSjZXTUtCTUFLWElNTyIsIm5hdHMiOnsicHViIjp7fSwic3ViIjp7fSwic3VicyI6LTEsImRhdGEiOi0xLCJwYXlsb2FkIjotMSwidHlwZSI6InVzZXIiLCJ2ZXJzaW9uIjoyfX0.VLhSDtGZEF_jdvgmhgkdISXAt5wFMMZxxwm5w8UrsnlM1hkUvtxBlTe4IP0xJIf4xf8JOR2Bmf73xUGJKUZECQ
+------END NATS USER JWT------
+
+************************* IMPORTANT *************************
+NKEY Seed printed below can be used to sign and prove identity.
+NKEYs are sensitive and should be treated as secrets.
+
+-----BEGIN USER NKEY SEED-----
+SUAPCDMU5TSHHLWUUZSOUABJXP2GXRCZVEOVWPSVM5XRSXYGQMRRFDYNMY
+------END USER NKEY SEED------
+
+*************************************************************

server/const.go

@@ -0,0 +1,251 @@
+// Copyright 2012-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"regexp"
+	"runtime/debug"
+	"time"
+)
+
+// Command is a signal used to control a running nats-server process.
+type Command string
+
+// Valid Command values.
+const (
+	CommandStop   = Command("stop")
+	CommandQuit   = Command("quit")
+	CommandReopen = Command("reopen")
+	CommandReload = Command("reload")
+
+	// private for now
+	commandLDMode = Command("ldm")
+	commandTerm   = Command("term")
+)
+
+var (
+	// gitCommit and serverVersion injected at build.
+	gitCommit, serverVersion string
+	// trustedKeys is a whitespace separated array of trusted operator's public nkeys.
+	trustedKeys string
+	// SemVer regexp to validate the VERSION.
+	semVerRe = regexp.MustCompile(`^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$`)
+)
+
+// formatRevision formats a VCS revision string for display.
+func formatRevision(revision string) string {
+	if len(revision) >= 7 {
+		return revision[:7]
+	}
+	return revision
+}
+
+func init() {
+	// Use build info if present, it would be if building using 'go build .'
+	// or when using a release.
+	if info, ok := debug.ReadBuildInfo(); ok {
+		for _, setting := range info.Settings {
+			switch setting.Key {
+			case "vcs.revision":
+				gitCommit = formatRevision(setting.Value)
+			}
+		}
+	}
+}
+
+const (
+	// VERSION is the current version for the server.
+	VERSION = "2.15.0-dev"
+
+	// PROTO is the currently supported protocol.
+	// 0 was the original
+	// 1 maintains proto 0, adds echo abilities for CONNECT from the client. Clients
+	// should not send echo unless proto in INFO is >= 1.
+	PROTO = 1
+
+	// DEFAULT_PORT is the default port for client connections.
+	DEFAULT_PORT = 4222
+
+	// RANDOM_PORT is the value for port that, when supplied, will cause the
+	// server to listen on a randomly-chosen available port. The resolved port
+	// is available via the Addr() method.
+	RANDOM_PORT = -1
+
+	// DEFAULT_HOST defaults to all interfaces.
+	DEFAULT_HOST = "0.0.0.0"
+
+	// MAX_CONTROL_LINE_SIZE is the maximum allowed protocol control line size.
+	// 4k should be plenty since payloads sans connect/info string are separate.
+	MAX_CONTROL_LINE_SIZE = 4096
+
+	// MAX_PAYLOAD_SIZE is the maximum allowed payload size. Should be using
+	// something different if > 1MB payloads are needed.
+	MAX_PAYLOAD_SIZE = (1024 * 1024)
+
+	// MAX_PAYLOAD_MAX_SIZE is the size at which the server will warn about
+	// max_payload being too high. In the future, the server may enforce/reject
+	// max_payload above this value.
+	MAX_PAYLOAD_MAX_SIZE = (8 * 1024 * 1024)
+
+	// MAX_PENDING_SIZE is the maximum outbound pending bytes per client.
+	MAX_PENDING_SIZE = (64 * 1024 * 1024)
+
+	// DEFAULT_MAX_CONNECTIONS is the default maximum connections allowed.
+	DEFAULT_MAX_CONNECTIONS = (64 * 1024)
+
+	// TLS_TIMEOUT is the TLS wait time.
+	TLS_TIMEOUT = 2 * time.Second
+
+	// DEFAULT_TLS_HANDSHAKE_FIRST_FALLBACK_DELAY is the default amount of
+	// time for the server to wait for the TLS handshake with a client to
+	// be initiated before falling back to sending the INFO protocol first.
+	// See TLSHandshakeFirst and TLSHandshakeFirstFallback options.
+	DEFAULT_TLS_HANDSHAKE_FIRST_FALLBACK_DELAY = 50 * time.Millisecond
+
+	// AUTH_TIMEOUT is the authorization wait time.
+	AUTH_TIMEOUT = 2 * time.Second
+
+	// DEFAULT_PING_INTERVAL is how often pings are sent to clients, etc...
+	DEFAULT_PING_INTERVAL = 2 * time.Minute
+
+	// DEFAULT_PING_MAX_OUT is maximum allowed pings outstanding before disconnect.
+	DEFAULT_PING_MAX_OUT = 2
+
+	// CR_LF string
+	CR_LF = "\r\n"
+
+	// LEN_CR_LF hold onto the computed size.
+	LEN_CR_LF = len(CR_LF)
+
+	// DEFAULT_FLUSH_DEADLINE is the write/flush deadlines.
+	DEFAULT_FLUSH_DEADLINE = 10 * time.Second
+
+	// DEFAULT_HTTP_PORT is the default monitoring port.
+	DEFAULT_HTTP_PORT = 8222
+
+	// DEFAULT_HTTP_BASE_PATH is the default base path for monitoring.
+	DEFAULT_HTTP_BASE_PATH = "/"
+
+	// ACCEPT_MIN_SLEEP is the minimum acceptable sleep times on temporary errors.
+	ACCEPT_MIN_SLEEP = 10 * time.Millisecond
+
+	// ACCEPT_MAX_SLEEP is the maximum acceptable sleep times on temporary errors
+	ACCEPT_MAX_SLEEP = 1 * time.Second
+
+	// DEFAULT_ROUTE_CONNECT Route solicitation intervals.
+	DEFAULT_ROUTE_CONNECT = 1 * time.Second
+
+	// DEFAULT_ROUTE_CONNECT_MAX Route solicitation intervals (max).
+	DEFAULT_ROUTE_CONNECT_MAX = 30 * time.Second
+
+	// DEFAULT_ROUTE_RECONNECT Route reconnect delay.
+	DEFAULT_ROUTE_RECONNECT = 1 * time.Second
+
+	// DEFAULT_ROUTE_DIAL Route dial timeout.
+	DEFAULT_ROUTE_DIAL = 1 * time.Second
+
+	// DEFAULT_ROUTE_POOL_SIZE Route default pool size
+	DEFAULT_ROUTE_POOL_SIZE = 3
+
+	// DEFAULT_LEAF_NODE_RECONNECT LeafNode reconnect interval.
+	DEFAULT_LEAF_NODE_RECONNECT = time.Second
+
+	// DEFAULT_LEAF_TLS_TIMEOUT TLS timeout for LeafNodes
+	DEFAULT_LEAF_TLS_TIMEOUT = 2 * time.Second
+
+	// PROTO_SNIPPET_SIZE is the default size of proto to print on parse errors.
+	PROTO_SNIPPET_SIZE = 32
+
+	// MAX_CONTROL_LINE_SNIPPET_SIZE is the default size of proto to print on max control line errors.
+	MAX_CONTROL_LINE_SNIPPET_SIZE = 128
+
+	// MAX_MSG_ARGS Maximum possible number of arguments from MSG proto.
+	MAX_MSG_ARGS = 4
+
+	// MAX_RMSG_ARGS Maximum possible number of arguments from RMSG proto.
+	MAX_RMSG_ARGS = 6
+
+	// MAX_HMSG_ARGS Maximum possible number of arguments from HMSG proto.
+	MAX_HMSG_ARGS = 7
+
+	// MAX_PUB_ARGS Maximum possible number of arguments from PUB proto.
+	MAX_PUB_ARGS = 3
+
+	// MAX_HPUB_ARGS Maximum possible number of arguments from HPUB proto.
+	MAX_HPUB_ARGS = 4
+
+	// MAX_RSUB_ARGS Maximum possible number of arguments from a RS+/LS+ proto.
+	MAX_RSUB_ARGS = 6
+
+	// DEFAULT_MAX_CLOSED_CLIENTS is the maximum number of closed connections we hold onto.
+	DEFAULT_MAX_CLOSED_CLIENTS = 10000
+
+	// DEFAULT_LAME_DUCK_DURATION is the time in which the server spreads
+	// the closing of clients when signaled to go in lame duck mode.
+	DEFAULT_LAME_DUCK_DURATION = 2 * time.Minute
+
+	// DEFAULT_LAME_DUCK_GRACE_PERIOD is the duration the server waits, after entering
+	// lame duck mode, before starting closing client connections.
+	DEFAULT_LAME_DUCK_GRACE_PERIOD = 10 * time.Second
+
+	// DEFAULT_LEAFNODE_INFO_WAIT Route dial timeout.
+	DEFAULT_LEAFNODE_INFO_WAIT = 1 * time.Second
+
+	// DEFAULT_LEAFNODE_PORT is the default port for remote leafnode connections.
+	DEFAULT_LEAFNODE_PORT = 7422
+
+	// DEFAULT_CONNECT_ERROR_REPORTS is the number of attempts at which a
+	// repeated failed route, gateway or leaf node connection is reported.
+	// This is used for initial connection, that is, when the server has
+	// never had a connection to the given endpoint. Once connected, and
+	// if a disconnect occurs, DEFAULT_RECONNECT_ERROR_REPORTS is used
+	// instead.
+	// The default is to report every 3600 attempts (roughly every hour).
+	DEFAULT_CONNECT_ERROR_REPORTS = 3600
+
+	// DEFAULT_RECONNECT_ERROR_REPORTS is the default number of failed
+	// attempt to reconnect a route, gateway or leaf node connection.
+	// The default is to report every attempt.
+	DEFAULT_RECONNECT_ERROR_REPORTS = 1
+
+	// DEFAULT_RTT_MEASUREMENT_INTERVAL is how often we want to measure RTT from
+	// this server to clients, routes, gateways or leafnode connections.
+	DEFAULT_RTT_MEASUREMENT_INTERVAL = time.Hour
+
+	// DEFAULT_ALLOW_RESPONSE_MAX_MSGS is the default number of responses allowed
+	// for a reply subject.
+	DEFAULT_ALLOW_RESPONSE_MAX_MSGS = 1
+
+	// DEFAULT_ALLOW_RESPONSE_EXPIRATION is the default time allowed for a given
+	// dynamic response permission.
+	DEFAULT_ALLOW_RESPONSE_EXPIRATION = 2 * time.Minute
+
+	// DEFAULT_SERVICE_EXPORT_RESPONSE_THRESHOLD is the default time that the system will
+	// expect a service export response to be delivered. This is used in corner cases for
+	// time based cleanup of reverse mapping structures.
+	DEFAULT_SERVICE_EXPORT_RESPONSE_THRESHOLD = 2 * time.Minute
+
+	// DEFAULT_SERVICE_LATENCY_SAMPLING is the default sampling rate for service
+	// latency metrics
+	DEFAULT_SERVICE_LATENCY_SAMPLING = 100
+
+	// DEFAULT_SYSTEM_ACCOUNT
+	DEFAULT_SYSTEM_ACCOUNT = "$SYS"
+
+	// DEFAULT GLOBAL_ACCOUNT
+	DEFAULT_GLOBAL_ACCOUNT = "$G"
+
+	// DEFAULT_FETCH_TIMEOUT is the default time that the system will wait for an account fetch to return.
+	DEFAULT_ACCOUNT_FETCH_TIMEOUT = 1900 * time.Millisecond
+)

server/core_benchmarks_test.go

@@ -0,0 +1,607 @@
+// Copyright 2023-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"crypto/tls"
+	"errors"
+	"fmt"
+	"math"
+	"math/rand/v2"
+	"os"
+	"strconv"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/nats-io/nats.go"
+)
+
+func BenchmarkCoreRequestReply(b *testing.B) {
+	const (
+		subject = "test-subject"
+	)
+
+	messageSizes := []int64{
+		1024,   // 1kb
+		4096,   // 4kb
+		40960,  // 40kb
+		409600, // 400kb
+	}
+
+	for _, messageSize := range messageSizes {
+		b.Run(fmt.Sprintf("msgSz=%db", messageSize), func(b *testing.B) {
+			// Start server
+			serverOpts := DefaultOptions()
+			server := RunServer(serverOpts)
+			defer server.Shutdown()
+
+			clientUrl := server.ClientURL()
+
+			// Create "echo" subscriber
+			ncSub, err := nats.Connect(clientUrl)
+			if err != nil {
+				b.Fatal(err)
+			}
+			defer ncSub.Close()
+			sub, err := ncSub.Subscribe(subject, func(msg *nats.Msg) {
+				// Responder echoes the request payload as-is
+				msg.Respond(msg.Data)
+			})
+			defer sub.Unsubscribe()
+			if err != nil {
+				b.Fatal(err)
+			}
+
+			// Create publisher
+			ncPub, err := nats.Connect(clientUrl)
+			if err != nil {
+				b.Fatal(err)
+			}
+			defer ncPub.Close()
+
+			errors := 0
+
+			// Create message
+			messageData := make([]byte, messageSize)
+			rand.NewChaCha8([32]byte{42}).Read(messageData)
+
+			b.SetBytes(messageSize)
+
+			// Benchmark
+			b.ResetTimer()
+			for i := 0; i < b.N; i++ {
+				fastRandomMutation(messageData, 10)
+
+				_, err := ncPub.Request(subject, messageData, time.Second)
+				if err != nil {
+					errors++
+				}
+			}
+			b.StopTimer()
+
+			b.ReportMetric(float64(errors), "errors")
+		})
+	}
+}
+
+func BenchmarkCoreTLSFanOut(b *testing.B) {
+	const (
+		subject            = "test-subject"
+		configsBasePath    = "./configs/tls"
+		maxPendingMessages = 25
+		maxPendingBytes    = 15 * 1024 * 1024 // 15MiB
+	)
+
+	keyTypeCases := []string{
+		"none",
+		"ed25519",
+		"rsa-1024",
+		"rsa-2048",
+		"rsa-4096",
+	}
+	messageSizeCases := []int64{
+		512 * 1024, // 512Kib
+	}
+	numSubsCases := []int{
+		5,
+	}
+
+	// Custom error handler that ignores ErrSlowConsumer.
+	// Lots of them are expected in this benchmark which indiscriminately publishes at a rate higher
+	// than what the server can relay to subscribers.
+	ignoreSlowConsumerErrorHandler := func(conn *nats.Conn, s *nats.Subscription, err error) {
+		if errors.Is(err, nats.ErrSlowConsumer) {
+			// Swallow this error
+		} else {
+			_, _ = fmt.Fprintf(os.Stderr, "Warning: %s\n", err)
+		}
+	}
+
+	for _, keyType := range keyTypeCases {
+		b.Run(
+			fmt.Sprintf("keyType=%s", keyType),
+			func(b *testing.B) {
+				for _, messageSize := range messageSizeCases {
+					b.Run(
+						fmt.Sprintf("msgSz=%db", messageSize),
+						func(b *testing.B) {
+							for _, numSubs := range numSubsCases {
+								b.Run(
+									fmt.Sprintf("subs=%d", numSubs),
+									func(b *testing.B) {
+										// Start server
+										configPath := fmt.Sprintf("%s/tls-%s.conf", configsBasePath, keyType)
+										server, _ := RunServerWithConfig(configPath)
+										defer server.Shutdown()
+
+										opts := []nats.Option{
+											nats.MaxReconnects(-1),
+											nats.ReconnectWait(0),
+											nats.ErrorHandler(ignoreSlowConsumerErrorHandler),
+										}
+
+										if keyType != "none" {
+											opts = append(opts, nats.Secure(&tls.Config{
+												InsecureSkipVerify: true,
+											}))
+										}
+
+										clientUrl := server.ClientURL()
+
+										// Count of messages received for by each subscriber
+										counters := make([]int, numSubs)
+
+										// Wait group for subscribers to signal they received b.N messages
+										var wg sync.WaitGroup
+										wg.Add(numSubs)
+
+										// Create subscribers
+										for i := 0; i < numSubs; i++ {
+											subIndex := i
+											ncSub, err := nats.Connect(clientUrl, opts...)
+											if err != nil {
+												b.Fatal(err)
+											}
+											defer ncSub.Close()
+											sub, err := ncSub.Subscribe(subject, func(msg *nats.Msg) {
+												counters[subIndex] += 1
+												if counters[subIndex] == b.N {
+													wg.Done()
+												}
+											})
+											if err != nil {
+												b.Fatalf("failed to subscribe: %s", err)
+											}
+											err = sub.SetPendingLimits(maxPendingMessages, maxPendingBytes)
+											if err != nil {
+												b.Fatalf("failed to set pending limits: %s", err)
+											}
+											defer sub.Unsubscribe()
+											if err != nil {
+												b.Fatal(err)
+											}
+										}
+
+										// publisher
+										ncPub, err := nats.Connect(clientUrl, opts...)
+										if err != nil {
+											b.Fatal(err)
+										}
+										defer ncPub.Close()
+
+										errorCount := 0
+
+										// random bytes as payload
+										messageData := make([]byte, messageSize)
+										rand.NewChaCha8([32]byte{42}).Read(messageData)
+
+										quitCh := make(chan bool, 1)
+
+										publish := func() {
+											for {
+												select {
+												case <-quitCh:
+													return
+												default:
+													// continue publishing
+												}
+
+												fastRandomMutation(messageData, 10)
+												err := ncPub.Publish(subject, messageData)
+												if err != nil {
+													errorCount += 1
+												}
+											}
+										}
+
+										// Set bytes per operation
+										b.SetBytes(messageSize)
+										// Start the clock
+										b.ResetTimer()
+										// Start publishing as fast as the server allows
+										go publish()
+										// Wait for all subscribers to have delivered b.N messages
+										wg.Wait()
+										// Stop the clock
+										b.StopTimer()
+
+										// Stop publisher
+										quitCh <- true
+
+										b.ReportMetric(float64(errorCount), "errors")
+									},
+								)
+							}
+						},
+					)
+				}
+			},
+		)
+	}
+}
+
+func BenchmarkCoreFanOut(b *testing.B) {
+	const (
+		subject            = "test-subject"
+		maxPendingMessages = 25
+		maxPendingBytes    = 15 * 1024 * 1024 // 15MiB
+	)
+
+	messageSizeCases := []int64{
+		100,        // 100B
+		1024,       // 1KiB
+		10240,      // 10KiB
+		512 * 1024, // 512KiB
+	}
+	numSubsCases := []int{
+		3,
+		5,
+		10,
+	}
+
+	// Custom error handler that ignores ErrSlowConsumer.
+	// Lots of them are expected in this benchmark which indiscriminately publishes at a rate higher
+	// than what the server can relay to subscribers.
+	ignoreSlowConsumerErrorHandler := func(_ *nats.Conn, _ *nats.Subscription, err error) {
+		if errors.Is(err, nats.ErrSlowConsumer) {
+			// Swallow this error
+		} else {
+			_, _ = fmt.Fprintf(os.Stderr, "Warning: %s\n", err)
+		}
+	}
+
+	for _, messageSize := range messageSizeCases {
+		b.Run(
+			fmt.Sprintf("msgSz=%db", messageSize),
+			func(b *testing.B) {
+				for _, numSubs := range numSubsCases {
+					b.Run(
+						fmt.Sprintf("subs=%d", numSubs),
+						func(b *testing.B) {
+							// Start server
+							defaultOpts := DefaultOptions()
+							server := RunServer(defaultOpts)
+							defer server.Shutdown()
+
+							opts := []nats.Option{
+								nats.MaxReconnects(-1),
+								nats.ReconnectWait(0),
+								nats.ErrorHandler(ignoreSlowConsumerErrorHandler),
+							}
+
+							clientUrl := server.ClientURL()
+
+							// Count of messages received for by each subscriber
+							counters := make([]int, numSubs)
+
+							// Wait group for subscribers to signal they received b.N messages
+							var wg sync.WaitGroup
+							wg.Add(numSubs)
+
+							// Create subscribers
+							for i := 0; i < numSubs; i++ {
+								subIndex := i
+								ncSub, err := nats.Connect(clientUrl, opts...)
+								if err != nil {
+									b.Fatal(err)
+								}
+								defer ncSub.Close()
+								sub, err := ncSub.Subscribe(subject, func(_ *nats.Msg) {
+									counters[subIndex] += 1
+									if counters[subIndex] == b.N {
+										wg.Done()
+									}
+								})
+								if err != nil {
+									b.Fatalf("failed to subscribe: %s", err)
+								}
+								err = sub.SetPendingLimits(maxPendingMessages, maxPendingBytes)
+								if err != nil {
+									b.Fatalf("failed to set pending limits: %s", err)
+								}
+								defer sub.Unsubscribe()
+							}
+
+							// publisher
+							ncPub, err := nats.Connect(clientUrl, opts...)
+							if err != nil {
+								b.Fatal(err)
+							}
+							defer ncPub.Close()
+
+							errorCount := 0
+
+							// random bytes as payload
+							messageData := make([]byte, messageSize)
+							rand.NewChaCha8([32]byte{42}).Read(messageData)
+
+							quitCh := make(chan bool, 1)
+
+							publish := func() {
+								for {
+									select {
+									case <-quitCh:
+										return
+									default:
+										// continue publishing
+									}
+
+									fastRandomMutation(messageData, 10)
+									err := ncPub.Publish(subject, messageData)
+									if err != nil {
+										errorCount += 1
+									}
+								}
+							}
+
+							// Set bytes per operation
+							b.SetBytes(messageSize)
+							// Start the clock
+							b.ResetTimer()
+							// Start publishing as fast as the server allows
+							go publish()
+							// Wait for all subscribers to have delivered b.N messages
+							wg.Wait()
+							// Stop the clock
+							b.StopTimer()
+
+							// Stop publisher
+							quitCh <- true
+
+							b.ReportMetric(100*float64(errorCount)/float64(b.N), "%error")
+						},
+					)
+				}
+			},
+		)
+	}
+}
+
+func BenchmarkCoreFanIn(b *testing.B) {
+	type BenchPublisher struct {
+		// nats connection for this publisher
+		conn *nats.Conn
+		// number of publishing errors encountered
+		publishErrors int
+		// number of messages published
+		publishCounter int
+		// quit channel which will terminate publishing
+		quitCh chan bool
+		// message data buffer
+		messageData []byte
+	}
+
+	const subjectBaseName = "test-subject"
+
+	messageSizeCases := []int64{
+		100,        // 100B
+		1024,       // 1KiB
+		10240,      // 10KiB
+		512 * 1024, // 512KiB
+	}
+	numPubsCases := []int{
+		3,
+		5,
+		10,
+	}
+
+	// Custom error handler that ignores ErrSlowConsumer.
+	// Lots of them are expected in this benchmark which indiscriminately publishes at a rate higher
+	// than what the server can relay to subscribers.
+	ignoreSlowConsumerErrorHandler := func(_ *nats.Conn, _ *nats.Subscription, err error) {
+		if errors.Is(err, nats.ErrSlowConsumer) {
+			// Swallow this error
+		} else {
+			_, _ = fmt.Fprintf(os.Stderr, "Warning: %s\n", err)
+		}
+	}
+
+	workload := func(b *testing.B, clientUrl string, numPubs int, messageSize int64) {
+		// connection options
+		opts := []nats.Option{
+			nats.MaxReconnects(-1),
+			nats.ReconnectWait(0),
+			nats.ErrorHandler(ignoreSlowConsumerErrorHandler),
+		}
+
+		// waits for all publishers sub-routines and for main thread to be ready
+		var publishersReadyWg sync.WaitGroup
+		publishersReadyWg.Add(numPubs + 1)
+
+		// wait group to ensure all publishers have been torn down
+		var finishedPublishersWg sync.WaitGroup
+		finishedPublishersWg.Add(numPubs)
+
+		publishers := make([]BenchPublisher, numPubs)
+		// create N publishers
+		for i := range publishers {
+			// create publisher connection
+			ncPub, err := nats.Connect(clientUrl, opts...)
+			if err != nil {
+				b.Fatal(err)
+			}
+			defer ncPub.Close()
+
+			// create bench publisher object
+			publisher := BenchPublisher{
+				conn:           ncPub,
+				publishErrors:  0,
+				publishCounter: 0,
+				quitCh:         make(chan bool, 1),
+				messageData:    make([]byte, messageSize),
+			}
+			rand.NewChaCha8([32]byte{byte(i)}).Read(publisher.messageData)
+			publishers[i] = publisher
+		}
+
+		// total number of publishers that have published b.N to the subscriber successfully
+		completedPublishersCount := 0
+
+		// wait group blocks main thread until publish workload is completed, it is decremented after subscriber receives b.N messages from all publishers
+		var benchCompleteWg sync.WaitGroup
+		benchCompleteWg.Add(1)
+
+		// start subscriber
+		ncSub, err := nats.Connect(clientUrl, opts...)
+		if err != nil {
+			b.Fatal(err)
+		}
+		defer ncSub.Close()
+
+		// subscriber
+		ncSub.Subscribe(fmt.Sprintf("%s.*", subjectBaseName), func(msg *nats.Msg) {
+			// get the publisher id from subject
+			pubIdx, err := strconv.Atoi(msg.Subject[len(subjectBaseName)+1:])
+			if err != nil {
+				b.Fatal(err)
+			}
+
+			// message successfully received from publisher
+			publishers[pubIdx].publishCounter += 1
+
+			// subscriber has received a total of b.N messages from this publisher
+			if publishers[pubIdx].publishCounter == b.N {
+				completedPublishersCount++
+				// every publisher has successfully sent b.N messages to subscriber
+				if completedPublishersCount == numPubs {
+					benchCompleteWg.Done()
+				}
+			}
+		})
+
+		// start publisher sub-routines
+		for i := range publishers {
+			go func(pubId int) {
+				// publisher sub-routine initialized
+				publishersReadyWg.Done()
+
+				publisher := publishers[pubId]
+				subject := fmt.Sprintf("%s.%d", subjectBaseName, pubId)
+
+				// signal that this publisher has been torn down
+				defer finishedPublishersWg.Done()
+
+				// wait till all other publishers are ready to start workload
+				publishersReadyWg.Wait()
+
+				// publish until quitCh is closed
+				for {
+					select {
+					case <-publisher.quitCh:
+						return
+					default:
+						// continue publishing
+					}
+					fastRandomMutation(publisher.messageData, 10)
+					err := publisher.conn.Publish(subject, publisher.messageData)
+					if err != nil {
+						publisher.publishErrors += 1
+					}
+				}
+			}(i)
+		}
+
+		// set bytes per operation
+		b.SetBytes(messageSize)
+		// main thread is ready
+		publishersReadyWg.Done()
+		// wait till publishers are ready
+		publishersReadyWg.Wait()
+
+		// start the clock
+		b.ResetTimer()
+		// wait till termination cond reached
+		benchCompleteWg.Wait()
+		// stop the clock
+		b.StopTimer()
+
+		// send quit signal to all publishers
+		for i := range publishers {
+			publishers[i].quitCh <- true
+		}
+		// wait for all publishers to shutdown
+		finishedPublishersWg.Wait()
+
+		// sum errors from all publishers
+		totalErrors := 0
+		for _, publisher := range publishers {
+			totalErrors += publisher.publishErrors
+		}
+		// sum total messages sent from all publishers
+		totalMessages := 0
+		for _, publisher := range publishers {
+			totalMessages += publisher.publishCounter
+		}
+		errorRate := 100 * float64(totalErrors) / float64(totalMessages)
+
+		// report error rate
+		b.ReportMetric(errorRate, "%error")
+	}
+
+	// benchmark case matrix
+	for _, messageSize := range messageSizeCases {
+		b.Run(
+			fmt.Sprintf("msgSz=%db", messageSize),
+			func(b *testing.B) {
+				for _, numPubs := range numPubsCases {
+					b.Run(
+						fmt.Sprintf("pubs=%d", numPubs),
+						func(b *testing.B) {
+							// start server
+							defaultOpts := DefaultOptions()
+							server := RunServer(defaultOpts)
+							defer server.Shutdown()
+
+							// get connection string
+							clientUrl := server.ClientURL()
+
+							// run fan-in workload
+							workload(b, clientUrl, numPubs, messageSize)
+						},
+					)
+				}
+			},
+		)
+	}
+}
+
+// fastRandomMutation performs a minor in-place mutation to the given buffer.
+// This is useful in benchmark to avoid sending the same payload every time (which could result in some optimizations
+// we do not want to measure), while not slowing down the benchmark with a full payload generated for each operation.
+func fastRandomMutation(data []byte, mutations int) {
+	for i := 0; i < mutations; i++ {
+		data[rand.Uint32N(uint32(len(data)))] = byte(rand.Uint32() % math.MaxUint8)
+	}
+}

server/cron.go

@@ -0,0 +1,327 @@
+// Copyright 2025-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+// Based on code from https://github.com/robfig/cron
+// Copyright (C) 2012 Rob Figueiredo
+// All Rights Reserved.
+//
+// MIT LICENSE
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy of
+// this software and associated documentation files (the "Software"), to deal in
+// the Software without restriction, including without limitation the rights to
+// use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+// the Software, and to permit persons to whom the Software is furnished to do so,
+// subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+// FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+// COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+// IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+// CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+package server
+
+import (
+	"errors"
+	"fmt"
+	"math"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// parseCron parses the given cron pattern and returns the next time it will fire based on the provided ts.
+func parseCron(pattern string, loc *time.Location, ts int64) (time.Time, error) {
+	fields := strings.Fields(pattern)
+	if len(fields) != 6 {
+		return time.Time{}, fmt.Errorf("pattern requires 6 fields, got %d", len(fields))
+	}
+
+	// If no time zone is passed, default to UTC.
+	if loc == nil {
+		loc = time.UTC
+	}
+
+	// Parse each field.
+	var err error
+	var second, minute, hour, dayOfMonth, month, dayOfWeek uint64
+	if second, err = getField(fields[0], seconds); err != nil {
+		return time.Time{}, err
+	}
+	if minute, err = getField(fields[1], minutes); err != nil {
+		return time.Time{}, err
+	}
+	if hour, err = getField(fields[2], hours); err != nil {
+		return time.Time{}, err
+	}
+	if dayOfMonth, err = getField(fields[3], dom); err != nil {
+		return time.Time{}, err
+	}
+	if month, err = getField(fields[4], months); err != nil {
+		return time.Time{}, err
+	}
+	if dayOfWeek, err = getField(fields[5], dow); err != nil {
+		return time.Time{}, err
+	}
+
+	// General approach
+	//
+	// For Month, Day, Hour, Minute, Second:
+	// Check if the time value matches. If yes, continue to the next field.
+	// If the field doesn't match the schedule, then increment the field until it matches.
+	// While incrementing the field, a wrap-around brings it back to the beginning
+	// of the field list (since it is necessary to re-verify previous field values)
+	next := time.Unix(0, ts).In(loc)
+
+	// Start at the earliest possible time (the upcoming second).
+	next = next.Truncate(time.Second).Add(time.Second)
+
+	// This flag indicates whether a field has been truncated at one point.
+	truncated := false
+
+	// If no time is found within five years, return error.
+	yearLimit := next.Year() + 5
+
+WRAP:
+	if next.Year() > yearLimit {
+		return time.Time{}, errors.New("pattern exceeds maximum range")
+	}
+	for 1<<uint(next.Month())&month == 0 {
+		if !truncated {
+			truncated = true
+			next = time.Date(next.Year(), next.Month(), 1, 0, 0, 0, 0, loc)
+		}
+		if next = next.AddDate(0, 1, 0); next.Month() == time.January {
+			goto WRAP
+		}
+	}
+	for !dayMatches(dayOfMonth, dayOfWeek, next) {
+		if !truncated {
+			truncated = true
+			next = time.Date(next.Year(), next.Month(), next.Day(), 0, 0, 0, 0, loc)
+		}
+		if next = next.AddDate(0, 0, 1); next.Day() == 1 {
+			goto WRAP
+		}
+	}
+	for 1<<uint(next.Hour())&hour == 0 {
+		if !truncated {
+			truncated = true
+			next = time.Date(next.Year(), next.Month(), next.Day(), next.Hour(), 0, 0, 0, loc)
+		}
+		if next = next.Add(time.Hour); next.Hour() == 0 {
+			goto WRAP
+		}
+	}
+	for 1<<uint(next.Minute())&minute == 0 {
+		if !truncated {
+			truncated = true
+			next = next.Truncate(time.Minute)
+		}
+		if next = next.Add(time.Minute); next.Minute() == 0 {
+			goto WRAP
+		}
+	}
+	for 1<<uint(next.Second())&second == 0 {
+		if !truncated {
+			truncated = true
+			next = next.Truncate(time.Second)
+		}
+		if next = next.Add(time.Second); next.Second() == 0 {
+			goto WRAP
+		}
+	}
+	return next, nil
+}
+
+// getField returns an Int with the bits set representing all of the times that
+// the field represents or error parsing field value.  A "field" is a comma-separated
+// list of "ranges".
+func getField(field string, r bounds) (uint64, error) {
+	var bits uint64
+	ranges := strings.FieldsFuncSeq(field, func(r rune) bool { return r == ',' })
+	for expr := range ranges {
+		bit, err := getRange(expr, r)
+		if err != nil {
+			return bits, err
+		}
+		bits |= bit
+	}
+	return bits, nil
+}
+
+// getRange returns the bits indicated by the given expression: number | number [ "-" number ] [ "/" number ]
+// or error parsing range.
+func getRange(expr string, r bounds) (uint64, error) {
+	var (
+		start, end, step uint
+		rangeAndStep     = strings.Split(expr, "/")
+		lowAndHigh       = strings.Split(rangeAndStep[0], "-")
+		singleDigit      = len(lowAndHigh) == 1
+		err              error
+	)
+
+	var extra uint64
+	if lowAndHigh[0] == "*" || lowAndHigh[0] == "?" {
+		start = r.min
+		end = r.max
+		extra = starBit
+	} else {
+		start, err = parseIntOrName(lowAndHigh[0], r.names)
+		if err != nil {
+			return 0, err
+		}
+		switch len(lowAndHigh) {
+		case 1:
+			end = start
+		case 2:
+			end, err = parseIntOrName(lowAndHigh[1], r.names)
+			if err != nil {
+				return 0, err
+			}
+		default:
+			return 0, fmt.Errorf("too many hyphens: %s", expr)
+		}
+	}
+
+	switch len(rangeAndStep) {
+	case 1:
+		step = 1
+	case 2:
+		step, err = mustParseInt(rangeAndStep[1])
+		if err != nil {
+			return 0, err
+		}
+		// Special handling: "N/step" means "N-max/step".
+		if singleDigit {
+			end = r.max
+		}
+		if step > 1 {
+			extra = 0
+		}
+	default:
+		return 0, fmt.Errorf("too many slashes: %s", expr)
+	}
+
+	if start < r.min {
+		return 0, fmt.Errorf("beginning of range (%d) below minimum (%d): %s", start, r.min, expr)
+	}
+	if end > r.max {
+		return 0, fmt.Errorf("end of range (%d) above maximum (%d): %s", end, r.max, expr)
+	}
+	if start > end {
+		return 0, fmt.Errorf("beginning of range (%d) beyond end of range (%d): %s", start, end, expr)
+	}
+	if step == 0 {
+		return 0, fmt.Errorf("step of range should be a positive number: %s", expr)
+	}
+	return getBits(start, end, step) | extra, nil
+}
+
+// parseIntOrName returns the (possibly-named) integer contained in expr.
+func parseIntOrName(expr string, names map[string]uint) (uint, error) {
+	if names != nil {
+		if namedInt, ok := names[strings.ToLower(expr)]; ok {
+			return namedInt, nil
+		}
+	}
+	return mustParseInt(expr)
+}
+
+// mustParseInt parses the given expression as an int or returns an error.
+func mustParseInt(expr string) (uint, error) {
+	num, err := strconv.Atoi(expr)
+	if err != nil {
+		return 0, fmt.Errorf("failed to parse int from %s: %s", expr, err)
+	}
+	if num < 0 {
+		return 0, fmt.Errorf("negative number (%d) not allowed: %s", num, expr)
+	}
+	return uint(num), nil
+}
+
+// getBits sets all bits in the range [min, max], modulo the given step size.
+func getBits(min, max, step uint) uint64 {
+	var bits uint64
+
+	// If step is 1, use shifts.
+	if step == 1 {
+		return ^(math.MaxUint64 << (max + 1)) & (math.MaxUint64 << min)
+	}
+
+	// Else, use a simple loop.
+	for i := min; i <= max; i += step {
+		bits |= 1 << i
+	}
+	return bits
+}
+
+// bounds provides a range of acceptable values (plus a map of name to value).
+type bounds struct {
+	min, max uint
+	names    map[string]uint
+}
+
+// The bounds for each field.
+var (
+	seconds = bounds{0, 59, nil}
+	minutes = bounds{0, 59, nil}
+	hours   = bounds{0, 23, nil}
+	dom     = bounds{1, 31, nil}
+	months  = bounds{1, 12, map[string]uint{
+		"jan": 1,
+		"feb": 2,
+		"mar": 3,
+		"apr": 4,
+		"may": 5,
+		"jun": 6,
+		"jul": 7,
+		"aug": 8,
+		"sep": 9,
+		"oct": 10,
+		"nov": 11,
+		"dec": 12,
+	}}
+	dow = bounds{0, 6, map[string]uint{
+		"sun": 0,
+		"mon": 1,
+		"tue": 2,
+		"wed": 3,
+		"thu": 4,
+		"fri": 5,
+		"sat": 6,
+	}}
+)
+
+const (
+	// Set the top bit if a star was included in the expression.
+	starBit = 1 << 63
+)
+
+// dayMatches returns true if the schedule's day-of-week and day-of-month
+// restrictions are satisfied by the given time.
+func dayMatches(dayOfMonth, dayOfWeek uint64, t time.Time) bool {
+	var (
+		domMatch = 1<<uint(t.Day())&dayOfMonth > 0
+		dowMatch = 1<<uint(t.Weekday())&dayOfWeek > 0
+	)
+	if dayOfMonth&starBit > 0 || dayOfWeek&starBit > 0 {
+		return domMatch && dowMatch
+	}
+	return domMatch || dowMatch
+}

server/dios.go

@@ -0,0 +1,85 @@
+// Copyright 2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"sync/atomic"
+	"time"
+)
+
+const defaultConcurrentIOs = 4096
+const minConcurrentIOs = 4
+const maxConcurrentIOs = 8192
+
+// Used to limit number of disk IO calls in flight since they could all be blocking an OS thread.
+// https://github.com/nats-io/nats-server/issues/2742
+type diskIOSemaphore struct {
+	ch           chan struct{}
+	waiters      atomic.Int64
+	waits        atomic.Uint64
+	waitNanos    atomic.Uint64
+	maxWaitNanos atomic.Uint64
+}
+
+func newDiskIOSemaphore(n int) *diskIOSemaphore {
+	n = max(minConcurrentIOs, min(n, maxConcurrentIOs))
+	d := &diskIOSemaphore{ch: make(chan struct{}, n)}
+	for range n {
+		d.ch <- struct{}{}
+	}
+	return d
+}
+
+func defaultDiskIOSemaphore() *diskIOSemaphore {
+	// The disk IO semaphore used to be sized based on the number
+	// of CPU cores. That policy led to poor use of devices that
+	// can handle many requests in parallel, so simply cap the
+	// number of concurrent IO requests handed to the Go runtime.
+	return newDiskIOSemaphore(defaultConcurrentIOs)
+}
+
+func (d *diskIOSemaphore) acquire() {
+	select {
+	case <-d.ch:
+		return
+	default:
+		// No slot available, count this
+		// waiter before blocking.
+		d.waiters.Add(1)
+		start := time.Now()
+		<-d.ch
+		waited := time.Since(start)
+		d.waiters.Add(-1)
+		d.countWait(uint64(waited.Nanoseconds()))
+	}
+}
+
+func (d *diskIOSemaphore) countWait(ns uint64) {
+	d.waits.Add(1)
+	d.waitNanos.Add(ns)
+	for {
+		cur := d.maxWaitNanos.Load()
+		if ns <= cur || d.maxWaitNanos.CompareAndSwap(cur, ns) {
+			return
+		}
+	}
+}
+
+func (d *diskIOSemaphore) release() {
+	d.ch <- struct{}{}
+}
+
+func (d *diskIOSemaphore) cap() int {
+	return cap(d.ch)
+}

server/dirstore.go

@@ -0,0 +1,727 @@
+// Copyright 2012-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"bytes"
+	"container/heap"
+	"container/list"
+	"crypto/sha256"
+	"errors"
+	"fmt"
+	"math"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/nats-io/nkeys"
+
+	"github.com/nats-io/jwt/v2" // only used to decode, not for storage
+)
+
+const (
+	fileExtension = ".jwt"
+)
+
+// validatePathExists checks that the provided path exists and is a dir if requested
+func validatePathExists(path string, dir bool) (string, error) {
+	if path == _EMPTY_ {
+		return _EMPTY_, errors.New("path is not specified")
+	}
+
+	abs, err := filepath.Abs(path)
+	if err != nil {
+		return _EMPTY_, fmt.Errorf("error parsing path [%s]: %v", abs, err)
+	}
+
+	var finfo os.FileInfo
+	if finfo, err = os.Stat(abs); err != nil {
+		if os.IsNotExist(err) {
+			return _EMPTY_, fmt.Errorf("the path [%s] doesn't exist", abs)
+		}
+		return _EMPTY_, fmt.Errorf("error accessing path [%s]: %v", abs, err)
+	}
+
+	mode := finfo.Mode()
+	if dir && mode.IsRegular() {
+		return _EMPTY_, fmt.Errorf("the path [%s] is not a directory", abs)
+	}
+
+	if !dir && mode.IsDir() {
+		return _EMPTY_, fmt.Errorf("the path [%s] is not a file", abs)
+	}
+
+	return abs, nil
+}
+
+// ValidateDirPath checks that the provided path exists and is a dir
+func validateDirPath(path string) (string, error) {
+	return validatePathExists(path, true)
+}
+
+// JWTChanged functions are called when the store file watcher notices a JWT changed
+type JWTChanged func(publicKey string)
+
+// DirJWTStore implements the JWT Store interface, keeping JWTs in an optionally sharded
+// directory structure
+type DirJWTStore struct {
+	sync.Mutex
+	directory  string
+	shard      bool
+	readonly   bool
+	deleteType deleteType
+	operator   map[string]struct{}
+	expiration *expirationTracker
+	changed    JWTChanged
+	deleted    JWTChanged
+}
+
+func newDir(dirPath string, create bool) (string, error) {
+	fullPath, err := validateDirPath(dirPath)
+	if err != nil {
+		if !create {
+			return _EMPTY_, err
+		}
+		if err = os.MkdirAll(dirPath, defaultDirPerms); err != nil {
+			return _EMPTY_, err
+		}
+		if fullPath, err = validateDirPath(dirPath); err != nil {
+			return _EMPTY_, err
+		}
+	}
+	return fullPath, nil
+}
+
+// future proofing in case new options will be added
+type dirJWTStoreOption any
+
+// Creates a directory based jwt store.
+// Reads files only, does NOT watch directories and files.
+func NewImmutableDirJWTStore(dirPath string, shard bool, _ ...dirJWTStoreOption) (*DirJWTStore, error) {
+	theStore, err := NewDirJWTStore(dirPath, shard, false, nil)
+	if err != nil {
+		return nil, err
+	}
+	theStore.readonly = true
+	return theStore, nil
+}
+
+// Creates a directory based jwt store.
+// Operates on files only, does NOT watch directories and files.
+func NewDirJWTStore(dirPath string, shard bool, create bool, _ ...dirJWTStoreOption) (*DirJWTStore, error) {
+	fullPath, err := newDir(dirPath, create)
+	if err != nil {
+		return nil, err
+	}
+	theStore := &DirJWTStore{
+		directory: fullPath,
+		shard:     shard,
+	}
+	return theStore, nil
+}
+
+type deleteType int
+
+const (
+	NoDelete deleteType = iota
+	RenameDeleted
+	HardDelete
+)
+
+// Creates a directory based jwt store.
+//
+// When ttl is set deletion of file is based on it and not on the jwt expiration
+// To completely disable expiration (including expiration in jwt) set ttl to max duration time.Duration(math.MaxInt64)
+//
+// limit defines how many files are allowed at any given time. Set to math.MaxInt64 to disable.
+// evictOnLimit determines the behavior once limit is reached.
+// * true - Evict based on lru strategy
+// * false - return an error
+func NewExpiringDirJWTStore(dirPath string, shard bool, create bool, delete deleteType, expireCheck time.Duration, limit int64,
+	evictOnLimit bool, ttl time.Duration, changeNotification JWTChanged, _ ...dirJWTStoreOption) (*DirJWTStore, error) {
+	fullPath, err := newDir(dirPath, create)
+	if err != nil {
+		return nil, err
+	}
+	theStore := &DirJWTStore{
+		directory:  fullPath,
+		shard:      shard,
+		deleteType: delete,
+		changed:    changeNotification,
+	}
+	if expireCheck <= 0 {
+		if ttl != 0 {
+			expireCheck = ttl / 2
+		}
+		if expireCheck == 0 || expireCheck > time.Minute {
+			expireCheck = time.Minute
+		}
+	}
+	if limit <= 0 {
+		limit = math.MaxInt64
+	}
+	theStore.startExpiring(expireCheck, limit, evictOnLimit, ttl)
+	theStore.Lock()
+	err = filepath.Walk(dirPath, func(path string, info os.FileInfo, err error) error {
+		if strings.HasSuffix(path, fileExtension) {
+			if theJwt, err := os.ReadFile(path); err == nil {
+				hash := sha256.Sum256(theJwt)
+				_, file := filepath.Split(path)
+				theStore.expiration.track(strings.TrimSuffix(file, fileExtension), &hash, string(theJwt))
+			}
+		}
+		return nil
+	})
+	theStore.Unlock()
+	if err != nil {
+		theStore.Close()
+		return nil, err
+	}
+	return theStore, err
+}
+
+func (store *DirJWTStore) IsReadOnly() bool {
+	return store.readonly
+}
+
+func (store *DirJWTStore) LoadAcc(publicKey string) (string, error) {
+	return store.load(publicKey)
+}
+
+func (store *DirJWTStore) SaveAcc(publicKey string, theJWT string) error {
+	return store.save(publicKey, theJWT)
+}
+
+func (store *DirJWTStore) LoadAct(hash string) (string, error) {
+	return store.load(hash)
+}
+
+func (store *DirJWTStore) SaveAct(hash string, theJWT string) error {
+	return store.save(hash, theJWT)
+}
+
+func (store *DirJWTStore) Close() {
+	store.Lock()
+	defer store.Unlock()
+	if store.expiration != nil {
+		store.expiration.close()
+		store.expiration = nil
+	}
+}
+
+// Pack up to maxJWTs into a package
+func (store *DirJWTStore) Pack(maxJWTs int) (string, error) {
+	count := 0
+	var pack []string
+	if maxJWTs > 0 {
+		pack = make([]string, 0, maxJWTs)
+	} else {
+		pack = []string{}
+	}
+	store.Lock()
+	err := filepath.Walk(store.directory, func(path string, info os.FileInfo, err error) error {
+		if info != nil && !info.IsDir() && strings.HasSuffix(path, fileExtension) { // this is a JWT
+			if count == maxJWTs { // won't match negative
+				return nil
+			}
+			pubKey := strings.TrimSuffix(filepath.Base(path), fileExtension)
+			if store.expiration != nil {
+				if _, ok := store.expiration.idx[pubKey]; !ok {
+					return nil // only include indexed files
+				}
+			}
+			jwtBytes, err := os.ReadFile(path)
+			if err != nil {
+				return err
+			}
+			if store.expiration != nil {
+				claim, err := jwt.DecodeGeneric(string(jwtBytes))
+				if err == nil && claim.Expires > 0 && claim.Expires < time.Now().Unix() {
+					return nil
+				}
+			}
+			pack = append(pack, fmt.Sprintf("%s|%s", pubKey, string(jwtBytes)))
+			count++
+		}
+		return nil
+	})
+	store.Unlock()
+	if err != nil {
+		return _EMPTY_, err
+	} else {
+		return strings.Join(pack, "\n"), nil
+	}
+}
+
+// Pack up to maxJWTs into a message and invoke callback with it
+func (store *DirJWTStore) PackWalk(maxJWTs int, cb func(partialPackMsg string)) error {
+	if maxJWTs <= 0 || cb == nil {
+		return errors.New("bad arguments to PackWalk")
+	}
+	var packMsg []string
+	store.Lock()
+	dir := store.directory
+	exp := store.expiration
+	store.Unlock()
+	err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
+		if info != nil && !info.IsDir() && strings.HasSuffix(path, fileExtension) { // this is a JWT
+			pubKey := strings.TrimSuffix(filepath.Base(path), fileExtension)
+			store.Lock()
+			if exp != nil {
+				if _, ok := exp.idx[pubKey]; !ok {
+					store.Unlock()
+					return nil // only include indexed files
+				}
+			}
+			store.Unlock()
+			jwtBytes, err := os.ReadFile(path)
+			if err != nil {
+				return err
+			}
+			if len(jwtBytes) == 0 {
+				// Skip if no contents in the JWT.
+				return nil
+			}
+			if exp != nil {
+				claim, err := jwt.DecodeGeneric(string(jwtBytes))
+				if err == nil && claim.Expires > 0 && claim.Expires < time.Now().Unix() {
+					return nil
+				}
+			}
+			packMsg = append(packMsg, fmt.Sprintf("%s|%s", pubKey, string(jwtBytes)))
+			if len(packMsg) == maxJWTs { // won't match negative
+				cb(strings.Join(packMsg, "\n"))
+				packMsg = nil
+			}
+		}
+		return nil
+	})
+	if packMsg != nil {
+		cb(strings.Join(packMsg, "\n"))
+	}
+	return err
+}
+
+// Merge takes the JWTs from package and adds them to the store
+// Merge is destructive in the sense that it doesn't check if the JWT
+// is newer or anything like that.
+func (store *DirJWTStore) Merge(pack string) error {
+	newJWTs := strings.Split(pack, "\n")
+	for _, line := range newJWTs {
+		if line == _EMPTY_ { // ignore blank lines
+			continue
+		}
+		split := strings.Split(line, "|")
+		if len(split) != 2 {
+			return fmt.Errorf("line in package didn't contain 2 entries: %q", line)
+		}
+		pubKey := split[0]
+		if !nkeys.IsValidPublicAccountKey(pubKey) {
+			return fmt.Errorf("key to merge is not a valid public account key")
+		}
+		if err := store.saveIfNewer(pubKey, split[1]); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (store *DirJWTStore) Reload() error {
+	store.Lock()
+	exp := store.expiration
+	if exp == nil || store.readonly {
+		store.Unlock()
+		return nil
+	}
+	idx := exp.idx
+	changed := store.changed
+	isCache := store.expiration.evictOnLimit
+	// clear out indexing data structures
+	exp.heap = make([]*jwtItem, 0, len(exp.heap))
+	exp.idx = make(map[string]*list.Element)
+	exp.lru = list.New()
+	exp.hash = [sha256.Size]byte{}
+	store.Unlock()
+	return filepath.Walk(store.directory, func(path string, info os.FileInfo, err error) error {
+		if strings.HasSuffix(path, fileExtension) {
+			if theJwt, err := os.ReadFile(path); err == nil {
+				hash := sha256.Sum256(theJwt)
+				_, file := filepath.Split(path)
+				pkey := strings.TrimSuffix(file, fileExtension)
+				notify := isCache // for cache, issue cb even when file not present (may have been evicted)
+				if i, ok := idx[pkey]; ok {
+					notify = !bytes.Equal(i.Value.(*jwtItem).hash[:], hash[:])
+				}
+				store.Lock()
+				exp.track(pkey, &hash, string(theJwt))
+				store.Unlock()
+				if notify && changed != nil {
+					changed(pkey)
+				}
+			}
+		}
+		return nil
+	})
+}
+
+func (store *DirJWTStore) pathForKey(publicKey string) string {
+	if len(publicKey) < 2 {
+		return _EMPTY_
+	}
+	if !nkeys.IsValidPublicKey(publicKey) {
+		return _EMPTY_
+	}
+	fileName := fmt.Sprintf("%s%s", publicKey, fileExtension)
+	if store.shard {
+		last := publicKey[len(publicKey)-2:]
+		return filepath.Join(store.directory, last, fileName)
+	} else {
+		return filepath.Join(store.directory, fileName)
+	}
+}
+
+// Load checks the memory store and returns the matching JWT or an error
+// Assumes lock is NOT held
+func (store *DirJWTStore) load(publicKey string) (string, error) {
+	store.Lock()
+	defer store.Unlock()
+	if path := store.pathForKey(publicKey); path == _EMPTY_ {
+		return _EMPTY_, fmt.Errorf("invalid public key")
+	} else if data, err := os.ReadFile(path); err != nil {
+		return _EMPTY_, err
+	} else {
+		if store.expiration != nil {
+			store.expiration.updateTrack(publicKey)
+		}
+		return string(data), nil
+	}
+}
+
+// write that keeps hash of all jwt in sync
+// Assumes the lock is held. Does return true or an error never both.
+func (store *DirJWTStore) write(path string, publicKey string, theJWT string) (bool, error) {
+	if len(theJWT) == 0 {
+		return false, fmt.Errorf("invalid JWT")
+	}
+	var newHash *[sha256.Size]byte
+	if store.expiration != nil {
+		h := sha256.Sum256([]byte(theJWT))
+		newHash = &h
+		if v, ok := store.expiration.idx[publicKey]; ok {
+			store.expiration.updateTrack(publicKey)
+			// this write is an update, move to back
+			it := v.Value.(*jwtItem)
+			oldHash := it.hash[:]
+			if bytes.Equal(oldHash, newHash[:]) {
+				return false, nil
+			}
+		} else if int64(store.expiration.Len()) >= store.expiration.limit {
+			if !store.expiration.evictOnLimit {
+				return false, errors.New("jwt store is full")
+			}
+			// this write is an add, pick the least recently used value for removal
+			i := store.expiration.lru.Front().Value.(*jwtItem)
+			if err := os.Remove(store.pathForKey(i.publicKey)); err != nil {
+				return false, err
+			} else {
+				store.expiration.unTrack(i.publicKey)
+			}
+		}
+	}
+	if err := os.WriteFile(path, []byte(theJWT), defaultFilePerms); err != nil {
+		return false, err
+	} else if store.expiration != nil {
+		store.expiration.track(publicKey, newHash, theJWT)
+	}
+	return true, nil
+}
+
+func (store *DirJWTStore) delete(publicKey string) error {
+	if store.readonly {
+		return fmt.Errorf("store is read-only")
+	} else if store.deleteType == NoDelete {
+		return fmt.Errorf("store is not set up to for delete")
+	}
+	store.Lock()
+	defer store.Unlock()
+	name := store.pathForKey(publicKey)
+	if store.deleteType == RenameDeleted {
+		if err := os.Rename(name, name+".deleted"); err != nil {
+			if os.IsNotExist(err) {
+				return nil
+			}
+			return err
+		}
+	} else if err := os.Remove(name); err != nil {
+		if os.IsNotExist(err) {
+			return nil
+		}
+		return err
+	}
+	store.expiration.unTrack(publicKey)
+	store.deleted(publicKey)
+	return nil
+}
+
+// Save puts the JWT in a map by public key and performs update callbacks
+// Assumes lock is NOT held
+func (store *DirJWTStore) save(publicKey string, theJWT string) error {
+	if store.readonly {
+		return fmt.Errorf("store is read-only")
+	}
+	store.Lock()
+	path := store.pathForKey(publicKey)
+	if path == _EMPTY_ {
+		store.Unlock()
+		return fmt.Errorf("invalid public key")
+	}
+	dirPath := filepath.Dir(path)
+	if _, err := validateDirPath(dirPath); err != nil {
+		if err := os.MkdirAll(dirPath, defaultDirPerms); err != nil {
+			store.Unlock()
+			return err
+		}
+	}
+	changed, err := store.write(path, publicKey, theJWT)
+	cb := store.changed
+	store.Unlock()
+	if changed && cb != nil {
+		cb(publicKey)
+	}
+	return err
+}
+
+// Assumes the lock is NOT held, and only updates if the jwt is new, or the one on disk is older
+// When changed, invokes jwt changed callback
+func (store *DirJWTStore) saveIfNewer(publicKey string, theJWT string) error {
+	if store.readonly {
+		return fmt.Errorf("store is read-only")
+	}
+	path := store.pathForKey(publicKey)
+	if path == _EMPTY_ {
+		return fmt.Errorf("invalid public key")
+	}
+	dirPath := filepath.Dir(path)
+	if _, err := validateDirPath(dirPath); err != nil {
+		if err := os.MkdirAll(dirPath, defaultDirPerms); err != nil {
+			return err
+		}
+	}
+	if _, err := os.Stat(path); err == nil {
+		if newJWT, err := jwt.DecodeGeneric(theJWT); err != nil {
+			return err
+		} else if existing, err := os.ReadFile(path); err != nil {
+			return err
+		} else if existingJWT, err := jwt.DecodeGeneric(string(existing)); err != nil {
+			// skip if it can't be decoded
+		} else if existingJWT.ID == newJWT.ID {
+			return nil
+		} else if existingJWT.IssuedAt > newJWT.IssuedAt {
+			return nil
+		} else if newJWT.Subject != publicKey {
+			return fmt.Errorf("jwt subject nkey and provided nkey do not match")
+		} else if existingJWT.Subject != newJWT.Subject {
+			return fmt.Errorf("subject of existing and new jwt do not match")
+		}
+	}
+	store.Lock()
+	cb := store.changed
+	changed, err := store.write(path, publicKey, theJWT)
+	store.Unlock()
+	if err != nil {
+		return err
+	} else if changed && cb != nil {
+		cb(publicKey)
+	}
+	return nil
+}
+
+func xorAssign(lVal *[sha256.Size]byte, rVal [sha256.Size]byte) {
+	for i := range rVal {
+		(*lVal)[i] ^= rVal[i]
+	}
+}
+
+// returns a hash representing all indexed jwt
+func (store *DirJWTStore) Hash() [sha256.Size]byte {
+	store.Lock()
+	defer store.Unlock()
+	if store.expiration == nil {
+		return [sha256.Size]byte{}
+	} else {
+		return store.expiration.hash
+	}
+}
+
+// An jwtItem is something managed by the priority queue
+type jwtItem struct {
+	index      int
+	publicKey  string
+	expiration int64 // consists of unix time of expiration (ttl when set or jwt expiration) in seconds
+	hash       [sha256.Size]byte
+}
+
+// A expirationTracker implements heap.Interface and holds Items.
+type expirationTracker struct {
+	heap         []*jwtItem // sorted by jwtItem.expiration
+	idx          map[string]*list.Element
+	lru          *list.List // keep which jwt are least used
+	limit        int64      // limit how many jwt are being tracked
+	evictOnLimit bool       // when limit is hit, error or evict using lru
+	ttl          time.Duration
+	hash         [sha256.Size]byte // xor of all jwtItem.hash in idx
+	quit         chan struct{}
+	wg           sync.WaitGroup
+}
+
+func (q *expirationTracker) Len() int { return len(q.heap) }
+
+func (q *expirationTracker) Less(i, j int) bool {
+	pq := q.heap
+	return pq[i].expiration < pq[j].expiration
+}
+
+func (q *expirationTracker) Swap(i, j int) {
+	pq := q.heap
+	pq[i], pq[j] = pq[j], pq[i]
+	pq[i].index = i
+	pq[j].index = j
+}
+
+func (q *expirationTracker) Push(x any) {
+	n := len(q.heap)
+	item := x.(*jwtItem)
+	item.index = n
+	q.heap = append(q.heap, item)
+	q.idx[item.publicKey] = q.lru.PushBack(item)
+}
+
+func (q *expirationTracker) Pop() any {
+	old := q.heap
+	n := len(old)
+	item := old[n-1]
+	old[n-1] = nil // avoid memory leak
+	item.index = -1
+	q.heap = old[0 : n-1]
+	q.lru.Remove(q.idx[item.publicKey])
+	delete(q.idx, item.publicKey)
+	return item
+}
+
+func (pq *expirationTracker) updateTrack(publicKey string) {
+	if e, ok := pq.idx[publicKey]; ok {
+		i := e.Value.(*jwtItem)
+		if pq.ttl != 0 {
+			// only update expiration when set
+			i.expiration = time.Now().Add(pq.ttl).UnixNano()
+			heap.Fix(pq, i.index)
+		}
+		if pq.evictOnLimit {
+			pq.lru.MoveToBack(e)
+		}
+	}
+}
+
+func (pq *expirationTracker) unTrack(publicKey string) {
+	if it, ok := pq.idx[publicKey]; ok {
+		xorAssign(&pq.hash, it.Value.(*jwtItem).hash)
+		heap.Remove(pq, it.Value.(*jwtItem).index)
+		delete(pq.idx, publicKey)
+	}
+}
+
+func (pq *expirationTracker) track(publicKey string, hash *[sha256.Size]byte, theJWT string) {
+	var exp int64
+	// prioritize ttl over expiration
+	if pq.ttl != 0 {
+		if pq.ttl == time.Duration(math.MaxInt64) {
+			exp = math.MaxInt64
+		} else {
+			exp = time.Now().Add(pq.ttl).UnixNano()
+		}
+	} else {
+		if g, err := jwt.DecodeGeneric(theJWT); err == nil {
+			exp = time.Unix(g.Expires, 0).UnixNano()
+		}
+		if exp == 0 {
+			exp = math.MaxInt64 // default to indefinite
+		}
+	}
+	if e, ok := pq.idx[publicKey]; ok {
+		i := e.Value.(*jwtItem)
+		xorAssign(&pq.hash, i.hash) // remove old hash
+		i.expiration = exp
+		i.hash = *hash
+		heap.Fix(pq, i.index)
+	} else {
+		heap.Push(pq, &jwtItem{-1, publicKey, exp, *hash})
+	}
+	xorAssign(&pq.hash, *hash) // add in new hash
+}
+
+func (pq *expirationTracker) close() {
+	if pq == nil || pq.quit == nil {
+		return
+	}
+	close(pq.quit)
+	pq.quit = nil
+}
+
+func (store *DirJWTStore) startExpiring(reCheck time.Duration, limit int64, evictOnLimit bool, ttl time.Duration) {
+	store.Lock()
+	defer store.Unlock()
+	quit := make(chan struct{})
+	pq := &expirationTracker{
+		make([]*jwtItem, 0, 10),
+		make(map[string]*list.Element),
+		list.New(),
+		limit,
+		evictOnLimit,
+		ttl,
+		[sha256.Size]byte{},
+		quit,
+		sync.WaitGroup{},
+	}
+	store.expiration = pq
+	pq.wg.Add(1)
+	go func() {
+		t := time.NewTicker(reCheck)
+		defer t.Stop()
+		defer pq.wg.Done()
+		for {
+			now := time.Now().UnixNano()
+			store.Lock()
+			if pq.Len() > 0 {
+				if it := pq.heap[0]; it.expiration <= now {
+					path := store.pathForKey(it.publicKey)
+					if err := os.Remove(path); err == nil {
+						heap.Pop(pq)
+						pq.unTrack(it.publicKey)
+						xorAssign(&pq.hash, it.hash)
+						store.Unlock()
+						continue // we removed an entry, check next one right away
+					}
+				}
+			}
+			store.Unlock()
+			select {
+			case <-t.C:
+			case <-quit:
+				return
+			}
+		}
+	}()
+}

server/disk_avail.go

@@ -0,0 +1,37 @@
+// Copyright 2020-2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build !windows && !openbsd && !netbsd && !wasm && !illumos && !solaris
+
+package server
+
+import (
+	"os"
+	"syscall"
+)
+
+func diskAvailable(storeDir string) int64 {
+	var ba int64
+	if _, err := os.Stat(storeDir); os.IsNotExist(err) {
+		os.MkdirAll(storeDir, defaultDirPerms)
+	}
+	var fs syscall.Statfs_t
+	if err := syscall.Statfs(storeDir, &fs); err == nil {
+		// Estimate 75% of available storage.
+		ba = int64(uint64(fs.Bavail) * uint64(fs.Bsize) / 4 * 3)
+	} else {
+		// Used 1TB default as a guess if all else fails.
+		ba = JetStreamMaxStoreDefault
+	}
+	return ba
+}

server/disk_avail_netbsd.go

@@ -0,0 +1,21 @@
+// Copyright 2022-2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build netbsd
+
+package server
+
+// TODO - See if there is a version of this for NetBSD.
+func diskAvailable(storeDir string) int64 {
+	return JetStreamMaxStoreDefault
+}

server/disk_avail_openbsd.go

@@ -0,0 +1,37 @@
+// Copyright 2021-2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build openbsd
+
+package server
+
+import (
+	"os"
+	"syscall"
+)
+
+func diskAvailable(storeDir string) int64 {
+	var ba int64
+	if _, err := os.Stat(storeDir); os.IsNotExist(err) {
+		os.MkdirAll(storeDir, defaultDirPerms)
+	}
+	var fs syscall.Statfs_t
+	if err := syscall.Statfs(storeDir, &fs); err == nil {
+		// Estimate 75% of available storage.
+		ba = int64(uint64(fs.F_bavail) * uint64(fs.F_bsize) / 4 * 3)
+	} else {
+		// Used 1TB default as a guess if all else fails.
+		ba = JetStreamMaxStoreDefault
+	}
+	return ba
+}

server/disk_avail_solaris.go

@@ -0,0 +1,38 @@
+// Copyright 2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build illumos || solaris
+
+package server
+
+import (
+	"os"
+	"golang.org/x/sys/unix"
+)
+
+func diskAvailable(storeDir string) int64 {
+	var ba int64
+	if _, err := os.Stat(storeDir); os.IsNotExist(err) {
+		os.MkdirAll(storeDir, defaultDirPerms)
+	}
+	var fs unix.Statvfs_t
+	if err := unix.Statvfs(storeDir, &fs); err == nil {
+		// Estimate 75% of available storage.
+		ba = int64(uint64(fs.Frsize) * uint64(fs.Bavail) / 4 * 3)
+	} else {
+		// Used 1TB default as a guess if all else fails.
+		ba = JetStreamMaxStoreDefault
+	}
+	return ba
+}
+

server/disk_avail_wasm.go

@@ -0,0 +1,20 @@
+// Copyright 2022-2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build wasm
+
+package server
+
+func diskAvailable(storeDir string) int64 {
+	return JetStreamMaxStoreDefault
+}

server/disk_avail_windows.go

@@ -0,0 +1,21 @@
+// Copyright 2020-2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build windows
+
+package server
+
+// TODO(dlc) - See if there is a version of this for windows.
+func diskAvailable(storeDir string) int64 {
+	return JetStreamMaxStoreDefault
+}

server/elastic/elastic.go

@@ -0,0 +1,60 @@
+// Copyright 2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package elastic
+
+import (
+	"weak"
+)
+
+func Make[T any](ptr *T) *Pointer[T] {
+	return &Pointer[T]{
+		weak: weak.Make(ptr),
+	}
+}
+
+type Pointer[T any] struct {
+	weak   weak.Pointer[T]
+	strong *T
+}
+
+func (e *Pointer[T]) Set(ptr *T) {
+	e.weak = weak.Make(ptr)
+	if e.strong != nil {
+		e.strong = ptr
+	}
+}
+
+func (e *Pointer[T]) Strengthen() {
+	if e == nil || e.strong != nil {
+		return
+	}
+	e.strong = e.weak.Value()
+}
+
+func (e *Pointer[T]) Weaken() {
+	if e == nil || e.strong == nil {
+		return
+	}
+	e.strong = nil
+}
+
+func (e *Pointer[T]) Value() *T {
+	if e == nil {
+		return nil
+	}
+	if e.strong != nil {
+		return e.strong
+	}
+	return e.weak.Value()
+}

server/errors.go

@@ -0,0 +1,413 @@
+// Copyright 2012-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"errors"
+	"fmt"
+)
+
+var (
+	// ErrConnectionClosed represents an error condition on a closed connection.
+	ErrConnectionClosed = errors.New("connection closed")
+
+	// ErrAuthentication represents an error condition on failed authentication.
+	ErrAuthentication = errors.New("authentication error")
+
+	// ErrAuthTimeout represents an error condition on failed authorization due to timeout.
+	ErrAuthTimeout = errors.New("authentication timeout")
+
+	// ErrAuthExpired represents an expired authorization due to timeout.
+	ErrAuthExpired = errors.New("authentication expired")
+
+	// ErrAuthProxyNotTrusted represents an error condition on failed authentication
+	// due to a connection from a proxy not in the list of trusted proxies.
+	ErrAuthProxyNotTrusted = errors.New("proxy is not trusted")
+
+	// ErrAuthProxyRequired represents an error condition on failed authentication
+	// due to a connection not coming from a proxy.
+	ErrAuthProxyRequired = errors.New("proxy connection required")
+
+	// ErrMaxPayload represents an error condition when the payload is too big.
+	ErrMaxPayload = errors.New("maximum payload exceeded")
+
+	// ErrMaxControlLine represents an error condition when the control line is too big.
+	ErrMaxControlLine = errors.New("maximum control line exceeded")
+
+	// ErrReservedPublishSubject represents an error condition when sending to a reserved subject, e.g. _SYS.>
+	ErrReservedPublishSubject = errors.New("reserved internal subject")
+
+	// ErrBadPublishSubject represents an error condition for an invalid publish subject.
+	ErrBadPublishSubject = errors.New("invalid publish subject")
+
+	// ErrBadSubject represents an error condition for an invalid subject.
+	ErrBadSubject = errors.New("invalid subject")
+
+	// ErrBadQualifier is used to error on a bad qualifier for a transform.
+	ErrBadQualifier = errors.New("bad qualifier")
+
+	// ErrBadClientProtocol signals a client requested an invalid client protocol.
+	ErrBadClientProtocol = errors.New("invalid client protocol")
+
+	// ErrTooManyConnections signals a client that the maximum number of connections supported by the
+	// server has been reached.
+	ErrTooManyConnections = errors.New("maximum connections exceeded")
+
+	// ErrTooManyAccountConnections signals that an account has reached its maximum number of active
+	// connections.
+	ErrTooManyAccountConnections = errors.New("maximum account active connections exceeded")
+
+	// ErrLeafNodeLoop signals a leafnode is trying to register for a cluster we already have registered.
+	ErrLeafNodeLoop = errors.New("leafnode loop detected")
+
+	// ErrTooManySubs signals a client that the maximum number of subscriptions per connection
+	// has been reached.
+	ErrTooManySubs = errors.New("maximum subscriptions exceeded")
+
+	// ErrTooManySubTokens signals a client that the subject has too many tokens.
+	ErrTooManySubTokens = errors.New("subject has exceeded number of tokens limit")
+
+	// ErrClientConnectedToRoutePort represents an error condition when a client
+	// attempted to connect to the route listen port.
+	ErrClientConnectedToRoutePort = errors.New("attempted to connect to route port")
+
+	// ErrClientConnectedToLeafNodePort represents an error condition when a client
+	// attempted to connect to the leaf node listen port.
+	ErrClientConnectedToLeafNodePort = errors.New("attempted to connect to leaf node port")
+
+	// ErrLeafNodeHasSameClusterName represents an error condition when a leafnode is a cluster
+	// and it has the same cluster name as the hub cluster.
+	ErrLeafNodeHasSameClusterName = errors.New("remote leafnode has same cluster name")
+
+	// ErrLeafNodeDisabled is when we disable leafnodes.
+	ErrLeafNodeDisabled = errors.New("leafnodes disabled")
+
+	// ErrConnectedToWrongPort represents an error condition when a connection is attempted
+	// to the wrong listen port (for instance a LeafNode to a client port, etc...)
+	ErrConnectedToWrongPort = errors.New("attempted to connect to wrong port")
+
+	// ErrAccountExists is returned when an account is attempted to be registered
+	// but already exists.
+	ErrAccountExists = errors.New("account exists")
+
+	// ErrBadAccount represents a malformed or incorrect account.
+	ErrBadAccount = errors.New("bad account")
+
+	// ErrReservedAccount represents a reserved account that can not be created.
+	ErrReservedAccount = errors.New("reserved account")
+
+	// ErrMissingAccount is returned when an account does not exist.
+	ErrMissingAccount = errors.New("account missing")
+
+	// ErrMissingService is returned when an account does not have an exported service.
+	ErrMissingService = errors.New("service missing")
+
+	// ErrBadServiceType is returned when latency tracking is being applied to non-singleton response types.
+	ErrBadServiceType = errors.New("bad service response type")
+
+	// ErrBadSampling is returned when the sampling for latency tracking is not 1 >= sample <= 100.
+	ErrBadSampling = errors.New("bad sampling percentage, should be 1-100")
+
+	// ErrAccountValidation is returned when an account has failed validation.
+	ErrAccountValidation = errors.New("account validation failed")
+
+	// ErrAccountExpired is returned when an account has expired.
+	ErrAccountExpired = errors.New("account expired")
+
+	// ErrNoAccountResolver is returned when we attempt an update but do not have an account resolver.
+	ErrNoAccountResolver = errors.New("account resolver missing")
+
+	// ErrAccountResolverUpdateTooSoon is returned when we attempt an update too soon to last request.
+	ErrAccountResolverUpdateTooSoon = errors.New("account resolver update too soon")
+
+	// ErrAccountResolverSameClaims is returned when same claims have been fetched.
+	ErrAccountResolverSameClaims = errors.New("account resolver no new claims")
+
+	// ErrStreamImportAuthorization is returned when a stream import is not authorized.
+	ErrStreamImportAuthorization = errors.New("stream import not authorized")
+
+	// ErrStreamImportBadPrefix is returned when a stream import prefix contains wildcards.
+	ErrStreamImportBadPrefix = errors.New("stream import prefix can not contain wildcard tokens")
+
+	// ErrStreamImportDuplicate is returned when a stream import is a duplicate of one that already exists.
+	ErrStreamImportDuplicate = errors.New("stream import already exists")
+
+	// ErrServiceImportAuthorization is returned when a service import is not authorized.
+	ErrServiceImportAuthorization = errors.New("service import not authorized")
+
+	// ErrImportFormsCycle is returned when an import would form a cycle.
+	ErrImportFormsCycle = errors.New("import forms a cycle")
+
+	// ErrCycleSearchDepth is returned when we have exceeded our maximum search depth..
+	ErrCycleSearchDepth = errors.New("search cycle depth exhausted")
+
+	// ErrClientOrRouteConnectedToGatewayPort represents an error condition when
+	// a client or route attempted to connect to the Gateway port.
+	ErrClientOrRouteConnectedToGatewayPort = errors.New("attempted to connect to gateway port")
+
+	// ErrWrongGateway represents an error condition when a server receives a connect
+	// request from a remote Gateway with a destination name that does not match the server's
+	// Gateway's name.
+	ErrWrongGateway = errors.New("wrong gateway")
+
+	// ErrGatewayNameHasSpaces signals that the gateway name contains spaces, which is not allowed.
+	ErrGatewayNameHasSpaces = errors.New("gateway name cannot contain spaces")
+
+	// ErrNoSysAccount is returned when an attempt to publish or subscribe is made
+	// when there is no internal system account defined.
+	ErrNoSysAccount = errors.New("system account not setup")
+
+	// ErrRevocation is returned when a credential has been revoked.
+	ErrRevocation = errors.New("credentials have been revoked")
+
+	// ErrServerNotRunning is used to signal an error that a server is not running.
+	ErrServerNotRunning = errors.New("server is not running")
+
+	// ErrServerNameHasSpaces signals that the server name contains spaces, which is not allowed.
+	ErrServerNameHasSpaces = errors.New("server name cannot contain spaces")
+
+	// ErrBadMsgHeader signals the parser detected a bad message header
+	ErrBadMsgHeader = errors.New("bad message header detected")
+
+	// ErrMsgHeadersNotSupported signals the parser detected a message header
+	// but they are not supported on this server.
+	ErrMsgHeadersNotSupported = errors.New("message headers not supported")
+
+	// ErrNoRespondersRequiresHeaders signals that a client needs to have headers
+	// on if they want no responders behavior.
+	ErrNoRespondersRequiresHeaders = errors.New("no responders requires headers support")
+
+	// ErrClusterNameConfigConflict signals that the options for cluster name in cluster and gateway are in conflict.
+	ErrClusterNameConfigConflict = errors.New("cluster name conflicts between cluster and gateway definitions")
+
+	// ErrClusterNameRemoteConflict signals that a remote server has a different cluster name.
+	ErrClusterNameRemoteConflict = errors.New("cluster name from remote server conflicts")
+
+	// ErrClusterNameHasSpaces signals that the cluster name contains spaces, which is not allowed.
+	ErrClusterNameHasSpaces = errors.New("cluster name cannot contain spaces")
+
+	// ErrClusterNameReserved signals that the cluster name is reserved for internal protocol use.
+	ErrClusterNameReserved = errors.New("cluster name is reserved")
+
+	// ErrMalformedSubject is returned when a subscription is made with a subject that does not conform to subject rules.
+	ErrMalformedSubject = errors.New("malformed subject")
+
+	// ErrSubscribePermissionViolation is returned when processing of a subscription fails due to permissions.
+	ErrSubscribePermissionViolation = errors.New("subscribe permission violation")
+
+	// ErrNoTransforms signals no subject transforms are available to map this subject.
+	ErrNoTransforms = errors.New("no matching transforms available")
+
+	// ErrCertNotPinned is returned when pinned certs are set and the certificate is not in it
+	ErrCertNotPinned = errors.New("certificate not pinned")
+
+	// ErrDuplicateServerName is returned when processing a server remote connection and
+	// the server reports that this server name is already used in the cluster.
+	ErrDuplicateServerName = errors.New("duplicate server name")
+
+	// ErrMinimumVersionRequired is returned when a connection is not at the minimum version required.
+	ErrMinimumVersionRequired = errors.New("minimum version required")
+	// ErrLeafNodeMinVersionRejected is the leafnode protocol error prefix used
+	// when rejecting a remote due to leafnodes.min_version.
+	ErrLeafNodeMinVersionRejected = errors.New("connection rejected since minimum version required is")
+
+	// ErrInvalidMappingDestination is used for all subject mapping destination errors
+	ErrInvalidMappingDestination = errors.New("invalid mapping destination")
+
+	// ErrInvalidMappingDestinationSubject is used to error on a bad transform destination mapping
+	ErrInvalidMappingDestinationSubject = fmt.Errorf("%w: invalid transform", ErrInvalidMappingDestination)
+
+	// ErrMappingDestinationNotUsingAllWildcards is used to error on a transform destination not using all of the token wildcards
+	ErrMappingDestinationNotUsingAllWildcards = fmt.Errorf("%w: not using all of the token wildcard(s)", ErrInvalidMappingDestination)
+
+	// ErrUnknownMappingDestinationFunction is returned when a subject mapping destination contains an unknown mustache-escaped mapping function.
+	ErrUnknownMappingDestinationFunction = fmt.Errorf("%w: unknown function", ErrInvalidMappingDestination)
+
+	// ErrMappingDestinationIndexOutOfRange is returned when the mapping destination function is passed an out of range wildcard index value for one of it's arguments
+	ErrMappingDestinationIndexOutOfRange = fmt.Errorf("%w: wildcard index out of range", ErrInvalidMappingDestination)
+
+	// ErrMappingDestinationNotEnoughArgs is returned when the mapping destination function is not passed enough arguments
+	ErrMappingDestinationNotEnoughArgs = fmt.Errorf("%w: not enough arguments passed to the function", ErrInvalidMappingDestination)
+
+	// ErrMappingDestinationInvalidArg is returned when the mapping destination function is passed and invalid argument
+	ErrMappingDestinationInvalidArg = fmt.Errorf("%w: function argument is invalid or in the wrong format", ErrInvalidMappingDestination)
+
+	// ErrMappingDestinationTooManyArgs is returned when the mapping destination function is passed too many arguments
+	ErrMappingDestinationTooManyArgs = fmt.Errorf("%w: too many arguments passed to the function", ErrInvalidMappingDestination)
+
+	// ErrMappingDestinationNotSupportedForImport is returned when you try to use a mapping function other than wildcard in a transform that needs to be reversible (i.e. an import)
+	ErrMappingDestinationNotSupportedForImport = fmt.Errorf("%w: the only mapping function allowed for import transforms is {{Wildcard()}}", ErrInvalidMappingDestination)
+)
+
+// mappingDestinationErr is a type of subject mapping destination error
+type mappingDestinationErr struct {
+	token string
+	err   error
+}
+
+func (e *mappingDestinationErr) Error() string {
+	if e.token == _EMPTY_ {
+		return e.err.Error()
+	}
+	return fmt.Sprintf("%s in %s", e.err, e.token)
+}
+
+func (e *mappingDestinationErr) Is(target error) bool {
+	return target == ErrInvalidMappingDestination
+}
+
+// configErr is a configuration error.
+type configErr struct {
+	token  token
+	reason string
+}
+
+// Source reports the location of a configuration error.
+func (e *configErr) Source() string {
+	return fmt.Sprintf("%s:%d:%d", e.token.SourceFile(), e.token.Line(), e.token.Position())
+}
+
+// Error reports the location and reason from a configuration error.
+func (e *configErr) Error() string {
+	if e.token != nil {
+		return fmt.Sprintf("%s: %s", e.Source(), e.reason)
+	}
+	return e.reason
+}
+
+// unknownConfigFieldErr is an error reported in pedantic mode.
+type unknownConfigFieldErr struct {
+	configErr
+	field string
+}
+
+// Error reports that an unknown field was in the configuration.
+func (e *unknownConfigFieldErr) Error() string {
+	return fmt.Sprintf("%s: unknown field %q", e.Source(), e.field)
+}
+
+// configWarningErr is an error reported in pedantic mode.
+type configWarningErr struct {
+	configErr
+	field string
+}
+
+// Error reports a configuration warning.
+func (e *configWarningErr) Error() string {
+	return fmt.Sprintf("%s: invalid use of field %q: %s", e.Source(), e.field, e.reason)
+}
+
+// processConfigErr is the result of processing the configuration from the server.
+type processConfigErr struct {
+	errors   []error
+	warnings []error
+}
+
+// Error returns the collection of errors separated by new lines,
+// warnings appear first then hard errors.
+func (e *processConfigErr) Error() string {
+	var msg string
+	for _, err := range e.Warnings() {
+		msg += err.Error() + "\n"
+	}
+	for _, err := range e.Errors() {
+		msg += err.Error() + "\n"
+	}
+	return msg
+}
+
+// Warnings returns the list of warnings.
+func (e *processConfigErr) Warnings() []error {
+	return e.warnings
+}
+
+// Errors returns the list of errors.
+func (e *processConfigErr) Errors() []error {
+	return e.errors
+}
+
+// errCtx wraps an error and stores additional ctx information for tracing.
+// Does not print or return it unless explicitly requested.
+type errCtx struct {
+	error
+	ctx string
+}
+
+func NewErrorCtx(err error, format string, args ...any) error {
+	return &errCtx{err, fmt.Sprintf(format, args...)}
+}
+
+// Unwrap implement to work with errors.Is and errors.As
+func (e *errCtx) Unwrap() error {
+	if e == nil {
+		return nil
+	}
+	return e.error
+}
+
+// Context for error
+func (e *errCtx) Context() string {
+	if e == nil {
+		return ""
+	}
+	return e.ctx
+}
+
+// UnpackIfErrorCtx return Error or, if type is right error and context
+func UnpackIfErrorCtx(err error) string {
+	if e, ok := err.(*errCtx); ok {
+		if _, ok := e.error.(*errCtx); ok {
+			return fmt.Sprint(UnpackIfErrorCtx(e.error), ": ", e.Context())
+		}
+		return fmt.Sprint(e.Error(), ": ", e.Context())
+	}
+	return err.Error()
+}
+
+// implements: go 1.13 errors.Unwrap(err error) error
+// TODO replace with native code once we no longer support go1.12
+func errorsUnwrap(err error) error {
+	u, ok := err.(interface {
+		Unwrap() error
+	})
+	if !ok {
+		return nil
+	}
+	return u.Unwrap()
+}
+
+// ErrorIs implements: go 1.13 errors.Is(err, target error) bool
+// TODO replace with native code once we no longer support go1.12
+func ErrorIs(err, target error) bool {
+	// this is an outright copy of go 1.13 errors.Is(err, target error) bool
+	// removed isComparable
+	if err == nil || target == nil {
+		return err == target
+	}
+
+	for {
+		if err == target {
+			return true
+		}
+		if x, ok := err.(interface{ Is(error) bool }); ok && x.Is(target) {
+			return true
+		}
+		// TODO: consider supporing target.Is(err). This would allow
+		// user-definable predicates, but also may allow for coping with sloppy
+		// APIs, thereby making it easier to get away with them.
+		if err = errorsUnwrap(err); err == nil {
+			return false
+		}
+	}
+}

server/errors_gen.go

@@ -0,0 +1,238 @@
+//go:build ignore
+
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"log"
+	"os"
+	"os/exec"
+	"regexp"
+	"sort"
+	"strings"
+	"text/template"
+
+	"github.com/nats-io/nats-server/v2/server"
+)
+
+var tagRe = regexp.MustCompile("\\{(.+?)}")
+
+var templ = `
+// Generated code, do not edit. See errors.json and run go generate to update
+
+package server
+
+import "strings"
+
+const (
+{{- range $i, $error := . }}
+{{- if .Comment }}
+	// {{ .Constant }} {{ .Comment }} ({{ .Description | print }})
+{{- else }}
+	// {{ .Constant }} {{ .Description | print }}
+{{- end }}
+	{{ .Constant }} ErrorIdentifier = {{ .ErrCode }}
+{{ end }}
+)
+
+var (
+	ApiErrors = map[ErrorIdentifier]*ApiError{
+{{- range $i, $error := . }}
+		{{ .Constant }}: {Code: {{ .Code }},ErrCode: {{ .ErrCode }},Description: {{ .Description | printf "%q" }}},{{- end }}
+	}
+
+{{- range $i, $error := . }}
+{{- if .Deprecates }}
+// {{ .Deprecates }} Deprecated by {{ .Constant }} ApiError, use IsNatsError() for comparisons
+{{ .Deprecates }} = ApiErrors[{{ .Constant }}]
+{{- end }}
+{{- end }}
+)
+
+{{- range $i, $error := . }}
+// {{ .Constant | funcNameForConstant }} creates a new {{ .Constant }} error: {{ .Description | printf "%q" }}
+func {{ .Constant | funcNameForConstant }}({{ .Description | funcArgsForTags }}) *ApiError {
+    eopts := parseOpts(opts)
+	if ae, ok := eopts.err.(*ApiError); ok {
+		return ae
+ 	}
+{{ if .Description | hasTags }}
+	e:=ApiErrors[{{.Constant}}]
+	args:=e.toReplacerArgs([]interface{}{ {{.Description | replacerArgsForTags }} })
+	return &ApiError{
+		Code:        e.Code,
+		ErrCode:     e.ErrCode,
+		Description: strings.NewReplacer(args...).Replace(e.Description),
+	}
+{{- else }}
+	return ApiErrors[{{.Constant}}]
+{{- end }}
+}
+
+{{- end }}
+`
+
+func panicIfErr(err error) {
+	if err == nil {
+		return
+	}
+	panic(err)
+}
+
+func goFmt(file string) error {
+	c := exec.Command("go", "fmt", file)
+	out, err := c.CombinedOutput()
+	if err != nil {
+		log.Printf("go fmt failed: %s", string(out))
+	}
+
+	return err
+}
+
+func checkIncrements(errs []server.ErrorsData) error {
+	sort.Slice(errs, func(i, j int) bool {
+		return errs[i].ErrCode < errs[j].ErrCode
+	})
+
+	last := errs[0].ErrCode
+	gaps := []uint16{}
+
+	for i := 1; i < len(errs); i++ {
+		if errs[i].ErrCode != last+1 {
+			gaps = append(gaps, last)
+		}
+		last = errs[i].ErrCode
+	}
+
+	if len(gaps) > 0 {
+		return fmt.Errorf("gaps found in sequences: %v", gaps)
+	}
+
+	return nil
+}
+
+func checkDupes(errs []server.ErrorsData) error {
+	codes := []uint16{}
+	highest := uint16(0)
+	for _, err := range errs {
+		codes = append(codes, err.ErrCode)
+		if highest < err.ErrCode {
+			highest = err.ErrCode
+		}
+	}
+
+	codeKeys := make(map[uint16]bool)
+	constKeys := make(map[string]bool)
+
+	for _, entry := range errs {
+		if _, found := codeKeys[entry.ErrCode]; found {
+			return fmt.Errorf("duplicate error code %+v, highest code is %d", entry, highest)
+		}
+
+		if _, found := constKeys[entry.Constant]; found {
+			return fmt.Errorf("duplicate error constant %+v", entry)
+		}
+
+		codeKeys[entry.ErrCode] = true
+		constKeys[entry.Constant] = true
+	}
+
+	return nil
+}
+
+func findTags(d string) []string {
+	tags := []string{}
+	for _, tag := range tagRe.FindAllStringSubmatch(d, -1) {
+		if len(tag) != 2 {
+			continue
+		}
+
+		tags = append(tags, tag[1])
+	}
+
+	sort.Strings(tags)
+
+	return tags
+}
+
+func main() {
+	ej, err := os.ReadFile("server/errors.json")
+	panicIfErr(err)
+
+	errs := []server.ErrorsData{}
+	panicIfErr(json.Unmarshal(ej, &errs))
+
+	panicIfErr(checkDupes(errs))
+	panicIfErr(checkIncrements(errs))
+
+	sort.Slice(errs, func(i, j int) bool {
+		return errs[i].Constant < errs[j].Constant
+	})
+
+	t := template.New("errors").Funcs(
+		template.FuncMap{
+			"inc": func(i int) int { return i + 1 },
+			"hasTags": func(d string) bool {
+				return strings.Contains(d, "{") && strings.Contains(d, "}")
+			},
+			"replacerArgsForTags": func(d string) string {
+				res := []string{}
+				for _, tag := range findTags(d) {
+					res = append(res, `"{`+tag+`}"`)
+					res = append(res, tag)
+				}
+
+				return strings.Join(res, ", ")
+			},
+			"funcArgsForTags": func(d string) string {
+				res := []string{}
+				for _, tag := range findTags(d) {
+					if tag == "err" {
+						res = append(res, "err error")
+					} else if tag == "seq" {
+						res = append(res, "seq uint64")
+					} else {
+						res = append(res, fmt.Sprintf("%s interface{}", tag))
+					}
+				}
+
+				res = append(res, "opts ...ErrorOption")
+
+				return strings.Join(res, ", ")
+			},
+			"funcNameForConstant": func(c string) string {
+				res := ""
+
+				switch {
+				case strings.HasSuffix(c, "ErrF"):
+					res = fmt.Sprintf("New%sError", strings.TrimSuffix(c, "ErrF"))
+				case strings.HasSuffix(c, "Err"):
+					res = fmt.Sprintf("New%sError", strings.TrimSuffix(c, "Err"))
+				case strings.HasSuffix(c, "ErrorF"):
+					res = fmt.Sprintf("New%s", strings.TrimSuffix(c, "F"))
+				case strings.HasSuffix(c, "F"):
+					res = fmt.Sprintf("New%sError", strings.TrimSuffix(c, "F"))
+				default:
+					res = fmt.Sprintf("New%s", c)
+				}
+
+				if !strings.HasSuffix(res, "Error") {
+					res = fmt.Sprintf("%sError", res)
+				}
+
+				return res
+			},
+		})
+	p, err := t.Parse(templ)
+	panicIfErr(err)
+
+	tf, err := os.CreateTemp("", "")
+	panicIfErr(err)
+	defer tf.Close()
+
+	panicIfErr(p.Execute(tf, errs))
+
+	panicIfErr(os.Rename(tf.Name(), "server/jetstream_errors_generated.go"))
+	panicIfErr(goFmt("server/jetstream_errors_generated.go"))
+}

server/errors_test.go

@@ -0,0 +1,74 @@
+// Copyright 2020-2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestErrCtx(t *testing.T) {
+	ctx := "Extra context information"
+	e := NewErrorCtx(ErrWrongGateway, "%s", ctx)
+
+	if e.Error() != ErrWrongGateway.Error() {
+		t.Fatalf("%v and %v are supposed to be identical", e, ErrWrongGateway)
+	}
+	if e == ErrWrongGateway {
+		t.Fatalf("%v and %v can't be compared this way", e, ErrWrongGateway)
+	}
+	if !ErrorIs(e, ErrWrongGateway) {
+		t.Fatalf("%s and %s ", e, ErrWrongGateway)
+	}
+	if UnpackIfErrorCtx(ErrWrongGateway) != ErrWrongGateway.Error() {
+		t.Fatalf("Error of different type should be processed unchanged")
+	}
+	trace := UnpackIfErrorCtx(e)
+	if !strings.HasPrefix(trace, ErrWrongGateway.Error()) {
+		t.Fatalf("original error needs to remain")
+	}
+	if !strings.HasSuffix(trace, ctx) {
+		t.Fatalf("ctx needs to be added")
+	}
+}
+
+func TestErrCtxWrapped(t *testing.T) {
+	ctxO := "Original Ctx"
+	eO := NewErrorCtx(ErrWrongGateway, "%s", ctxO)
+	ctx := "Extra context information"
+	e := NewErrorCtx(eO, "%s", ctx)
+
+	if e.Error() != ErrWrongGateway.Error() {
+		t.Fatalf("%v and %v are supposed to be identical", e, ErrWrongGateway)
+	}
+	if e == ErrWrongGateway {
+		t.Fatalf("%v and %v can't be compared this way", e, ErrWrongGateway)
+	}
+	if !ErrorIs(e, ErrWrongGateway) {
+		t.Fatalf("%s and %s ", e, ErrWrongGateway)
+	}
+	if UnpackIfErrorCtx(ErrWrongGateway) != ErrWrongGateway.Error() {
+		t.Fatalf("Error of different type should be processed unchanged")
+	}
+	trace := UnpackIfErrorCtx(e)
+	if !strings.HasPrefix(trace, ErrWrongGateway.Error()) {
+		t.Fatalf("original error needs to remain")
+	}
+	if !strings.HasSuffix(trace, ctx) {
+		t.Fatalf("ctx needs to be added")
+	}
+	if !strings.Contains(trace, ctxO) {
+		t.Fatalf("Needs to contain every context")
+	}
+}

server/feature_flags.go

@@ -0,0 +1,157 @@
+// Copyright 2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"maps"
+	"slices"
+	"strings"
+)
+
+const (
+	FeatureFlagJsAckFormatV2      = "js_ack_fc_v2"
+	FeatureFlagJsAPIReplyFormatV2 = "js_api_reply_v2"
+	FeatureFlagJsRaftDeleteRange  = "js_raft_delete_range"
+	FeatureFlagJsSnapshotSources  = "js_snapshot_sources"
+)
+
+var featureFlags = map[string]bool{
+	// Use v2 format for `$JS.ACK.>` and `$JS.FC.>`.
+	// - Introduced: 2.14.0, both v1 and v2 supported, only using v1.
+	// - Enabled: TBD, both supported, v2 becomes the default.
+	//
+	// - v1: $JS.ACK.<stream name>.<consumer name>.<num delivered>.<stream sequence>.<consumer sequence>.<timestamp>.<num pending>
+	// - v2: $JS.ACK.<domain>.<account hash>.<stream name>.<consumer name>.<num delivered>.<stream sequence>.<consumer sequence>.<timestamp>.<num pending>
+	// See also: https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-15.md#jsack
+	FeatureFlagJsAckFormatV2: false,
+
+	// Use a scoped v2 format for replies to internally-issued JetStream API
+	// requests. This permits a leafnode to authorize only the replies for a
+	// specific remote stream/consumer instead of every $JSC.R reply in an
+	// account.
+	//
+	// - v1: $JSC.R.<uid>
+	// - v2: $JSC.R.<domain>.<account hash>.<stream name>.<consumer name>.<uid>
+	//
+	// The reply subject is opaque to the remote API server, so this is safe to
+	// enable independently on the server issuing the request.
+	FeatureFlagJsAPIReplyFormatV2: false,
+
+	// Propose delete range gaps as a single `deleteRangeOp` Raft append entry
+	// instead of one entry per deleted sequence. Dramatically reduces Raft cost
+	// on mirrors whose origin has a large number of interior deletes.
+	// - Introduced: 2.14.0, apply-side always supports receiving `deleteRangeOp`.
+	// - Enabled: 2.15.0, when upgrading all servers should be 2.14.0+
+	// - To be removed in 2.16.0
+	//
+	// WARNING: Only enable once every peer in the cluster is on a version that
+	// supports receiving `deleteRangeOp`. Older peers panic on apply of an
+	// unknown stream entry operation.
+	FeatureFlagJsRaftDeleteRange: true,
+
+	// Include the stream's sourcing state in the replicated stream snapshot.
+	// Unlike other per-message derived state, it outlives the messages it was
+	// collected from, so a replica that caught up after those messages were
+	// removed can't derive it locally and would resume sourcing from an earlier
+	// position once it becomes leader.
+	// - Introduced: 2.15.0, both encoding versions are accepted, only emitting v1.
+	// - Enabled: TBD, once all supported versions accept v2.
+	//
+	// WARNING: Only enable once every peer in the cluster is on a version that
+	// accepts the v2 encoding. Older peers reject the snapshot outright.
+	FeatureFlagJsSnapshotSources: false,
+}
+
+// getFeatureFlag is used to retrieve either the default or overwritten value for a feature flag.
+// The user's value takes precedence over the system's default. However, if the flag doesn't exist, it's disabled.
+// The *Options returned by Server.getOpts() is treated as immutable, mutations go through setOpts,
+// so no lock is required on the map read here.
+func (o *Options) getFeatureFlag(k string) bool {
+	defaultValue, ok := featureFlags[k]
+	if !ok {
+		return false // Not supported.
+	}
+	if userValue, ok := o.FeatureFlags[k]; ok {
+		return userValue
+	}
+	return defaultValue
+}
+
+// getMergedFeatureFlags returns a merged map of feature flags, with the user's values taking precedence.
+func (o *Options) getMergedFeatureFlags() map[string]bool {
+	merged := make(map[string]bool)
+	for k, v := range featureFlags {
+		merged[k] = v
+	}
+	for k, v := range o.FeatureFlags {
+		if _, ok := featureFlags[k]; !ok {
+			continue
+		}
+		merged[k] = v
+	}
+	return merged
+}
+
+// printFeatureFlags logs the currently used feature flags on server startup.
+func (s *Server) printFeatureFlags(o *Options) {
+	if len(o.FeatureFlags) == 0 {
+		return
+	}
+	keys := slices.Sorted(maps.Keys(o.FeatureFlags))
+
+	var (
+		configured  strings.Builder
+		unsupported strings.Builder
+	)
+
+	for _, k := range keys {
+		// Unsupported
+		defaultValue, ok := featureFlags[k]
+		if !ok {
+			if unsupported.Len() > 0 {
+				unsupported.WriteString(", ")
+			}
+			unsupported.WriteString(k)
+			continue
+		}
+
+		v := o.FeatureFlags[k]
+		if configured.Len() > 0 {
+			configured.WriteString(", ")
+		}
+		configured.WriteString(k)
+		configured.WriteString(" (")
+		if defaultValue {
+			if v {
+				configured.WriteString("enabled")
+			} else {
+				configured.WriteString("opt-out")
+			}
+		} else if v {
+			configured.WriteString("opt-in")
+		} else {
+			configured.WriteString("disabled")
+		}
+		configured.WriteString(")")
+	}
+	if configured.Len() == 0 {
+		configured.WriteString("none")
+	}
+
+	s.Noticef("  Feature flags:")
+	s.Noticef("    Configured:    %s", configured.String())
+	if unsupported.Len() > 0 {
+		s.Noticef("    Unsupported:   %s", unsupported.String())
+	}
+}

server/feature_flags_test.go

@@ -0,0 +1,69 @@
+// Copyright 2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"reflect"
+	"testing"
+)
+
+func TestGetFeatureFlag(t *testing.T) {
+	featureFlags["test_flag"] = true
+	t.Cleanup(func() { delete(featureFlags, "test_flag") })
+
+	// Unset feature flags gets defaults.
+	o := &Options{}
+	require_Equal(t, o.getFeatureFlag(""), false)
+	require_Equal(t, o.getFeatureFlag("test_flag"), true)
+
+	// User disables the feature flag, it takes precedence.
+	o.FeatureFlags = map[string]bool{"test_flag": false}
+	require_Equal(t, o.getFeatureFlag("test_flag"), false)
+
+	// User enables the feature flag, but we don't support it anymore.
+	// It's allowed for the user to provide stale flags, but we shouldn't return they're enabled.
+	delete(featureFlags, "test_flag")
+	o.FeatureFlags = map[string]bool{"test_flag": true}
+	require_Equal(t, o.getFeatureFlag("test_flag"), false)
+}
+
+func TestFeatureFlagsConfigParseNonBool(t *testing.T) {
+	conf := createConfFile(t, []byte(`
+		feature_flags {
+		    opt_in_flag: "true"
+		}
+	`))
+
+	_, err := ProcessConfigFile(conf)
+	require_Error(t, err)
+	require_Contains(t, err.Error(), `error parsing feature flag "opt_in_flag": expected bool, got string`)
+}
+
+func TestGetMergedFeatureFlags(t *testing.T) {
+	previous := featureFlags
+	featureFlags = make(map[string]bool)
+	featureFlags["test_flag"] = true
+	featureFlags["test_flag_default"] = true
+	t.Cleanup(func() { featureFlags = previous })
+
+	o := &Options{}
+	o.FeatureFlags = map[string]bool{"test_flag": false, "unknown_flag": false}
+
+	// The merged flags should only contain known flags, with the user's values taking precedence.
+	merged := o.getMergedFeatureFlags()
+	expected := map[string]bool{"test_flag": false, "test_flag_default": true}
+	if !reflect.DeepEqual(expected, merged) {
+		t.Fatalf("Feature flags are incorrect.\nexpected: %+v\ngot: %+v", expected, merged)
+	}
+}

server/gsl/gsl.go

@@ -0,0 +1,566 @@
+// Copyright 2025-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package gsl
+
+import (
+	"errors"
+	"strings"
+	"sync"
+)
+
+// Sublist is a routing mechanism to handle subject distribution and
+// provides a facility to match subjects from published messages to
+// interested subscribers. Subscribers can have wildcard subjects to
+// match multiple published subjects.
+
+// Common byte variables for wildcards and token separator.
+const (
+	pwc     = '*'
+	pwcs    = "*"
+	fwc     = '>'
+	fwcs    = ">"
+	tsep    = "."
+	btsep   = '.'
+	_EMPTY_ = ""
+)
+
+// Sublist related errors
+var (
+	ErrInvalidSubject    = errors.New("gsl: invalid subject")
+	ErrNotFound          = errors.New("gsl: no matches found")
+	ErrNilChan           = errors.New("gsl: nil channel")
+	ErrAlreadyRegistered = errors.New("gsl: notification already registered")
+)
+
+// SimpleSublist is an alias type for GenericSublist that takes
+// empty values, useful for tracking interest only without any
+// unnecessary allocations.
+type SimpleSublist = GenericSublist[struct{}]
+
+// NewSimpleSublist will create a simple sublist.
+func NewSimpleSublist() *SimpleSublist {
+	return &GenericSublist[struct{}]{root: newLevel[struct{}]()}
+}
+
+// A GenericSublist stores and efficiently retrieves subscriptions.
+type GenericSublist[T comparable] struct {
+	sync.RWMutex
+	root  *level[T]
+	count uint32
+}
+
+// A node contains subscriptions and a pointer to the next level.
+type node[T comparable] struct {
+	next *level[T]
+	subs map[T]string // value -> subject
+}
+
+// A level represents a group of nodes and special pointers to
+// wildcard nodes.
+type level[T comparable] struct {
+	nodes    map[string]*node[T]
+	pwc, fwc *node[T]
+}
+
+// Create a new default node.
+func newNode[T comparable]() *node[T] {
+	return &node[T]{subs: make(map[T]string)}
+}
+
+// Create a new default level.
+func newLevel[T comparable]() *level[T] {
+	return &level[T]{nodes: make(map[string]*node[T])}
+}
+
+// NewSublist will create a default sublist with caching enabled per the flag.
+func NewSublist[T comparable]() *GenericSublist[T] {
+	return &GenericSublist[T]{root: newLevel[T]()}
+}
+
+// Insert adds a subscription into the sublist
+func (s *GenericSublist[T]) Insert(subject string, value T) error {
+	s.Lock()
+
+	var sfwc bool
+	var n *node[T]
+	l := s.root
+
+	for t := range strings.SplitSeq(subject, tsep) {
+		lt := len(t)
+		if lt == 0 || sfwc {
+			s.Unlock()
+			return ErrInvalidSubject
+		}
+
+		if lt > 1 {
+			n = l.nodes[t]
+		} else {
+			switch t[0] {
+			case pwc:
+				n = l.pwc
+			case fwc:
+				n = l.fwc
+				sfwc = true
+			default:
+				n = l.nodes[t]
+			}
+		}
+		if n == nil {
+			n = newNode[T]()
+			if lt > 1 {
+				l.nodes[t] = n
+			} else {
+				switch t[0] {
+				case pwc:
+					l.pwc = n
+				case fwc:
+					l.fwc = n
+				default:
+					l.nodes[t] = n
+				}
+			}
+		}
+		if n.next == nil {
+			n.next = newLevel[T]()
+		}
+		l = n.next
+	}
+
+	n.subs[value] = subject
+
+	s.count++
+	s.Unlock()
+
+	return nil
+}
+
+// Match will match all entries to the literal subject.
+// It will return a set of results for both normal and queue subscribers.
+func (s *GenericSublist[T]) Match(subject string, cb func(T)) {
+	s.match(subject, cb, true)
+}
+
+// MatchBytes will match all entries to the literal subject.
+// It will return a set of results for both normal and queue subscribers.
+func (s *GenericSublist[T]) MatchBytes(subject []byte, cb func(T)) {
+	s.match(string(subject), cb, true)
+}
+
+// HasInterest will return whether or not there is any interest in the subject.
+// In cases where more detail is not required, this may be faster than Match.
+func (s *GenericSublist[T]) HasInterest(subject string) bool {
+	return s.hasInterest(subject, true, nil)
+}
+
+// NumInterest will return the number of subs interested in the subject.
+// In cases where more detail is not required, this may be faster than Match.
+func (s *GenericSublist[T]) NumInterest(subject string) (np int) {
+	s.hasInterest(subject, true, &np)
+	return
+}
+
+// MatchesFullWildcard returns true if there is top-level ">" interest.
+func (s *GenericSublist[T]) MatchesFullWildcard() bool {
+	if s == nil {
+		return false
+	}
+	s.RLock()
+	defer s.RUnlock()
+	return s.root.fwc != nil
+}
+
+// MatchesSingleFilter returns the filter when the sublist contains exactly one unique subject.
+func (s *GenericSublist[T]) MatchesSingleFilter() (string, bool) {
+	if s == nil {
+		return _EMPTY_, false
+	}
+	s.RLock()
+	defer s.RUnlock()
+	return singleFilter(s.root, _EMPTY_)
+}
+
+func singleFilter[T comparable](l *level[T], filter string) (string, bool) {
+	if l == nil {
+		return filter, filter != _EMPTY_
+	}
+	if len(l.nodes) > 1 {
+		return _EMPTY_, false
+	}
+	var next *node[T]
+	branches := 0
+	if l.pwc != nil {
+		next = l.pwc
+		branches++
+	}
+	if l.fwc != nil {
+		next = l.fwc
+		branches++
+	}
+	for _, n := range l.nodes {
+		next = n
+		branches++
+	}
+	if branches != 1 {
+		return _EMPTY_, false
+	}
+	for _, subj := range next.subs {
+		filter = subj
+		break
+	}
+	if next.next == nil {
+		return filter, filter != _EMPTY_
+	}
+	if filter != _EMPTY_ {
+		if next.next.numNodes() > 0 {
+			return _EMPTY_, false
+		}
+		return filter, true
+	}
+	return singleFilter(next.next, filter)
+}
+
+func (s *GenericSublist[T]) match(subject string, cb func(T), doLock bool) {
+	tsa := [32]string{}
+	tokens := tsa[:0]
+	start := 0
+	for i := 0; i < len(subject); i++ {
+		if subject[i] == btsep {
+			if i-start == 0 {
+				return
+			}
+			tokens = append(tokens, subject[start:i])
+			start = i + 1
+		}
+	}
+	if start >= len(subject) {
+		return
+	}
+	tokens = append(tokens, subject[start:])
+
+	if doLock {
+		s.RLock()
+		defer s.RUnlock()
+	}
+	matchLevel(s.root, tokens, cb)
+}
+
+func (s *GenericSublist[T]) hasInterest(subject string, doLock bool, np *int) bool {
+	tsa := [32]string{}
+	tokens := tsa[:0]
+	start := 0
+	for i := 0; i < len(subject); i++ {
+		if subject[i] == btsep {
+			if i-start == 0 {
+				return false
+			}
+			tokens = append(tokens, subject[start:i])
+			start = i + 1
+		}
+	}
+	if start >= len(subject) {
+		return false
+	}
+	tokens = append(tokens, subject[start:])
+
+	if doLock {
+		s.RLock()
+		defer s.RUnlock()
+	}
+	return matchLevelForAny(s.root, tokens, np)
+}
+
+func matchLevelForAny[T comparable](l *level[T], toks []string, np *int) bool {
+	var pwc, n *node[T]
+	for i, t := range toks {
+		if l == nil {
+			return false
+		}
+		if l.fwc != nil {
+			if np != nil {
+				*np += len(l.fwc.subs)
+			}
+			return true
+		}
+		if pwc = l.pwc; pwc != nil {
+			if match := matchLevelForAny(pwc.next, toks[i+1:], np); match {
+				return true
+			}
+		}
+		n = l.nodes[t]
+		if n != nil {
+			l = n.next
+		} else {
+			l = nil
+		}
+	}
+	if n != nil {
+		if np != nil {
+			*np += len(n.subs)
+		}
+		if len(n.subs) > 0 {
+			return true
+		}
+	}
+	if pwc != nil {
+		if np != nil {
+			*np += len(pwc.subs)
+		}
+		return len(pwc.subs) > 0
+	}
+	return false
+}
+
+// callbacksForResults will make the necessary callbacks for each
+// result in this node.
+func callbacksForResults[T comparable](n *node[T], cb func(T)) {
+	for sub := range n.subs {
+		cb(sub)
+	}
+}
+
+// matchLevel is used to recursively descend into the trie.
+func matchLevel[T comparable](l *level[T], toks []string, cb func(T)) {
+	var pwc, n *node[T]
+	for i, t := range toks {
+		if l == nil {
+			return
+		}
+		if l.fwc != nil {
+			callbacksForResults(l.fwc, cb)
+		}
+		if pwc = l.pwc; pwc != nil {
+			matchLevel(pwc.next, toks[i+1:], cb)
+		}
+		n = l.nodes[t]
+		if n != nil {
+			l = n.next
+		} else {
+			l = nil
+		}
+	}
+	if n != nil {
+		callbacksForResults(n, cb)
+	}
+	if pwc != nil {
+		callbacksForResults(pwc, cb)
+	}
+}
+
+// lnt is used to track descent into levels for a removal for pruning.
+type lnt[T comparable] struct {
+	l *level[T]
+	n *node[T]
+	t string
+}
+
+// Raw low level remove, can do batches with lock held outside.
+func (s *GenericSublist[T]) remove(subject string, value T, shouldLock bool) error {
+	if shouldLock {
+		s.Lock()
+		defer s.Unlock()
+	}
+
+	var sfwc bool
+	var n *node[T]
+	l := s.root
+
+	// Track levels for pruning
+	var lnts [32]lnt[T]
+	levels := lnts[:0]
+
+	for t := range strings.SplitSeq(subject, tsep) {
+		lt := len(t)
+		if lt == 0 || sfwc {
+			return ErrInvalidSubject
+		}
+		if l == nil {
+			return ErrNotFound
+		}
+		if lt > 1 {
+			n = l.nodes[t]
+		} else {
+			switch t[0] {
+			case pwc:
+				n = l.pwc
+			case fwc:
+				n = l.fwc
+				sfwc = true
+			default:
+				n = l.nodes[t]
+			}
+		}
+		if n != nil {
+			levels = append(levels, lnt[T]{l, n, t})
+			l = n.next
+		} else {
+			l = nil
+		}
+	}
+
+	if !s.removeFromNode(n, value) {
+		return ErrNotFound
+	}
+
+	s.count--
+
+	for i := len(levels) - 1; i >= 0; i-- {
+		l, n, t := levels[i].l, levels[i].n, levels[i].t
+		if n.isEmpty() {
+			l.pruneNode(n, t)
+		}
+	}
+
+	return nil
+}
+
+// Remove will remove a subscription.
+func (s *GenericSublist[T]) Remove(subject string, value T) error {
+	return s.remove(subject, value, true)
+}
+
+// HasInterestStartingIn is a helper for subject tree intersection.
+func (s *GenericSublist[T]) HasInterestStartingIn(subj string) bool {
+	s.RLock()
+	defer s.RUnlock()
+	var _tokens [64]string
+	tokens := tokenizeSubjectIntoSlice(_tokens[:0], subj)
+	return hasInterestStartingIn(s.root, tokens)
+}
+
+func hasInterestStartingIn[T comparable](l *level[T], tokens []string) bool {
+	if l == nil {
+		return false
+	}
+	if len(tokens) == 0 {
+		return true
+	}
+	token := tokens[0]
+	if l.fwc != nil {
+		return true
+	}
+	found := false
+	if pwc := l.pwc; pwc != nil {
+		found = found || hasInterestStartingIn(pwc.next, tokens[1:])
+	}
+	if n := l.nodes[token]; n != nil {
+		found = found || hasInterestStartingIn(n.next, tokens[1:])
+	}
+	return found
+}
+
+// pruneNode is used to prune an empty node from the tree.
+func (l *level[T]) pruneNode(n *node[T], t string) {
+	if n == nil {
+		return
+	}
+	if n == l.fwc {
+		l.fwc = nil
+	} else if n == l.pwc {
+		l.pwc = nil
+	} else {
+		delete(l.nodes, t)
+	}
+}
+
+// isEmpty will test if the node has any entries. Used
+// in pruning.
+func (n *node[T]) isEmpty() bool {
+	return len(n.subs) == 0 && (n.next == nil || n.next.numNodes() == 0)
+}
+
+// Return the number of nodes for the given level.
+func (l *level[T]) numNodes() int {
+	if l == nil {
+		return 0
+	}
+	num := len(l.nodes)
+	if l.pwc != nil {
+		num++
+	}
+	if l.fwc != nil {
+		num++
+	}
+	return num
+}
+
+// Remove the sub for the given node.
+func (s *GenericSublist[T]) removeFromNode(n *node[T], value T) (found bool) {
+	if n == nil {
+		return false
+	}
+	if _, found = n.subs[value]; found {
+		delete(n.subs, value)
+	}
+	return found
+}
+
+// Count returns the number of subscriptions.
+func (s *GenericSublist[T]) Count() uint32 {
+	s.RLock()
+	defer s.RUnlock()
+	return s.count
+}
+
+// numLevels will return the maximum number of levels
+// contained in the Sublist tree.
+func (s *GenericSublist[T]) numLevels() int {
+	return visitLevel(s.root, 0)
+}
+
+// visitLevel is used to descend the Sublist tree structure
+// recursively.
+func visitLevel[T comparable](l *level[T], depth int) int {
+	if l == nil || l.numNodes() == 0 {
+		return depth
+	}
+
+	depth++
+	maxDepth := depth
+
+	for _, n := range l.nodes {
+		if n == nil {
+			continue
+		}
+		newDepth := visitLevel(n.next, depth)
+		if newDepth > maxDepth {
+			maxDepth = newDepth
+		}
+	}
+	if l.pwc != nil {
+		pwcDepth := visitLevel(l.pwc.next, depth)
+		if pwcDepth > maxDepth {
+			maxDepth = pwcDepth
+		}
+	}
+	if l.fwc != nil {
+		fwcDepth := visitLevel(l.fwc.next, depth)
+		if fwcDepth > maxDepth {
+			maxDepth = fwcDepth
+		}
+	}
+	return maxDepth
+}
+
+// use similar to append. meaning, the updated slice will be returned
+func tokenizeSubjectIntoSlice(tts []string, subject string) []string {
+	start := 0
+	for i := 0; i < len(subject); i++ {
+		if subject[i] == btsep {
+			tts = append(tts, subject[start:i])
+			start = i + 1
+		}
+	}
+	tts = append(tts, subject[start:])
+	return tts
+}

server/gsl/gsl_test.go

@@ -0,0 +1,489 @@
+// Copyright 2016-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package gsl
+
+import (
+	"strings"
+	"testing"
+
+	"github.com/nats-io/nats-server/v2/internal/antithesis"
+)
+
+func TestGenericSublistInit(t *testing.T) {
+	s := NewSublist[struct{}]()
+	require_Equal(t, s.count, 0)
+	require_Equal(t, s.Count(), s.count)
+}
+
+func TestGenericSublistInsertCount(t *testing.T) {
+	s := NewSublist[struct{}]()
+	require_NoError(t, s.Insert("foo", struct{}{}))
+	require_NoError(t, s.Insert("bar", struct{}{}))
+	require_NoError(t, s.Insert("foo.bar", struct{}{}))
+	require_Equal(t, s.Count(), 3)
+}
+
+func TestGenericSublistSimple(t *testing.T) {
+	s := NewSublist[struct{}]()
+	require_NoError(t, s.Insert("foo", struct{}{}))
+	require_Matches(t, s, "foo", 1)
+}
+
+func TestGenericSublistSimpleMultiTokens(t *testing.T) {
+	s := NewSublist[struct{}]()
+	require_NoError(t, s.Insert("foo.bar.baz", struct{}{}))
+	require_Matches(t, s, "foo.bar.baz", 1)
+}
+
+func TestGenericSublistPartialWildcard(t *testing.T) {
+	s := NewSublist[struct{}]()
+	require_NoError(t, s.Insert("a.b.c", struct{}{}))
+	require_NoError(t, s.Insert("a.*.c", struct{}{}))
+	require_Matches(t, s, "a.b.c", 2)
+}
+
+func TestGenericSublistPartialWildcardAtEnd(t *testing.T) {
+	s := NewSublist[struct{}]()
+	require_NoError(t, s.Insert("a.b.c", struct{}{}))
+	require_NoError(t, s.Insert("a.b.*", struct{}{}))
+	require_Matches(t, s, "a.b.c", 2)
+}
+
+func TestGenericSublistFullWildcard(t *testing.T) {
+	s := NewSublist[struct{}]()
+	require_NoError(t, s.Insert("a.b.c", struct{}{}))
+	require_NoError(t, s.Insert("a.>", struct{}{}))
+	require_Matches(t, s, "a.b.c", 2)
+	require_Matches(t, s, "a.>", 1)
+}
+
+func TestGenericSublistRemove(t *testing.T) {
+	s := NewSublist[struct{}]()
+
+	require_NoError(t, s.Insert("a.b.c.d", struct{}{}))
+	require_Equal(t, s.Count(), 1)
+	require_Matches(t, s, "a.b.c.d", 1)
+
+	require_NoError(t, s.Remove("a.b.c.d", struct{}{}))
+	require_Equal(t, s.Count(), 0)
+	require_Matches(t, s, "a.b.c.d", 0)
+}
+
+func TestGenericSublistRemoveWildcard(t *testing.T) {
+	s := NewSublist[int]()
+
+	require_NoError(t, s.Insert("a.b.c.d", 11))
+	require_NoError(t, s.Insert("a.b.*.d", 22))
+	require_NoError(t, s.Insert("a.b.>", 33))
+	require_Equal(t, s.Count(), 3)
+	require_Matches(t, s, "a.b.c.d", 3)
+
+	require_NoError(t, s.Remove("a.b.*.d", 22))
+	require_Equal(t, s.Count(), 2)
+	require_Matches(t, s, "a.b.c.d", 2)
+
+	require_NoError(t, s.Remove("a.b.>", 33))
+	require_Equal(t, s.Count(), 1)
+	require_Matches(t, s, "a.b.c.d", 1)
+
+	require_NoError(t, s.Remove("a.b.c.d", 11))
+	require_Equal(t, s.Count(), 0)
+	require_Matches(t, s, "a.b.c.d", 0)
+}
+
+func TestGenericSublistRemoveCleanup(t *testing.T) {
+	s := NewSublist[struct{}]()
+	require_Equal(t, s.numLevels(), 0)
+	require_NoError(t, s.Insert("a.b.c.d.e.f", struct{}{}))
+	require_Equal(t, s.numLevels(), 6)
+	require_NoError(t, s.Remove("a.b.c.d.e.f", struct{}{}))
+	require_Equal(t, s.numLevels(), 0)
+}
+
+func TestGenericSublistRemoveCleanupWildcards(t *testing.T) {
+	s := NewSublist[struct{}]()
+	require_Equal(t, s.numLevels(), 0)
+	require_NoError(t, s.Insert("a.b.*.d.e.>", struct{}{}))
+	require_Equal(t, s.numLevels(), 6)
+	require_NoError(t, s.Remove("a.b.*.d.e.>", struct{}{}))
+	require_Equal(t, s.numLevels(), 0)
+}
+
+func TestGenericSublistInvalidSubjectsInsert(t *testing.T) {
+	s := NewSublist[struct{}]()
+	// Insert, or subscriptions, can have wildcards, but not empty tokens,
+	// and can not have a FWC that is not the terminal token.
+	require_Error(t, s.Insert(".foo", struct{}{}), ErrInvalidSubject)
+	require_Error(t, s.Insert("foo.", struct{}{}), ErrInvalidSubject)
+	require_Error(t, s.Insert("foo..bar", struct{}{}), ErrInvalidSubject)
+	require_Error(t, s.Insert("foo.bar..baz", struct{}{}), ErrInvalidSubject)
+	require_Error(t, s.Insert("foo.>.baz", struct{}{}), ErrInvalidSubject)
+}
+
+func TestGenericSublistBadSubjectOnRemove(t *testing.T) {
+	s := NewSublist[struct{}]()
+	require_Error(t, s.Insert("a.b..d", struct{}{}), ErrInvalidSubject)
+	require_Error(t, s.Remove("a.b..d", struct{}{}), ErrInvalidSubject)
+	require_Error(t, s.Remove("a.>.b", struct{}{}), ErrInvalidSubject)
+}
+
+func TestGenericSublistTwoTokenPubMatchSingleTokenSub(t *testing.T) {
+	s := NewSublist[struct{}]()
+	require_NoError(t, s.Insert("foo", struct{}{}))
+	require_Matches(t, s, "foo", 1)
+	require_Matches(t, s, "foo.bar", 0)
+}
+
+func TestGenericSublistInsertWithWildcardsAsLiterals(t *testing.T) {
+	s := NewSublist[int]()
+	for i, subject := range []string{"foo.*-", "foo.>-"} {
+		require_NoError(t, s.Insert(subject, i))
+		require_Matches(t, s, "foo.bar", 0)
+		require_Matches(t, s, subject, 1)
+	}
+}
+
+func TestGenericSublistRemoveWithWildcardsAsLiterals(t *testing.T) {
+	s := NewSublist[int]()
+	for i, subject := range []string{"foo.*-", "foo.>-"} {
+		require_NoError(t, s.Insert(subject, i))
+		require_Matches(t, s, "foo.bar", 0)
+		require_Matches(t, s, subject, 1)
+		require_Error(t, s.Remove("foo.bar", i), ErrNotFound)
+		require_Equal(t, s.Count(), 1)
+		require_NoError(t, s.Remove(subject, i))
+		require_Equal(t, s.Count(), 0)
+	}
+}
+
+func TestGenericSublistMatchWithEmptyTokens(t *testing.T) {
+	s := NewSublist[struct{}]()
+	require_NoError(t, s.Insert(">", struct{}{}))
+	for _, subject := range []string{".foo", "..foo", "foo..", "foo.", "foo..bar", "foo...bar"} {
+		t.Run(subject, func(t *testing.T) {
+			require_Matches(t, s, subject, 0)
+		})
+	}
+}
+
+func TestGenericSublistHasInterest(t *testing.T) {
+	s := NewSublist[int]()
+	require_NoError(t, s.Insert("foo", 11))
+
+	// Expect to find that "foo" matches but "bar" doesn't.
+	// At this point nothing should be in the cache.
+	require_True(t, s.HasInterest("foo"))
+	require_False(t, s.HasInterest("bar"))
+
+	// Call Match on a subject we know there is no match.
+	require_Matches(t, s, "bar", 0)
+	require_False(t, s.HasInterest("bar"))
+
+	// Remove fooSub and check interest again
+	require_NoError(t, s.Remove("foo", 11))
+	require_False(t, s.HasInterest("foo"))
+
+	// Try with some wildcards
+	require_NoError(t, s.Insert("foo.*", 22))
+	require_False(t, s.HasInterest("foo"))
+	require_True(t, s.HasInterest("foo.bar"))
+	require_False(t, s.HasInterest("foo.bar.baz"))
+
+	// Remove sub, there should be no interest
+	require_NoError(t, s.Remove("foo.*", 22))
+	require_False(t, s.HasInterest("foo"))
+	require_False(t, s.HasInterest("foo.bar"))
+	require_False(t, s.HasInterest("foo.bar.baz"))
+
+	require_NoError(t, s.Insert("foo.>", 33))
+	require_False(t, s.HasInterest("foo"))
+	require_True(t, s.HasInterest("foo.bar"))
+	require_True(t, s.HasInterest("foo.bar.baz"))
+
+	require_NoError(t, s.Remove("foo.>", 33))
+	require_False(t, s.HasInterest("foo"))
+	require_False(t, s.HasInterest("foo.bar"))
+	require_False(t, s.HasInterest("foo.bar.baz"))
+
+	require_NoError(t, s.Insert("*.>", 44))
+	require_False(t, s.HasInterest("foo"))
+	require_True(t, s.HasInterest("foo.bar"))
+	require_True(t, s.HasInterest("foo.baz"))
+	require_NoError(t, s.Remove("*.>", 44))
+
+	require_NoError(t, s.Insert("*.bar", 55))
+	require_False(t, s.HasInterest("foo"))
+	require_True(t, s.HasInterest("foo.bar"))
+	require_False(t, s.HasInterest("foo.baz"))
+	require_NoError(t, s.Remove("*.bar", 55))
+
+	require_NoError(t, s.Insert("*", 66))
+	require_True(t, s.HasInterest("foo"))
+	require_False(t, s.HasInterest("foo.bar"))
+	require_NoError(t, s.Remove("*", 66))
+}
+
+func TestGenericSublistHasInterestOverlapping(t *testing.T) {
+	s := NewSublist[int]()
+	require_NoError(t, s.Insert("stream.A.child", 11))
+	require_NoError(t, s.Insert("stream.*", 11))
+	require_True(t, s.HasInterest("stream.A.child"))
+	require_True(t, s.HasInterest("stream.A"))
+}
+
+func TestGenericSublistMatchesFullWildcard(t *testing.T) {
+	s := NewSublist[int]()
+	require_NoError(t, s.Insert("stream.A.child", 11))
+	require_NoError(t, s.Insert("stream.*", 11))
+	require_True(t, s.HasInterest("stream.A.child"))
+	require_True(t, s.HasInterest("stream.A"))
+	require_False(t, s.MatchesFullWildcard())
+
+	require_NoError(t, s.Insert(">", 11))
+	require_True(t, s.MatchesFullWildcard())
+}
+
+func TestGenericSublistSingleFilter(t *testing.T) {
+	var sn *GenericSublist[int]
+	filter, ok := sn.MatchesSingleFilter()
+	require_False(t, ok)
+	require_Equal(t, filter, _EMPTY_)
+
+	s := NewSublist[int]()
+	filter, ok = s.MatchesSingleFilter()
+	require_False(t, ok)
+	require_Equal(t, filter, _EMPTY_)
+
+	require_NoError(t, s.Insert("stream.A.child", 11))
+	filter, ok = s.MatchesSingleFilter()
+	require_True(t, ok)
+	require_Equal(t, filter, "stream.A.child")
+
+	require_NoError(t, s.Insert("stream.A.child", 22))
+	filter, ok = s.MatchesSingleFilter()
+	require_True(t, ok)
+	require_Equal(t, filter, "stream.A.child")
+
+	require_NoError(t, s.Insert("stream.A.other", 22))
+	filter, ok = s.MatchesSingleFilter()
+	require_False(t, ok)
+	require_Equal(t, filter, _EMPTY_)
+
+	require_NoError(t, s.Insert("stream.*", 33))
+	filter, ok = s.MatchesSingleFilter()
+	require_False(t, ok)
+	require_Equal(t, filter, _EMPTY_)
+
+	// Test that an ancestor with descendants correctly returns false.
+	s2 := NewSublist[int]()
+	require_NoError(t, s2.Insert("foo", 1))
+	require_NoError(t, s2.Insert("foo.bar", 2))
+	require_NoError(t, s2.Insert("foo.baz", 3))
+	filter, ok = s2.MatchesSingleFilter()
+	require_False(t, ok)
+	require_Equal(t, filter, _EMPTY_)
+
+	// Test that an ancestor with only descendants correctly returns false.
+	s3 := NewSublist[int]()
+	require_NoError(t, s3.Insert("foo.bar", 2))
+	require_NoError(t, s3.Insert("foo.baz", 3))
+	filter, ok = s3.MatchesSingleFilter()
+	require_False(t, ok)
+	require_Equal(t, filter, _EMPTY_)
+}
+
+// TestGenericSublistHasInterestStartingInRace tests that HasInterestStartingIn
+// is safe to call concurrently with modifications to the sublist.
+func TestGenericSublistHasInterestStartingInRace(t *testing.T) {
+	s := NewSublist[int]()
+
+	// Pre-populate with some patterns
+	for i := 0; i < 10; i++ {
+		s.Insert("foo.bar.baz", i)
+		s.Insert("foo.*.baz", i+10)
+		s.Insert("foo.>", i+20)
+	}
+
+	done := make(chan struct{})
+	const iterations = 1000
+
+	// Goroutine 1: repeatedly call HasInterestStartingIn
+	go func() {
+		for i := 0; i < iterations; i++ {
+			s.HasInterestStartingIn("foo")
+			s.HasInterestStartingIn("foo.bar")
+			s.HasInterestStartingIn("foo.bar.baz")
+			s.HasInterestStartingIn("other.subject")
+		}
+		done <- struct{}{}
+	}()
+
+	// Goroutine 2: repeatedly modify the sublist
+	go func() {
+		for i := 0; i < iterations; i++ {
+			val := 1000 + i
+			s.Insert("test.subject."+string(rune('a'+i%26)), val)
+			s.Insert("foo.*.test", val)
+			s.Remove("test.subject."+string(rune('a'+i%26)), val)
+			s.Remove("foo.*.test", val)
+		}
+		done <- struct{}{}
+	}()
+
+	// Goroutine 3: also call HasInterest (which does lock)
+	go func() {
+		for i := 0; i < iterations; i++ {
+			s.HasInterest("foo.bar.baz")
+			s.HasInterest("foo.something.baz")
+		}
+		done <- struct{}{}
+	}()
+
+	// Wait for all goroutines
+	<-done
+	<-done
+	<-done
+}
+
+func TestGenericSublistNumInterest(t *testing.T) {
+	s := NewSublist[int]()
+	require_NoError(t, s.Insert("foo", 11))
+
+	require_NumInterest := func(t *testing.T, subj string, wnp int) {
+		t.Helper()
+		require_Matches(t, s, subj, wnp)
+		require_Equal(t, s.NumInterest(subj), wnp)
+	}
+
+	// Expect to find that "foo" matches but "bar" doesn't.
+	// At this point nothing should be in the cache.
+	require_NumInterest(t, "foo", 1)
+	require_NumInterest(t, "bar", 0)
+
+	// Remove fooSub and check interest again
+	require_NoError(t, s.Remove("foo", 11))
+	require_NumInterest(t, "foo", 0)
+
+	// Try with some wildcards
+	require_NoError(t, s.Insert("foo.*", 22))
+	require_NumInterest(t, "foo", 0)
+	require_NumInterest(t, "foo.bar", 1)
+	require_NumInterest(t, "foo.bar.baz", 0)
+
+	// Remove sub, there should be no interest
+	require_NoError(t, s.Remove("foo.*", 22))
+	require_NumInterest(t, "foo", 0)
+	require_NumInterest(t, "foo.bar", 0)
+	require_NumInterest(t, "foo.bar.baz", 0)
+
+	require_NoError(t, s.Insert("foo.>", 33))
+	require_NumInterest(t, "foo", 0)
+	require_NumInterest(t, "foo.bar", 1)
+	require_NumInterest(t, "foo.bar.baz", 1)
+
+	require_NoError(t, s.Remove("foo.>", 33))
+	require_NumInterest(t, "foo", 0)
+	require_NumInterest(t, "foo.bar", 0)
+	require_NumInterest(t, "foo.bar.baz", 0)
+
+	require_NoError(t, s.Insert("*.>", 44))
+	require_NumInterest(t, "foo", 0)
+	require_NumInterest(t, "foo.bar", 1)
+	require_NumInterest(t, "foo.bar.baz", 1)
+	require_NoError(t, s.Remove("*.>", 44))
+
+	require_NoError(t, s.Insert("*.bar", 55))
+	require_NumInterest(t, "foo", 0)
+	require_NumInterest(t, "foo.bar", 1)
+	require_NumInterest(t, "foo.bar.baz", 0)
+	require_NoError(t, s.Remove("*.bar", 55))
+
+	require_NoError(t, s.Insert("*", 66))
+	require_NumInterest(t, "foo", 1)
+	require_NumInterest(t, "foo.bar", 0)
+	require_NoError(t, s.Remove("*", 66))
+}
+
+// --- TEST HELPERS ---
+
+func require_Matches[T comparable](t *testing.T, s *GenericSublist[T], sub string, c int) {
+	t.Helper()
+	matches := 0
+	s.Match(sub, func(_ T) {
+		matches++
+	})
+	require_Equal(t, matches, c)
+}
+
+func require_True(t testing.TB, b bool) {
+	t.Helper()
+	if !b {
+		antithesis.AssertUnreachable(t, "Failed require_True check", nil)
+		t.Fatalf("require true, but got false")
+	}
+}
+
+func require_False(t testing.TB, b bool) {
+	t.Helper()
+	if b {
+		antithesis.AssertUnreachable(t, "Failed require_False check", nil)
+		t.Fatalf("require false, but got true")
+	}
+}
+
+func require_NoError(t testing.TB, err error) {
+	t.Helper()
+	if err != nil {
+		antithesis.AssertUnreachable(t, "Failed require_NoError check", map[string]any{
+			"error": err.Error(),
+		})
+		t.Fatalf("require no error, but got: %v", err)
+	}
+}
+
+func require_Error(t testing.TB, err error, expected ...error) {
+	t.Helper()
+	if err == nil {
+		antithesis.AssertUnreachable(t, "Failed require_Error check (nil error)", nil)
+		t.Fatalf("require error, but got none")
+	}
+	if len(expected) == 0 {
+		return
+	}
+	// Try to strip nats prefix from Go library if present.
+	const natsErrPre = "nats: "
+	eStr := err.Error()
+	if strings.HasPrefix(eStr, natsErrPre) {
+		eStr = strings.Replace(eStr, natsErrPre, _EMPTY_, 1)
+	}
+
+	for _, e := range expected {
+		if err == e || strings.Contains(eStr, e.Error()) || strings.Contains(e.Error(), eStr) {
+			return
+		}
+	}
+
+	antithesis.AssertUnreachable(t, "Failed require_Error check (unexpected error)", map[string]any{
+		"error": err.Error(),
+	})
+	t.Fatalf("Expected one of %v, got '%v'", expected, err)
+}
+
+func require_Equal[T comparable](t testing.TB, a, b T) {
+	t.Helper()
+	if a != b {
+		antithesis.AssertUnreachable(t, "Failed require_Equal check", nil)
+		t.Fatalf("require %T equal, but got: %v != %v", a, a, b)
+	}
+}

server/ipqueue.go

@@ -0,0 +1,334 @@
+// Copyright 2021-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"errors"
+	"iter"
+	"sync"
+	"sync/atomic"
+)
+
+const ipQueueDefaultMaxRecycleSize = 4 * 1024
+
+// This is a generic intra-process queue.
+type ipQueue[T any] struct {
+	inprogress int64
+	sync.Mutex
+	ch   chan struct{}
+	elts []T
+	pos  int
+	pool *sync.Pool
+	sz   uint64 // Calculated size (only if calc != nil)
+	name string
+	m    *sync.Map
+	ipQueueOpts[T]
+}
+
+type ipQueueOpts[T any] struct {
+	mrs  int              // Max recycle size
+	calc func(e T) uint64 // Calc function for tracking size
+	msz  uint64           // Limit by total calculated size
+	mlen int              // Limit by number of entries
+}
+
+type ipQueueOpt[T any] func(*ipQueueOpts[T])
+
+// This option allows to set the maximum recycle size when attempting
+// to put back a slice to the pool.
+func ipqMaxRecycleSize[T any](max int) ipQueueOpt[T] {
+	return func(o *ipQueueOpts[T]) {
+		o.mrs = max
+	}
+}
+
+// This option enables total queue size counting by passing in a function
+// that evaluates the size of each entry as it is pushed/popped. This option
+// enables the size() function.
+func ipqSizeCalculation[T any](calc func(e T) uint64) ipQueueOpt[T] {
+	return func(o *ipQueueOpts[T]) {
+		o.calc = calc
+	}
+}
+
+// This option allows setting the maximum queue size. Once the limit is
+// reached, then push() will stop returning true and no more entries will
+// be stored until some more are popped. The ipQueue_SizeCalculation must
+// be provided for this to work.
+func ipqLimitBySize[T any](max uint64) ipQueueOpt[T] {
+	return func(o *ipQueueOpts[T]) {
+		o.msz = max
+	}
+}
+
+// This option allows setting the maximum queue length. Once the limit is
+// reached, then push() will stop returning true and no more entries will
+// be stored until some more are popped.
+func ipqLimitByLen[T any](max int) ipQueueOpt[T] {
+	return func(o *ipQueueOpts[T]) {
+		o.mlen = max
+	}
+}
+
+var errIPQLenLimitReached = errors.New("IPQ len limit reached")
+var errIPQSizeLimitReached = errors.New("IPQ size limit reached")
+
+func newIPQueue[T any](s *Server, name string, opts ...ipQueueOpt[T]) *ipQueue[T] {
+	q := &ipQueue[T]{
+		ch: make(chan struct{}, 1),
+		pool: &sync.Pool{
+			New: func() any {
+				// Reason we use pointer to slice instead of slice is explained
+				// here: https://staticcheck.io/docs/checks#SA6002
+				res := make([]T, 0, 32)
+				return &res
+			},
+		},
+		name: name,
+		m:    &s.ipQueues,
+		ipQueueOpts: ipQueueOpts[T]{
+			mrs: ipQueueDefaultMaxRecycleSize,
+		},
+	}
+	for _, o := range opts {
+		o(&q.ipQueueOpts)
+	}
+	s.ipQueues.Store(name, q)
+	return q
+}
+
+// Add the element `e` to the queue, notifying the queue channel's `ch` if the
+// entry is the first to be added, and returns the length of the queue after
+// this element is added.
+func (q *ipQueue[T]) push(e T) (int, error) {
+	q.Lock()
+	l := len(q.elts) - q.pos
+	if q.mlen > 0 && l == q.mlen {
+		q.Unlock()
+		return l, errIPQLenLimitReached
+	}
+	if q.calc != nil {
+		sz := q.calc(e)
+		if q.msz > 0 && q.sz+sz > q.msz {
+			q.Unlock()
+			return l, errIPQSizeLimitReached
+		}
+		q.sz += sz
+	}
+	if q.elts == nil {
+		// What comes out of the pool is already of size 0, so no need for [:0].
+		q.elts = *(q.pool.Get().(*[]T))
+	}
+	q.elts = append(q.elts, e)
+	q.Unlock()
+	if l == 0 {
+		select {
+		case q.ch <- struct{}{}:
+		default:
+		}
+	}
+	return l + 1, nil
+}
+
+// Add all elements yielded by seq to the queue while holding the queue
+// lock, preventing other producers from pushing interleaving elements.
+// On success, it returns the queue length after adding all yielded elements.
+// If a queue limit is reached, no yielded elements are retained, and it
+// returns the unchanged queue length and the first limit error.
+func (q *ipQueue[T]) pushMany(seq iter.Seq[T]) (int, error) {
+	q.Lock()
+	l, added, start := len(q.elts)-q.pos, 0, len(q.elts)
+	initialSize := q.sz
+	revert := func() {
+		clear(q.elts[start:])
+		q.elts = q.elts[:start]
+		q.sz, added = initialSize, 0
+	}
+	defer func() {
+		q.Unlock()
+		if l == 0 && added > 0 {
+			select {
+			case q.ch <- struct{}{}:
+			default:
+			}
+		}
+	}()
+
+	for e := range seq {
+		if q.mlen > 0 && l+added == q.mlen {
+			revert()
+			return l, errIPQLenLimitReached
+		}
+		if q.calc != nil {
+			sz := q.calc(e)
+			if q.msz > 0 && q.sz+sz > q.msz {
+				revert()
+				return l, errIPQSizeLimitReached
+			}
+			q.sz += sz
+		}
+		if q.elts == nil {
+			// What comes out of the pool is already of size 0, so no need for [:0].
+			q.elts = *(q.pool.Get().(*[]T))
+		}
+		q.elts = append(q.elts, e)
+		added++
+	}
+	return l + added, nil
+}
+
+// Returns the whole list of elements currently present in the queue,
+// emptying the queue. This should be called after receiving a notification
+// from the queue's `ch` notification channel that indicates that there
+// is something in the queue.
+// However, in cases where `drain()` may be called from another go
+// routine, it is possible that a routine is notified that there is
+// something, but by the time it calls `pop()`, the drain() would have
+// emptied the queue. So the caller should never assume that pop() will
+// return a slice of 1 or more, it could return `nil`.
+func (q *ipQueue[T]) pop() []T {
+	if q == nil {
+		return nil
+	}
+	q.Lock()
+	if len(q.elts)-q.pos == 0 {
+		q.Unlock()
+		return nil
+	}
+	var elts []T
+	if q.pos == 0 {
+		elts = q.elts
+	} else {
+		elts = q.elts[q.pos:]
+	}
+	q.elts, q.pos, q.sz = nil, 0, 0
+	atomic.AddInt64(&q.inprogress, int64(len(elts)))
+	q.Unlock()
+	return elts
+}
+
+// Returns the first element from the queue, if any. See comment above
+// regarding calling after being notified that there is something and
+// the use of drain(). In short, the caller should always check the
+// boolean return value to ensure that the value is genuine and not a
+// default empty value.
+func (q *ipQueue[T]) popOne() (T, bool) {
+	q.Lock()
+	l := len(q.elts) - q.pos
+	if l == 0 {
+		q.Unlock()
+		var empty T
+		return empty, false
+	}
+	e := q.elts[q.pos]
+	if l--; l > 0 {
+		q.pos++
+		if q.calc != nil {
+			q.sz -= q.calc(e)
+		}
+		// We need to re-signal
+		select {
+		case q.ch <- struct{}{}:
+		default:
+		}
+	} else {
+		// We have just emptied the queue, so we can reuse unless it is too big.
+		if cap(q.elts) <= q.mrs {
+			q.elts = q.elts[:0]
+		} else {
+			q.elts = nil
+		}
+		q.pos, q.sz = 0, 0
+	}
+	q.Unlock()
+	return e, true
+}
+
+// After a pop(), the slice can be recycled for the next push() when
+// a first element is added to the queue.
+// This will also decrement the "in progress" count with the length
+// of the slice.
+// WARNING: The caller MUST never reuse `elts`.
+func (q *ipQueue[T]) recycle(elts *[]T) {
+	// If invoked with a nil list, nothing to do.
+	if elts == nil || *elts == nil {
+		return
+	}
+	// Update the in progress count.
+	if len(*elts) > 0 {
+		atomic.AddInt64(&q.inprogress, int64(-(len(*elts))))
+	}
+	// We also don't want to recycle huge slices, so check against the max.
+	// q.mrs is normally immutable but can be changed, in a safe way, in some tests.
+	if cap(*elts) > q.mrs {
+		return
+	}
+	(*elts) = (*elts)[:0]
+	q.pool.Put(elts)
+}
+
+// Returns the current length of the queue.
+func (q *ipQueue[T]) len() int {
+	q.Lock()
+	defer q.Unlock()
+	return len(q.elts) - q.pos
+}
+
+// Returns the calculated size of the queue (if ipQueue_SizeCalculation has been
+// passed in), otherwise returns zero.
+func (q *ipQueue[T]) size() uint64 {
+	q.Lock()
+	defer q.Unlock()
+	return q.sz
+}
+
+// Empty the queue and consumes the notification signal if present.
+// Returns the number of items that were drained from the queue.
+// Note that this could cause a reader go routine that has been
+// notified that there is something in the queue (reading from queue's `ch`)
+// may then get nothing if `drain()` is invoked before the `pop()` or `popOne()`.
+func (q *ipQueue[T]) drain() int {
+	if q == nil {
+		return 0
+	}
+	q.Lock()
+	olen := len(q.elts) - q.pos
+	q.elts, q.pos, q.sz = nil, 0, 0
+	// Consume the signal if it was present to reduce the chance of a reader
+	// routine to be think that there is something in the queue...
+	select {
+	case <-q.ch:
+	default:
+	}
+	q.Unlock()
+	return olen
+}
+
+// Since the length of the queue goes to 0 after a pop(), it is good to
+// have an insight on how many elements are yet to be processed after a pop().
+// For that reason, the queue maintains a count of elements returned through
+// the pop() API. When the caller will call q.recycle(), this count will
+// be reduced by the size of the slice returned by pop().
+func (q *ipQueue[T]) inProgress() int64 {
+	return atomic.LoadInt64(&q.inprogress)
+}
+
+// Remove this queue from the server's map of ipQueues.
+// All ipQueue operations (such as push/pop/etc..) are still possible.
+func (q *ipQueue[T]) unregister() {
+	if q == nil {
+		return
+	}
+	q.m.Delete(q.name)
+}

server/ipqueue_test.go

@@ -0,0 +1,735 @@
+// Copyright 2021-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+)
+
+func TestIPQueueBasic(t *testing.T) {
+	s := &Server{}
+	q := newIPQueue[int](s, "test")
+	// Check that pool has been created
+	if q.pool == nil {
+		t.Fatal("Expected pool to have been created")
+	}
+	// Check for the default mrs
+	if q.mrs != ipQueueDefaultMaxRecycleSize {
+		t.Fatalf("Expected default max recycle size to be %v, got %v",
+			ipQueueDefaultMaxRecycleSize, q.mrs)
+	}
+	select {
+	case <-q.ch:
+		t.Fatalf("Should not have been notified")
+	default:
+		// OK!
+	}
+	if l := q.len(); l != 0 {
+		t.Fatalf("Expected len to be 0, got %v", l)
+	}
+
+	// Try to change the max recycle size
+	q2 := newIPQueue[int](s, "test2", ipqMaxRecycleSize[int](10))
+	if q2.mrs != 10 {
+		t.Fatalf("Expected max recycle size to be 10, got %v", q2.mrs)
+	}
+
+	// Check that those 2 queues are registered
+	var gotFirst bool
+	var gotSecond bool
+	s.ipQueues.Range(func(k, v any) bool {
+		switch k.(string) {
+		case "test":
+			gotFirst = true
+		case "test2":
+			gotSecond = true
+		default:
+			t.Fatalf("Unknown queue: %q", k.(string))
+		}
+		return true
+	})
+	if !gotFirst {
+		t.Fatalf("Did not find queue %q", "test")
+	}
+	if !gotSecond {
+		t.Fatalf("Did not find queue %q", "test2")
+	}
+	// Unregister them
+	q.unregister()
+	q2.unregister()
+	// They should have been removed from the map
+	s.ipQueues.Range(func(k, v any) bool {
+		t.Fatalf("Got queue %q", k.(string))
+		return false
+	})
+	// But verify that we can still push/pop
+	q.push(1)
+	elts := q.pop()
+	if len(elts) != 1 {
+		t.Fatalf("Should have gotten 1 element, got %v", len(elts))
+	}
+	q2.push(2)
+	if e, ok := q2.popOne(); !ok || e != 2 {
+		t.Fatalf("popOne failed: %+v", e)
+	}
+}
+
+func TestIPQueuePush(t *testing.T) {
+	s := &Server{}
+	q := newIPQueue[int](s, "test")
+	q.push(1)
+	if l := q.len(); l != 1 {
+		t.Fatalf("Expected len to be 1, got %v", l)
+	}
+	select {
+	case <-q.ch:
+		// OK
+	default:
+		t.Fatalf("Should have been notified of addition")
+	}
+	// Push a new element, we should not be notified.
+	q.push(2)
+	if l := q.len(); l != 2 {
+		t.Fatalf("Expected len to be 2, got %v", l)
+	}
+	select {
+	case <-q.ch:
+		t.Fatalf("Should not have been notified of addition")
+	default:
+		// OK
+	}
+}
+
+func TestIPQueuePushMany(t *testing.T) {
+	s := &Server{}
+	q := newIPQueue[int](s, "test")
+
+	l, err := q.pushMany(func(yield func(int) bool) {
+		for i := 1; i <= 3; i++ {
+			if !yield(i) {
+				return
+			}
+		}
+	})
+	require_NoError(t, err)
+	require_Equal(t, l, 3)
+	select {
+	case <-q.ch:
+	default:
+		t.Fatal("Should have been notified of additions")
+	}
+	values := q.pop()
+	require_Len(t, len(values), 3)
+	for i, value := range values {
+		require_Equal(t, value, i+1)
+	}
+	q.recycle(&values)
+
+	l, err = q.pushMany(func(func(int) bool) {})
+	require_NoError(t, err)
+	require_Equal(t, l, 0)
+	select {
+	case <-q.ch:
+		t.Fatal("Should not have been notified of additions")
+	default:
+	}
+
+	limited := newIPQueue[int](s, "limited", ipqLimitByLen[int](2))
+	l, err = limited.pushMany(func(yield func(int) bool) {
+		for i := 1; i <= 3; i++ {
+			if !yield(i) {
+				return
+			}
+		}
+	})
+	require_Error(t, err, errIPQLenLimitReached)
+	require_Equal(t, l, 0)
+	require_Equal(t, limited.len(), 0)
+	select {
+	case <-limited.ch:
+		t.Fatal("Should not have been notified of rejected additions")
+	default:
+	}
+}
+
+func TestIPQueuePop(t *testing.T) {
+	s := &Server{}
+	q := newIPQueue[int](s, "test")
+	q.push(1)
+	<-q.ch
+	elts := q.pop()
+	if l := len(elts); l != 1 {
+		t.Fatalf("Expected 1 elt, got %v", l)
+	}
+	if l := q.len(); l != 0 {
+		t.Fatalf("Expected len to be 0, got %v", l)
+	}
+	// The channel notification should be empty
+	select {
+	case <-q.ch:
+		t.Fatalf("Should not have been notified of addition")
+	default:
+		// OK
+	}
+	// Since pop() brings the number of pending to 0, we keep track of the
+	// number of "in progress" elements. Check that the value is 1 here.
+	if n := q.inProgress(); n != 1 {
+		t.Fatalf("Expected count to be 1, got %v", n)
+	}
+	// Recycling will bring it down to 0.
+	q.recycle(&elts)
+	if n := q.inProgress(); n != 0 {
+		t.Fatalf("Expected count to be 0, got %v", n)
+	}
+	// If we call pop() now, we should get nil.
+	if elts = q.pop(); elts != nil {
+		t.Fatalf("Expected nil, got %v", elts)
+	}
+	// The in progress count should still be 0
+	if n := q.inProgress(); n != 0 {
+		t.Fatalf("Expected count to be 0, got %v", n)
+	}
+	// Calling pop() on an unitialized queue will return nil
+	var q2 *ipQueue[int]
+	elts = q2.pop()
+	require_True(t, elts == nil)
+}
+
+func TestIPQueuePopOne(t *testing.T) {
+	s := &Server{}
+	q := newIPQueue[int](s, "test")
+	q.push(1)
+	<-q.ch
+	e, ok := q.popOne()
+	if !ok {
+		t.Fatal("Got nil")
+	}
+	if i := e; i != 1 {
+		t.Fatalf("Expected 1, got %v", i)
+	}
+	if l := q.len(); l != 0 {
+		t.Fatalf("Expected len to be 0, got %v", l)
+	}
+	// That does not affect the number of notProcessed
+	if n := q.inProgress(); n != 0 {
+		t.Fatalf("Expected count to be 0, got %v", n)
+	}
+	select {
+	case <-q.ch:
+		t.Fatalf("Should not have been notified of addition")
+	default:
+		// OK
+	}
+	q.push(2)
+	q.push(3)
+	e, ok = q.popOne()
+	if !ok {
+		t.Fatal("Got nil")
+	}
+	if i := e; i != 2 {
+		t.Fatalf("Expected 2, got %v", i)
+	}
+	if l := q.len(); l != 1 {
+		t.Fatalf("Expected len to be 1, got %v", l)
+	}
+	select {
+	case <-q.ch:
+		// OK
+	default:
+		t.Fatalf("Should have been notified that there is more")
+	}
+	e, ok = q.popOne()
+	if !ok {
+		t.Fatal("Got nil")
+	}
+	if i := e; i != 3 {
+		t.Fatalf("Expected 3, got %v", i)
+	}
+	if l := q.len(); l != 0 {
+		t.Fatalf("Expected len to be 0, got %v", l)
+	}
+	select {
+	case <-q.ch:
+		t.Fatalf("Should not have been notified that there is more")
+	default:
+		// OK
+	}
+	// Calling it again now that we know there is nothing, we
+	// should get nil.
+	if e, ok = q.popOne(); ok {
+		t.Fatalf("Expected nil, got %v", e)
+	}
+
+	q = newIPQueue[int](s, "test2")
+	q.push(1)
+	q.push(2)
+	// Capture current capacity
+	q.Lock()
+	c := cap(q.elts)
+	q.Unlock()
+	e, ok = q.popOne()
+	if !ok || e != 1 {
+		t.Fatalf("Invalid value: %v", e)
+	}
+	if l := q.len(); l != 1 {
+		t.Fatalf("Expected len to be 1, got %v", l)
+	}
+	values := q.pop()
+	if len(values) != 1 || values[0] != 2 {
+		t.Fatalf("Unexpected values: %v", values)
+	}
+	if cap(values) != c-1 {
+		t.Fatalf("Unexpected capacity: %v vs %v", cap(values), c-1)
+	}
+	if l := q.len(); l != 0 {
+		t.Fatalf("Expected len to be 0, got %v", l)
+	}
+	// Just make sure that this is ok...
+	q.recycle(&values)
+}
+
+func TestIPQueueMultiProducers(t *testing.T) {
+	s := &Server{}
+	q := newIPQueue[int](s, "test")
+
+	wg := sync.WaitGroup{}
+	wg.Add(3)
+	send := func(start, end int) {
+		defer wg.Done()
+
+		for i := start; i <= end; i++ {
+			q.push(i)
+		}
+	}
+	go send(1, 100)
+	go send(101, 200)
+	go send(201, 300)
+
+	tm := time.NewTimer(2 * time.Second)
+	m := make(map[int]struct{})
+	for done := false; !done; {
+		select {
+		case <-q.ch:
+			values := q.pop()
+			for _, v := range values {
+				m[v] = struct{}{}
+			}
+			q.recycle(&values)
+			if n := q.inProgress(); n != 0 {
+				t.Fatalf("Expected count to be 0, got %v", n)
+			}
+			done = len(m) == 300
+		case <-tm.C:
+			t.Fatalf("Did not receive all elements: %v", m)
+		}
+	}
+	wg.Wait()
+}
+
+func TestIPQueueRecycle(t *testing.T) {
+	s := &Server{}
+	q := newIPQueue[int](s, "test")
+	total := 1000
+	for iter := 0; iter < 5; iter++ {
+		var sz int
+		for i := 0; i < total; i++ {
+			sz, _ = q.push(i)
+		}
+		if sz != total {
+			t.Fatalf("Expected size to be %v, got %v", total, sz)
+		}
+		values := q.pop()
+		preRecycleCap := cap(values)
+		q.recycle(&values)
+		sz, _ = q.push(1001)
+		if sz != 1 {
+			t.Fatalf("Expected size to be %v, got %v", 1, sz)
+		}
+		values = q.pop()
+		if l := len(values); l != 1 {
+			t.Fatalf("Len should be 1, got %v", l)
+		}
+		if c := cap(values); c == preRecycleCap {
+			break
+		} else if iter == 4 {
+			// We can't fail the test since there is no guarantee that the slice
+			// is still present in the pool when we do a Get(), but let's log that
+			// recycling did not occur even after all iterations..
+			t.Logf("Seem like the previous slice was not recycled, old cap=%v new cap=%v",
+				preRecycleCap, c)
+		}
+	}
+
+	q = newIPQueue[int](s, "test2", ipqMaxRecycleSize[int](10))
+	for i := 0; i < 100; i++ {
+		q.push(i)
+	}
+	values := q.pop()
+	preRecycleCap := cap(values)
+	q.recycle(&values)
+	q.push(1001)
+	values = q.pop()
+	if l := len(values); l != 1 {
+		t.Fatalf("Len should be 1, got %v", l)
+	}
+	// This time, we should not have recycled it, so the new cap should be whatever
+	// the default is, but less than the preRecycleCap.
+	if c := cap(values); c >= preRecycleCap {
+		t.Fatalf("The slice should not have been put back in the pool, got cap of %v", c)
+	}
+	q.recycle(&values)
+
+	// Check that we don't crash when recycling a nil or empty slice
+	values = q.pop()
+	q.recycle(&values)
+	q.recycle(nil)
+}
+
+func TestIPQueueDrain(t *testing.T) {
+	s := &Server{}
+	q := newIPQueue[int](s, "test")
+	for iter, recycled := 0, false; iter < 5 && !recycled; iter++ {
+		for i := 0; i < 100; i++ {
+			q.push(i + 1)
+		}
+		q.drain()
+		// Try to get something from the pool right away
+		s := q.pool.Get()
+		recycled := s != nil
+		if !recycled {
+			// We can't fail the test, since we have no guarantee it will be recycled
+			// especially when running with `-race` flag...
+			if iter == 4 {
+				t.Log("nothing was recycled")
+			}
+		}
+		// Check that we have consumed the signal...
+		select {
+		case <-q.ch:
+			t.Fatal("Signal should have been consumed by drain")
+		default:
+			// OK!
+		}
+		// Check len
+		if l := q.len(); l != 0 {
+			t.Fatalf("Expected len to be 0, got %v", l)
+		}
+		if recycled {
+			break
+		}
+	}
+}
+
+func TestIPQueueSizeCalculation(t *testing.T) {
+	type testType = [16]byte
+	var testValue testType
+
+	calc := ipqSizeCalculation[testType](func(e testType) uint64 {
+		return uint64(len(e))
+	})
+	s := &Server{}
+	q := newIPQueue[testType](s, "test", calc)
+
+	for i := 0; i < 10; i++ {
+		q.push(testValue)
+		require_Equal(t, q.len(), i+1)
+		require_Equal(t, q.size(), uint64(i+1)*uint64(len(testValue)))
+	}
+
+	for i := 10; i > 5; i-- {
+		q.popOne()
+		require_Equal(t, q.len(), i-1)
+		require_Equal(t, q.size(), uint64(i-1)*uint64(len(testValue)))
+	}
+
+	q.pop()
+	require_Equal(t, q.len(), 0)
+	require_Equal(t, q.size(), 0)
+}
+
+func TestIPQueueSizeCalculationWithLimits(t *testing.T) {
+	type testType = [16]byte
+	var testValue testType
+
+	calc := ipqSizeCalculation[testType](func(e testType) uint64 {
+		return uint64(len(e))
+	})
+	s := &Server{}
+
+	t.Run("LimitByLen", func(t *testing.T) {
+		q := newIPQueue[testType](s, "test", calc, ipqLimitByLen[testType](5))
+		for i := 0; i < 10; i++ {
+			n, err := q.push(testValue)
+			if i >= 5 {
+				require_Error(t, err, errIPQLenLimitReached)
+			} else {
+				require_NoError(t, err)
+			}
+			require_LessThan(t, n, 6)
+		}
+	})
+
+	t.Run("LimitBySize", func(t *testing.T) {
+		q := newIPQueue[testType](s, "test", calc, ipqLimitBySize[testType](16*5))
+		for i := 0; i < 10; i++ {
+			n, err := q.push(testValue)
+			if i >= 5 {
+				require_Error(t, err, errIPQSizeLimitReached)
+			} else {
+				require_NoError(t, err)
+			}
+			require_LessThan(t, n, 6)
+		}
+	})
+}
+
+func Benchmark_IPQueueSizeCalculation(b *testing.B) {
+	type testType = [16]byte
+	var testValue testType
+
+	s := &Server{}
+
+	runPopOne := func(b *testing.B, q *ipQueue[testType]) {
+		b.SetBytes(16)
+		for i := 0; i < b.N; i++ {
+			q.push(testValue)
+		}
+		for i := b.N; i > 0; i-- {
+			q.popOne()
+		}
+	}
+	runPopAll := func(b *testing.B, q *ipQueue[testType]) {
+		b.SetBytes(16)
+		for i := 0; i < b.N; i++ {
+			q.push(testValue)
+		}
+		elts := q.pop()
+		for _, v := range elts {
+			_ = v
+		}
+		q.recycle(&elts)
+	}
+	for _, test := range []struct {
+		name string
+		run  func(b *testing.B, q *ipQueue[testType])
+	}{
+		{"pop one", runPopOne},
+		{"pop all", runPopAll},
+	} {
+		b.Run(test.name, func(b *testing.B) {
+			// Measures without calculation function overheads.
+			b.Run("WithoutCalc", func(b *testing.B) {
+				test.run(b, newIPQueue[testType](s, "test"))
+			})
+
+			// Measures the raw overhead of having a calculation function.
+			b.Run("WithEmptyCalc", func(b *testing.B) {
+				calc := ipqSizeCalculation[testType](func(e testType) uint64 {
+					return 0
+				})
+				test.run(b, newIPQueue[testType](s, "test", calc))
+			})
+
+			// Measures the overhead of having a calculation function that
+			// actually measures something useful.
+			b.Run("WithLenCalc", func(b *testing.B) {
+				calc := ipqSizeCalculation[testType](func(e testType) uint64 {
+					return uint64(len(e))
+				})
+				test.run(b, newIPQueue[testType](s, "test", calc))
+			})
+		})
+	}
+}
+
+func Benchmark_IPQueue1Prod1ConsPopOne(b *testing.B) {
+	benchIPQueue(b, 1, 1, true, false)
+}
+
+func Benchmark_IPQueue5Prod1ConsPopOne(b *testing.B) {
+	benchIPQueue(b, 5, 1, true, false)
+}
+
+func Benchmark_IPQueue1Prod5ConsPopOne(b *testing.B) {
+	benchIPQueue(b, 1, 5, true, false)
+}
+
+func Benchmark_IPQueue5Prod5ConsPopOne(b *testing.B) {
+	benchIPQueue(b, 5, 5, true, false)
+}
+
+func Benchmark_IPQueue1Prod1ConsPopAll(b *testing.B) {
+	benchIPQueue(b, 1, 1, false, false)
+}
+
+func Benchmark_IPQueue5Prod1ConsPopAll(b *testing.B) {
+	benchIPQueue(b, 5, 1, false, false)
+}
+
+func Benchmark_IPQueue1Prod5ConsPopAll(b *testing.B) {
+	benchIPQueue(b, 1, 5, false, false)
+}
+
+func Benchmark_IPQueue5Prod5ConsPopAll(b *testing.B) {
+	benchIPQueue(b, 5, 5, false, false)
+}
+
+func Benchmark_IPQueueWithLim1Prod1ConsPopOne(b *testing.B) {
+	benchIPQueue(b, 1, 1, true, true)
+}
+
+func Benchmark_IPQueueWithLim5Prod1ConsPopOne(b *testing.B) {
+	benchIPQueue(b, 5, 1, true, true)
+}
+
+func Benchmark_IPQueueWithLim1Prod5ConsPopOne(b *testing.B) {
+	benchIPQueue(b, 1, 5, true, true)
+}
+
+func Benchmark_IPQueueWithLim5Prod5ConsPopOne(b *testing.B) {
+	benchIPQueue(b, 5, 5, true, true)
+}
+
+func Benchmark_IPQueueWithLim1Prod1ConsPopAll(b *testing.B) {
+	benchIPQueue(b, 1, 1, false, true)
+}
+
+func Benchmark_IPQueueWithLim5Prod1ConsPopAll(b *testing.B) {
+	benchIPQueue(b, 5, 1, false, true)
+}
+
+func Benchmark_IPQueueWithLim1Prod5ConsPopAll(b *testing.B) {
+	benchIPQueue(b, 1, 5, false, true)
+}
+
+func Benchmark_IPQueueWithLim5Prod5ConsPopAll(b *testing.B) {
+	benchIPQueue(b, 5, 5, false, true)
+}
+
+func benchIPQueue(b *testing.B, numProd, numCons int, popOne, withLimits bool) {
+	var count atomic.Int64
+	var sum atomic.Int64
+
+	perProd := (b.N / numProd) + 1
+	total := int64(perProd * numProd)
+
+	var opts []ipQueueOpt[int]
+	if withLimits {
+		// Add len/size options but make sure they are big enough that we don't fail push() calls.
+		// What we want is to activate code that is checking for limits.
+		opts = append(opts, ipqLimitByLen[int](int(total+100)))
+		opts = append(opts, ipqLimitBySize[int](uint64((total+100)*5)))
+		opts = append(opts, ipqSizeCalculation[int](func(e int) uint64 { return uint64(5) }))
+	}
+	queue := newIPQueue[int](&Server{}, "bench", opts...)
+
+	cdone := make(chan struct{}, numCons)
+	for i := 0; i < numCons; i++ {
+		go func() {
+			for range queue.ch {
+				if popOne {
+					v, ok := queue.popOne()
+					if !ok {
+						continue
+					}
+					sum.Add(int64(v))
+					if count.Add(1) >= total {
+						cdone <- struct{}{}
+						return
+					}
+				} else {
+					elts := queue.pop()
+					for _, v := range elts {
+						sum.Add(int64(v))
+						if count.Add(1) >= total {
+							queue.recycle(&elts)
+							cdone <- struct{}{}
+							return
+						}
+					}
+					queue.recycle(&elts)
+				}
+			}
+		}()
+	}
+
+	pwg := sync.WaitGroup{}
+	pwg.Add(numProd)
+	var expected atomic.Int64
+	for i := 0; i < numProd; i++ {
+		go func() {
+			defer pwg.Done()
+			for i := 0; i < perProd; i++ {
+				expected.Add(int64(i + 1))
+				queue.push(i + 1)
+			}
+		}()
+	}
+
+	pwg.Wait()
+	<-cdone
+	b.StopTimer()
+
+	for i := 1; i < numCons; i++ {
+		queue.push(0)
+		<-cdone
+	}
+	if sum.Load() != expected.Load() {
+		b.Fatalf("Invalid sum, expected %v, got %v", expected.Load(), sum.Load())
+	}
+}
+
+func Benchmark_IPQueuePushAndPopOne(b *testing.B) {
+	s := &Server{}
+	q := newIPQueue[int](s, "test")
+	for i := 0; i < b.N; i++ {
+		q.push(i + 1)
+	}
+	require_Equal[int](b, q.len(), b.N)
+	for i := 0; i < b.N; i++ {
+		v, ok := q.popOne()
+		require_True(b, ok)
+		require_Equal[int](b, v, i+1)
+	}
+	require_Equal[int](b, q.len(), 0)
+	require_Equal(b, q.inProgress(), 0)
+}
+
+func Benchmark_IPQueuePushAndPopAll(b *testing.B) {
+	s := &Server{}
+	q := newIPQueue[int](s, "test")
+	expected := 0
+	for i := 0; i < b.N; i++ {
+		q.push(i + 1)
+		expected += i + 1
+	}
+	require_Equal(b, q.len(), b.N)
+	<-q.ch
+
+	elts := q.pop()
+	require_Equal(b, len(elts), b.N)
+	require_Equal(b, q.len(), 0)
+	require_Equal(b, q.inProgress(), int64(b.N))
+	sum := 0
+	for _, v := range elts {
+		sum += v
+	}
+	q.recycle(&elts)
+	require_Equal(b, expected, sum)
+	require_Equal(b, q.len(), 0)
+	require_Equal(b, q.inProgress(), 0)
+}

server/jetstream_errors.go

@@ -0,0 +1,102 @@
+package server
+
+import (
+	"fmt"
+)
+
+type errOpts struct {
+	err error
+}
+
+// ErrorOption configures a NATS Error helper
+type ErrorOption func(*errOpts)
+
+// Unless ensures that if err is a ApiErr that err will be returned rather than the one being created via the helper
+func Unless(err error) ErrorOption {
+	return func(opts *errOpts) {
+		opts.err = err
+	}
+}
+
+func parseOpts(opts []ErrorOption) *errOpts {
+	eopts := &errOpts{}
+	for _, opt := range opts {
+		opt(eopts)
+	}
+	return eopts
+}
+
+type ErrorIdentifier uint16
+
+// IsNatsErr determines if an error matches ID, if multiple IDs are given if the error matches any of these the function will be true
+func IsNatsErr(err error, ids ...ErrorIdentifier) bool {
+	if err == nil {
+		return false
+	}
+
+	ce, ok := err.(*ApiError)
+	if !ok || ce == nil {
+		return false
+	}
+
+	for _, id := range ids {
+		ae, ok := ApiErrors[id]
+		if !ok || ae == nil {
+			continue
+		}
+
+		if ce.ErrCode == ae.ErrCode {
+			return true
+		}
+	}
+
+	return false
+}
+
+// ApiError is included in all responses if there was an error.
+type ApiError struct {
+	Code        int    `json:"code"`
+	ErrCode     uint16 `json:"err_code,omitempty"`
+	Description string `json:"description,omitempty"`
+}
+
+// ErrorsData is the source data for generated errors as found in errors.json
+type ErrorsData struct {
+	Constant    string `json:"constant"`
+	Code        int    `json:"code"`
+	ErrCode     uint16 `json:"error_code"`
+	Description string `json:"description"`
+	Comment     string `json:"comment"`
+	Help        string `json:"help"`
+	URL         string `json:"url"`
+	Deprecates  string `json:"deprecates"`
+}
+
+func (e *ApiError) Error() string {
+	return fmt.Sprintf("%s (%d)", e.Description, e.ErrCode)
+}
+
+func (e *ApiError) toReplacerArgs(replacements []any) []string {
+	var (
+		ra  []string
+		key string
+	)
+
+	for i, replacement := range replacements {
+		if i%2 == 0 {
+			key = replacement.(string)
+			continue
+		}
+
+		switch v := replacement.(type) {
+		case string:
+			ra = append(ra, key, v)
+		case error:
+			ra = append(ra, key, v.Error())
+		default:
+			ra = append(ra, key, fmt.Sprintf("%v", v))
+		}
+	}
+
+	return ra
+}

server/jetstream_errors_test.go

@@ -0,0 +1,83 @@
+package server
+
+import (
+	"errors"
+	"testing"
+)
+
+func TestIsNatsErr(t *testing.T) {
+	if !IsNatsErr(ApiErrors[JSNotEnabledForAccountErr], JSNotEnabledForAccountErr) {
+		t.Fatalf("Expected error match")
+	}
+
+	if IsNatsErr(ApiErrors[JSNotEnabledForAccountErr], JSClusterNotActiveErr) {
+		t.Fatalf("Expected error mismatch")
+	}
+
+	if IsNatsErr(ApiErrors[JSNotEnabledForAccountErr], JSClusterNotActiveErr, JSClusterNotAvailErr) {
+		t.Fatalf("Expected error mismatch")
+	}
+
+	if !IsNatsErr(ApiErrors[JSNotEnabledForAccountErr], JSClusterNotActiveErr, JSNotEnabledForAccountErr) {
+		t.Fatalf("Expected error match")
+	}
+
+	if !IsNatsErr(&ApiError{ErrCode: 10039}, 1, JSClusterNotActiveErr, JSNotEnabledForAccountErr) {
+		t.Fatalf("Expected error match")
+	}
+
+	if IsNatsErr(&ApiError{ErrCode: 10039}, 1, 2, JSClusterNotActiveErr) {
+		t.Fatalf("Expected error mismatch")
+	}
+
+	if IsNatsErr(nil, JSClusterNotActiveErr) {
+		t.Fatalf("Expected error mismatch")
+	}
+
+	if IsNatsErr(errors.New("x"), JSClusterNotActiveErr) {
+		t.Fatalf("Expected error mismatch")
+	}
+}
+
+func TestApiError_Error(t *testing.T) {
+	if es := ApiErrors[JSClusterNotActiveErr].Error(); es != "JetStream not in clustered mode (10006)" {
+		t.Fatalf("Expected 'JetStream not in clustered mode (10006)', got %q", es)
+	}
+}
+
+func TestApiError_NewWithTags(t *testing.T) {
+	ne := NewJSRestoreSubscribeFailedError(errors.New("failed error"), "the.subject")
+	if ne.Description != "JetStream unable to subscribe to restore snapshot the.subject: failed error" {
+		t.Fatalf("Expected 'JetStream unable to subscribe to restore snapshot the.subject: failed error' got %q", ne.Description)
+	}
+
+	if ne == ApiErrors[JSRestoreSubscribeFailedErrF] {
+		t.Fatalf("Expected a new instance")
+	}
+}
+
+func TestApiError_NewWithUnless(t *testing.T) {
+	if ne := NewJSStreamRestoreError(errors.New("failed error"), Unless(ApiErrors[JSNotEnabledForAccountErr])); !IsNatsErr(ne, JSNotEnabledForAccountErr) {
+		t.Fatalf("Expected JSNotEnabledForAccountErr got %s", ne)
+	}
+
+	if ne := NewJSStreamRestoreError(errors.New("failed error")); !IsNatsErr(ne, JSStreamRestoreErrF) {
+		t.Fatalf("Expected JSStreamRestoreErrF got %s", ne)
+	}
+
+	if ne := NewJSStreamRestoreError(errors.New("failed error"), Unless(errors.New("other error"))); !IsNatsErr(ne, JSStreamRestoreErrF) {
+		t.Fatalf("Expected JSStreamRestoreErrF got %s", ne)
+	}
+
+	if ne := NewJSPeerRemapError(Unless(ApiErrors[JSNotEnabledForAccountErr])); !IsNatsErr(ne, JSNotEnabledForAccountErr) {
+		t.Fatalf("Expected JSNotEnabledForAccountErr got %s", ne)
+	}
+
+	if ne := NewJSPeerRemapError(Unless(nil)); !IsNatsErr(ne, JSPeerRemapErr) {
+		t.Fatalf("Expected JSPeerRemapErr got %s", ne)
+	}
+
+	if ne := NewJSPeerRemapError(Unless(errors.New("other error"))); !IsNatsErr(ne, JSPeerRemapErr) {
+		t.Fatalf("Expected JSPeerRemapErr got %s", ne)
+	}
+}

server/jetstream_events.go

@@ -0,0 +1,383 @@
+// Copyright 2020-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"encoding/json"
+	"time"
+)
+
+// publishAdvisory sends the given advisory into the account. Returns true if
+// it was sent, false if not (i.e. due to lack of interest or a marshal error).
+func (s *Server) publishAdvisory(acc *Account, subject string, adv any) bool {
+	if acc == nil {
+		acc = s.SystemAccount()
+		if acc == nil {
+			return false
+		}
+	}
+
+	// If there is no one listening for this advisory then save ourselves the effort
+	// and don't bother encoding the JSON or sending it.
+	if sl := acc.sl; (sl != nil && !sl.HasInterest(subject)) && !s.hasGatewayInterest(acc.Name, subject) {
+		return false
+	}
+
+	ej, err := json.Marshal(adv)
+	if err == nil {
+		err = s.sendInternalAccountMsg(acc, subject, ej)
+		if err != nil {
+			s.Warnf("Advisory could not be sent for account %q: %v", acc.Name, err)
+		}
+	} else {
+		s.Warnf("Advisory could not be serialized for account %q: %v", acc.Name, err)
+	}
+	return err == nil
+}
+
+// JSAPIAudit is an advisory about administrative actions taken on JetStream
+type JSAPIAudit struct {
+	TypedEvent
+	Server   string      `json:"server"`
+	Client   *ClientInfo `json:"client"`
+	Subject  string      `json:"subject"`
+	Request  string      `json:"request,omitempty"`
+	Response string      `json:"response"`
+	Domain   string      `json:"domain,omitempty"`
+}
+
+const JSAPIAuditType = "io.nats.jetstream.advisory.v1.api_audit"
+
+// ActionAdvisoryType indicates which action against a stream, consumer or template triggered an advisory
+type ActionAdvisoryType string
+
+const (
+	CreateEvent ActionAdvisoryType = "create"
+	DeleteEvent ActionAdvisoryType = "delete"
+	ModifyEvent ActionAdvisoryType = "modify"
+)
+
+// JSStreamActionAdvisory indicates that a stream was created, edited or deleted
+type JSStreamActionAdvisory struct {
+	TypedEvent
+	Stream string             `json:"stream"`
+	Action ActionAdvisoryType `json:"action"`
+	Domain string             `json:"domain,omitempty"`
+}
+
+const JSStreamActionAdvisoryType = "io.nats.jetstream.advisory.v1.stream_action"
+
+// JSConsumerActionAdvisory indicates that a consumer was created or deleted
+type JSConsumerActionAdvisory struct {
+	TypedEvent
+	Stream   string             `json:"stream"`
+	Consumer string             `json:"consumer"`
+	Action   ActionAdvisoryType `json:"action"`
+	Domain   string             `json:"domain,omitempty"`
+}
+
+const JSConsumerActionAdvisoryType = "io.nats.jetstream.advisory.v1.consumer_action"
+
+// JSConsumerPauseAdvisory indicates that a consumer was paused or unpaused
+type JSConsumerPauseAdvisory struct {
+	TypedEvent
+	Stream     string    `json:"stream"`
+	Consumer   string    `json:"consumer"`
+	Paused     bool      `json:"paused"`
+	PauseUntil time.Time `json:"pause_until,omitempty"`
+	Domain     string    `json:"domain,omitempty"`
+}
+
+const JSConsumerPauseAdvisoryType = "io.nats.jetstream.advisory.v1.consumer_pause"
+
+// JSConsumerAckMetric is a metric published when a user acknowledges a message, the
+// number of these that will be published is dependent on SampleFrequency
+type JSConsumerAckMetric struct {
+	TypedEvent
+	Stream      string `json:"stream"`
+	Consumer    string `json:"consumer"`
+	ConsumerSeq uint64 `json:"consumer_seq"`
+	StreamSeq   uint64 `json:"stream_seq"`
+	Delay       int64  `json:"ack_time"`
+	Deliveries  uint64 `json:"deliveries"`
+	Domain      string `json:"domain,omitempty"`
+}
+
+// JSConsumerAckMetricType is the schema type for JSConsumerAckMetricType
+const JSConsumerAckMetricType = "io.nats.jetstream.metric.v1.consumer_ack"
+
+// JSConsumerDeliveryExceededAdvisory is an advisory informing that a message hit
+// its MaxDeliver threshold and so might be a candidate for DLQ handling
+type JSConsumerDeliveryExceededAdvisory struct {
+	TypedEvent
+	Stream     string `json:"stream"`
+	Consumer   string `json:"consumer"`
+	StreamSeq  uint64 `json:"stream_seq"`
+	Deliveries uint64 `json:"deliveries"`
+	Domain     string `json:"domain,omitempty"`
+}
+
+// JSConsumerDeliveryExceededAdvisoryType is the schema type for JSConsumerDeliveryExceededAdvisory
+const JSConsumerDeliveryExceededAdvisoryType = "io.nats.jetstream.advisory.v1.max_deliver"
+
+// JSConsumerDeliveryNakAdvisory is an advisory informing that a message was
+// naked by the consumer
+type JSConsumerDeliveryNakAdvisory struct {
+	TypedEvent
+	Stream      string `json:"stream"`
+	Consumer    string `json:"consumer"`
+	ConsumerSeq uint64 `json:"consumer_seq"`
+	StreamSeq   uint64 `json:"stream_seq"`
+	Deliveries  uint64 `json:"deliveries"`
+	Domain      string `json:"domain,omitempty"`
+}
+
+// JSConsumerDeliveryNakAdvisoryType is the schema type for JSConsumerDeliveryNakAdvisory
+const JSConsumerDeliveryNakAdvisoryType = "io.nats.jetstream.advisory.v1.nak"
+
+// JSConsumerDeliveryTerminatedAdvisory is an advisory informing that a message was
+// terminated by the consumer, so might be a candidate for DLQ handling
+type JSConsumerDeliveryTerminatedAdvisory struct {
+	TypedEvent
+	Stream      string `json:"stream"`
+	Consumer    string `json:"consumer"`
+	ConsumerSeq uint64 `json:"consumer_seq"`
+	StreamSeq   uint64 `json:"stream_seq"`
+	Deliveries  uint64 `json:"deliveries"`
+	Reason      string `json:"reason,omitempty"`
+	Domain      string `json:"domain,omitempty"`
+}
+
+// JSConsumerDeliveryTerminatedAdvisoryType is the schema type for JSConsumerDeliveryTerminatedAdvisory
+const JSConsumerDeliveryTerminatedAdvisoryType = "io.nats.jetstream.advisory.v1.terminated"
+
+// JSSnapshotCreateAdvisory is an advisory sent after a snapshot is successfully started
+type JSSnapshotCreateAdvisory struct {
+	TypedEvent
+	Stream string      `json:"stream"`
+	State  StreamState `json:"state"`
+	Client *ClientInfo `json:"client"`
+	Domain string      `json:"domain,omitempty"`
+}
+
+// JSSnapshotCreatedAdvisoryType is the schema type for JSSnapshotCreateAdvisory
+const JSSnapshotCreatedAdvisoryType = "io.nats.jetstream.advisory.v1.snapshot_create"
+
+// JSSnapshotCompleteAdvisory is an advisory sent after a snapshot is successfully started
+type JSSnapshotCompleteAdvisory struct {
+	TypedEvent
+	Stream string      `json:"stream"`
+	Start  time.Time   `json:"start"`
+	End    time.Time   `json:"end"`
+	Client *ClientInfo `json:"client"`
+	Domain string      `json:"domain,omitempty"`
+	Error  string      `json:"error,omitempty"`
+}
+
+// JSSnapshotCompleteAdvisoryType is the schema type for JSSnapshotCreateAdvisory
+const JSSnapshotCompleteAdvisoryType = "io.nats.jetstream.advisory.v1.snapshot_complete"
+
+// JSRestoreCreateAdvisory is an advisory sent after a snapshot is successfully started
+type JSRestoreCreateAdvisory struct {
+	TypedEvent
+	Stream string      `json:"stream"`
+	Client *ClientInfo `json:"client"`
+	Domain string      `json:"domain,omitempty"`
+}
+
+// JSRestoreCreateAdvisoryType is the schema type for JSSnapshotCreateAdvisory
+const JSRestoreCreateAdvisoryType = "io.nats.jetstream.advisory.v1.restore_create"
+
+// JSRestoreCompleteAdvisory is an advisory sent after a snapshot is successfully started
+type JSRestoreCompleteAdvisory struct {
+	TypedEvent
+	Stream string      `json:"stream"`
+	Start  time.Time   `json:"start"`
+	End    time.Time   `json:"end"`
+	Bytes  int64       `json:"bytes"`
+	Client *ClientInfo `json:"client"`
+	Domain string      `json:"domain,omitempty"`
+}
+
+// JSRestoreCompleteAdvisoryType is the schema type for JSSnapshotCreateAdvisory
+const JSRestoreCompleteAdvisoryType = "io.nats.jetstream.advisory.v1.restore_complete"
+
+// Clustering specific.
+
+// JSClusterLeaderElectedAdvisoryType is sent when the system elects a new meta leader.
+const JSDomainLeaderElectedAdvisoryType = "io.nats.jetstream.advisory.v1.domain_leader_elected"
+
+// JSClusterLeaderElectedAdvisory indicates that a domain has elected a new leader.
+type JSDomainLeaderElectedAdvisory struct {
+	TypedEvent
+	Leader   string      `json:"leader"`
+	Replicas []*PeerInfo `json:"replicas"`
+	Cluster  string      `json:"cluster"`
+	Domain   string      `json:"domain,omitempty"`
+}
+
+// JSStreamLeaderElectedAdvisoryType is sent when the system elects a new leader for a stream.
+const JSStreamLeaderElectedAdvisoryType = "io.nats.jetstream.advisory.v1.stream_leader_elected"
+
+// JSStreamLeaderElectedAdvisory indicates that a stream has elected a new leader.
+type JSStreamLeaderElectedAdvisory struct {
+	TypedEvent
+	Account  string      `json:"account,omitempty"`
+	Stream   string      `json:"stream"`
+	Leader   string      `json:"leader"`
+	Replicas []*PeerInfo `json:"replicas"`
+	Domain   string      `json:"domain,omitempty"`
+}
+
+// JSStreamQuorumLostAdvisoryType is sent when the system detects a clustered stream and
+// its consumers are stalled and unable to make progress.
+const JSStreamQuorumLostAdvisoryType = "io.nats.jetstream.advisory.v1.stream_quorum_lost"
+
+// JSStreamQuorumLostAdvisory indicates that a stream has lost quorum and is stalled.
+type JSStreamQuorumLostAdvisory struct {
+	TypedEvent
+	Account  string      `json:"account,omitempty"`
+	Stream   string      `json:"stream"`
+	Replicas []*PeerInfo `json:"replicas"`
+	Domain   string      `json:"domain,omitempty"`
+}
+
+// JSStreamBatchAbandonedAdvisoryType is sent when a stream's atomic batch is abandoned.
+const JSStreamBatchAbandonedAdvisoryType = "io.nats.jetstream.advisory.v1.stream_batch_abandoned"
+
+// JSStreamBatchAbandonedAdvisory indicates that a stream's batch was abandoned.
+type JSStreamBatchAbandonedAdvisory struct {
+	TypedEvent
+	Account string             `json:"account,omitempty"`
+	Stream  string             `json:"stream"`
+	Domain  string             `json:"domain,omitempty"`
+	BatchId string             `json:"batch"`
+	Reason  BatchAbandonReason `json:"reason"`
+}
+
+type BatchAbandonReason string
+
+var (
+	BatchTimeout            BatchAbandonReason = "timeout"
+	BatchLarge              BatchAbandonReason = "large"
+	BatchIncomplete         BatchAbandonReason = "incomplete"
+	BatchRequirementsNotMet BatchAbandonReason = "unsupported"
+)
+
+// JSConsumerLeaderElectedAdvisoryType is sent when the system elects a leader for a consumer.
+const JSConsumerLeaderElectedAdvisoryType = "io.nats.jetstream.advisory.v1.consumer_leader_elected"
+
+// JSConsumerLeaderElectedAdvisory indicates that a consumer has elected a new leader.
+type JSConsumerLeaderElectedAdvisory struct {
+	TypedEvent
+	Account  string      `json:"account,omitempty"`
+	Stream   string      `json:"stream"`
+	Consumer string      `json:"consumer"`
+	Leader   string      `json:"leader"`
+	Replicas []*PeerInfo `json:"replicas"`
+	Domain   string      `json:"domain,omitempty"`
+}
+
+// JSConsumerQuorumLostAdvisoryType is sent when the system detects a clustered consumer and
+// is stalled and unable to make progress.
+const JSConsumerQuorumLostAdvisoryType = "io.nats.jetstream.advisory.v1.consumer_quorum_lost"
+
+// JSConsumerQuorumLostAdvisory indicates that a consumer has lost quorum and is stalled.
+type JSConsumerQuorumLostAdvisory struct {
+	TypedEvent
+	Account  string      `json:"account,omitempty"`
+	Stream   string      `json:"stream"`
+	Consumer string      `json:"consumer"`
+	Replicas []*PeerInfo `json:"replicas"`
+	Domain   string      `json:"domain,omitempty"`
+}
+
+const JSConsumerGroupPinnedAdvisoryType = "io.nats.jetstream.advisory.v1.consumer_group_pinned"
+
+// JSConsumerGroupPinnedAdvisory that a group switched to a new pinned client
+type JSConsumerGroupPinnedAdvisory struct {
+	TypedEvent
+	Account        string `json:"account,omitempty"`
+	Stream         string `json:"stream"`
+	Consumer       string `json:"consumer"`
+	Domain         string `json:"domain,omitempty"`
+	Group          string `json:"group"`
+	PinnedClientId string `json:"pinned_id"`
+}
+
+const JSConsumerGroupUnpinnedAdvisoryType = "io.nats.jetstream.advisory.v1.consumer_group_unpinned"
+
+// JSConsumerGroupUnpinnedAdvisory indicates that a pin was lost
+type JSConsumerGroupUnpinnedAdvisory struct {
+	TypedEvent
+	Account  string `json:"account,omitempty"`
+	Stream   string `json:"stream"`
+	Consumer string `json:"consumer"`
+	Domain   string `json:"domain,omitempty"`
+	Group    string `json:"group"`
+	// one of "admin" or "timeout", could be an enum up to the implementor to decide
+	Reason string `json:"reason"`
+}
+
+// JSServerOutOfStorageAdvisoryType is sent when the server is out of storage space.
+const JSServerOutOfStorageAdvisoryType = "io.nats.jetstream.advisory.v1.server_out_of_space"
+
+// JSServerOutOfSpaceAdvisory indicates that a stream has lost quorum and is stalled.
+type JSServerOutOfSpaceAdvisory struct {
+	TypedEvent
+	Server   string `json:"server"`
+	ServerID string `json:"server_id"`
+	Stream   string `json:"stream,omitempty"`
+	Cluster  string `json:"cluster"`
+	Domain   string `json:"domain,omitempty"`
+}
+
+// JSServerRemovedAdvisoryType is sent when the server has been removed and JS disabled.
+const JSServerRemovedAdvisoryType = "io.nats.jetstream.advisory.v1.server_removed"
+
+// JSServerRemovedAdvisory indicates that a stream has lost quorum and is stalled.
+type JSServerRemovedAdvisory struct {
+	TypedEvent
+	Server   string `json:"server"`
+	ServerID string `json:"server_id"`
+	Cluster  string `json:"cluster"`
+	Domain   string `json:"domain,omitempty"`
+}
+
+// JSMetaRescueAdvisoryType is sent when a server unsafely lowers the meta
+// group's quorum requirement for disaster recovery.
+const JSMetaRescueAdvisoryType = "io.nats.jetstream.advisory.v1.meta_rescue"
+
+// JSMetaRescueAdvisory indicates that a server has unsafely lowered the meta
+// group's quorum requirement for disaster recovery.
+type JSMetaRescueAdvisory struct {
+	TypedEvent
+	Server     string `json:"server"`
+	ServerID   string `json:"server_id"`
+	PrevQuorum int    `json:"prev_quorum"`
+	NewQuorum  int    `json:"new_quorum"`
+	Cluster    string `json:"cluster"`
+	Domain     string `json:"domain,omitempty"`
+}
+
+// JSAPILimitReachedAdvisoryType is sent when the JS API request queue limit is reached.
+const JSAPILimitReachedAdvisoryType = "io.nats.jetstream.advisory.v1.api_limit_reached"
+
+// JSAPILimitReachedAdvisory is a advisory published when JetStream hits the queue length limit.
+type JSAPILimitReachedAdvisory struct {
+	TypedEvent
+	Server  string `json:"server"`           // Server that created the event, name or ID
+	Domain  string `json:"domain,omitempty"` // Domain the server belongs to
+	Dropped int64  `json:"dropped"`          // How many messages did we drop from the queue
+}

server/jetstream_meta_benchmark_test.go

@@ -0,0 +1,350 @@
+// Copyright 2024-2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build !skip_js_tests && !skip_js_cluster_tests && !skip_js_cluster_tests_2
+
+package server
+
+import (
+	"fmt"
+	"sync"
+	"sync/atomic"
+	"testing"
+
+	"github.com/nats-io/nats.go"
+)
+
+func BenchmarkJetStreamCreate(b *testing.B) {
+
+	const (
+		verbose        = false
+		resourcePrefix = "S"
+		concurrency    = 12
+	)
+
+	// Types of resource that this benchmark creates
+	type ResourceType string
+	const (
+		Stream      ResourceType = "Stream"
+		KVBucket    ResourceType = "KVBucket"
+		ObjectStore ResourceType = "ObjStore"
+	)
+
+	resourceTypeCases := []ResourceType{
+		Stream,
+		KVBucket,
+		ObjectStore,
+	}
+
+	benchmarksCases := []struct {
+		clusterSize int
+		replicas    int
+		storage     nats.StorageType
+	}{
+		{1, 1, nats.MemoryStorage},
+		{3, 3, nats.MemoryStorage},
+		{3, 3, nats.FileStorage},
+	}
+
+	for _, bc := range benchmarksCases {
+		bName := fmt.Sprintf(
+			"N=%d,R=%d,storage=%s,C=%d",
+			bc.clusterSize,
+			bc.replicas,
+			bc.storage.String(),
+			concurrency,
+		)
+
+		b.Run(
+			bName,
+			func(b *testing.B) {
+				for _, rt := range resourceTypeCases {
+					//for _, bc := range benchmarksCases {
+					rName := fmt.Sprintf("resource=%s", rt)
+					b.Run(
+						rName,
+						func(b *testing.B) {
+
+							if verbose {
+								b.Logf(
+									"Creating %d %s resources in cluster with %d nodes, R=%d, %s storage",
+									b.N,
+									string(rt),
+									bc.clusterSize,
+									bc.replicas,
+									bc.storage,
+								)
+							}
+
+							// Setup server or cluster
+							_, leaderServer, shutdown, nc, _ := startJSClusterAndConnect(b, bc.clusterSize)
+							defer shutdown()
+							defer nc.Close()
+
+							// All clients connect to cluster (meta) leader for lower variability
+							connectURL := leaderServer.ClientURL()
+
+							// Wait for all clients and main routine to be ready
+							wgReady := sync.WaitGroup{}
+							wgReady.Add(concurrency + 1)
+							// Wait for all routines to complete
+							wgComplete := sync.WaitGroup{}
+							wgComplete.Add(concurrency)
+
+							// Number of operations (divided amongst clients)
+							opsLeft := atomic.Int64{}
+							opsLeft.Store(int64(b.N))
+							totalErrors := atomic.Int64{}
+
+							// Pre-create connections and JS contexts
+							for i := 1; i <= concurrency; i++ {
+								nc, js := jsClientConnectURL(b, connectURL)
+								defer nc.Close()
+								go func(clientId int, nc *nats.Conn, js nats.JetStreamContext) {
+									defer wgComplete.Done()
+
+									// Config struct (reused and modified in place for each call)
+									streamConfig := nats.StreamConfig{
+										Name:     "?",
+										Storage:  bc.storage,
+										Replicas: bc.replicas,
+									}
+									kvConfig := nats.KeyValueConfig{
+										Bucket:   "?",
+										Storage:  bc.storage,
+										Replicas: bc.replicas,
+									}
+									objConfig := nats.ObjectStoreConfig{
+										Bucket:   "?",
+										Storage:  bc.storage,
+										Replicas: bc.replicas,
+									}
+
+									// Block until everyone is ready
+									wgReady.Done()
+									wgReady.Wait()
+
+									errCount := int64(0)
+									defer func() {
+										// Roll up error count on completion
+										totalErrors.Add(errCount)
+									}()
+
+									// Track per-client opCount (just for logging/debugging)
+									opCount := 0
+									for opsLeft.Add(-1) >= 0 {
+										var err error
+										// Create unique resource name
+										resourceName := fmt.Sprintf("%s_%d_%d", resourcePrefix, clientId, opCount)
+										switch rt {
+										case Stream:
+											streamConfig.Name = resourceName
+											_, err = js.AddStream(&streamConfig)
+										case KVBucket:
+											kvConfig.Bucket = resourceName
+											_, err = js.CreateKeyValue(&kvConfig)
+										case ObjectStore:
+											objConfig.Bucket = resourceName
+											_, err = js.CreateObjectStore(&objConfig)
+										}
+										opCount += 1
+										if err != nil {
+											b.Logf("Error creating %s (%s): %s", rt, resourceName, err)
+											errCount += 1
+										}
+									}
+
+									if verbose {
+										b.Logf("Client %d completed %d operations", clientId, opCount)
+									}
+
+								}(i, nc, js)
+							}
+
+							// Wait for all clients to be ready
+							wgReady.Done()
+							wgReady.Wait()
+
+							// Start benchmark clock
+							b.ResetTimer()
+
+							wgComplete.Wait()
+							b.StopTimer()
+
+							b.ReportMetric(float64(100*(totalErrors.Load()))/float64(b.N), "%error")
+						},
+					)
+				}
+			},
+		)
+	}
+}
+
+func BenchmarkJetStreamCreateConsumers(b *testing.B) {
+
+	const (
+		verbose        = false
+		streamName     = "S"
+		consumerPrefix = "C"
+		concurrency    = 12
+	)
+
+	benchmarksCases := []struct {
+		clusterSize      int
+		consumerReplicas int
+		consumerStorage  nats.StorageType
+	}{
+		{1, 1, nats.MemoryStorage},
+		{3, 3, nats.MemoryStorage},
+		{3, 3, nats.FileStorage},
+	}
+
+	type ConsumerType string
+	const (
+		Ephemeral ConsumerType = "Ephemeral"
+		Durable   ConsumerType = "Durable"
+	)
+
+	consumerTypeCases := []ConsumerType{
+		Ephemeral,
+		Durable,
+	}
+
+	for _, bc := range benchmarksCases {
+
+		bName := fmt.Sprintf(
+			"N=%d,R=%d,storage=%s,C=%d",
+			bc.clusterSize,
+			bc.consumerReplicas,
+			bc.consumerStorage.String(),
+			concurrency,
+		)
+
+		b.Run(
+			bName,
+			func(b *testing.B) {
+
+				for _, ct := range consumerTypeCases {
+
+					cName := fmt.Sprintf("Consumer=%s", ct)
+
+					b.Run(
+						cName,
+						func(b *testing.B) {
+							if verbose {
+								b.Logf(
+									"Creating %d consumers in cluster with %d nodes, R=%d, %s storage",
+									b.N,
+									bc.clusterSize,
+									bc.consumerReplicas,
+									bc.consumerStorage,
+								)
+							}
+
+							// Setup server or cluster
+							_, leaderServer, shutdown, nc, js := startJSClusterAndConnect(b, bc.clusterSize)
+							defer shutdown()
+							defer nc.Close()
+
+							// All clients connect to cluster (meta) leader for lower variability
+							connectURL := leaderServer.ClientURL()
+
+							// Create stream
+							streamConfig := nats.StreamConfig{
+								Name:     streamName,
+								Storage:  nats.FileStorage,
+								Replicas: bc.clusterSize,
+							}
+
+							_, err := js.AddStream(&streamConfig)
+							if err != nil {
+								b.Fatalf("Failed to create stream: %s", err)
+							}
+
+							// Wait for all clients and main routine to be ready
+							wgReady := sync.WaitGroup{}
+							wgReady.Add(concurrency + 1)
+							// Wait for all routines to complete
+							wgComplete := sync.WaitGroup{}
+							wgComplete.Add(concurrency)
+
+							// Number of operations (divided amongst clients)
+							opsLeft := atomic.Int64{}
+							opsLeft.Store(int64(b.N))
+							// Total number of errors
+							totalErrors := atomic.Int64{}
+
+							// Pre-create connections and JS contexts
+							for i := 1; i <= concurrency; i++ {
+								nc, js := jsClientConnectURL(b, connectURL)
+								defer nc.Close()
+
+								go func(clientId int, nc *nats.Conn, js nats.JetStreamContext) {
+									defer wgComplete.Done()
+
+									// Config struct (reused and modified in place for each call)
+									cfg := nats.ConsumerConfig{
+										Durable:       "",
+										Name:          "",
+										Replicas:      bc.consumerReplicas,
+										MemoryStorage: bc.consumerStorage == nats.MemoryStorage,
+									}
+
+									// Block until everyone is ready
+									wgReady.Done()
+									wgReady.Wait()
+
+									errCount := int64(0)
+									opCount := 0
+									for opsLeft.Add(-1) >= 0 {
+										var err error
+										// Set unique consumer name
+										cfg.Name = fmt.Sprintf("%s_%d_%d", consumerPrefix, clientId, opCount)
+										if ct == Durable {
+											cfg.Durable = cfg.Name
+										}
+										_, err = js.AddConsumer(streamName, &cfg)
+										if err != nil {
+											b.Logf("Failed to add consumer: %s", err)
+											errCount += 1
+										}
+										opCount += 1
+									}
+
+									if verbose {
+										b.Logf("Client %d completed %d operations", clientId, opCount)
+									}
+
+									totalErrors.Add(errCount)
+
+								}(i, nc, js)
+							}
+
+							// Wait for all clients to be ready
+							wgReady.Done()
+							wgReady.Wait()
+
+							// Start benchmark clock
+							b.ResetTimer()
+
+							wgComplete.Wait()
+							b.StopTimer()
+
+							b.ReportMetric(float64(100*(totalErrors.Load()))/float64(b.N), "%error")
+						},
+					)
+				}
+			},
+		)
+	}
+}

server/jetstream_sourcing_scaling_test.go

@@ -0,0 +1,283 @@
+// Copyright 2024 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"fmt"
+	"strconv"
+	"testing"
+	"time"
+
+	"github.com/nats-io/nats.go"
+)
+
+var serverConfig1 = `
+	server_name: server1
+	listen: 127.0.0.1:4222
+	http: 8222
+
+	prof_port = 18222
+
+	jetstream: {max_mem_store: 256MB, max_file_store: 2GB, store_dir: '%s'}
+
+	cluster {
+  		name: my_cluster
+  		listen: 127.0.0.1:4248
+  		routes: [nats://127.0.0.1:4249,nats://127.0.0.1:4250]
+	}
+
+	accounts { $SYS { users = [ { user: "admin", pass: "s3cr3t!" } ] }}
+`
+
+var serverConfig2 = `
+	server_name: server2
+	listen: 127.0.0.1:5222
+	http: 8223
+
+	prof_port = 18223
+
+	jetstream: {max_mem_store: 256MB, max_file_store: 2GB, store_dir: '%s'}
+
+	cluster {
+  		name: my_cluster
+  		listen: 127.0.0.1:4249
+  		routes: [nats://127.0.0.1:4248,nats://127.0.0.1:4250]
+	}
+
+	accounts { $SYS { users = [ { user: "admin", pass: "s3cr3t!" } ] }}
+`
+
+var serverConfig3 = `
+	server_name: server3
+	listen: 127.0.0.1:6222
+	http: 8224
+
+	prof_port = 18224
+
+	jetstream: {max_mem_store: 256MB, max_file_store: 2GB, store_dir: '%s'}
+
+	cluster {
+  		name: my_cluster
+  		listen: 127.0.0.1:4250
+  		routes: [nats://127.0.0.1:4248,nats://127.0.0.1:4249]
+	}
+
+	accounts { $SYS { users = [ { user: "admin", pass: "s3cr3t!" } ] }}
+`
+
+var connectURL = "nats://127.0.0.1:4222,nats://127.0.0.1:4223,nats://127.0.0.1:4224"
+
+func createMyLocalCluster(t *testing.T) *cluster {
+	c := &cluster{servers: make([]*Server, 0, 3), opts: make([]*Options, 0, 3), name: "C3"}
+
+	storeDir1 := t.TempDir()
+	s1, o := RunServerWithConfig(createConfFile(t, []byte(fmt.Sprintf(serverConfig1, storeDir1))))
+	c.servers = append(c.servers, s1)
+	c.opts = append(c.opts, o)
+
+	storeDir2 := t.TempDir()
+	s2, o := RunServerWithConfig(createConfFile(t, []byte(fmt.Sprintf(serverConfig2, storeDir2))))
+	c.servers = append(c.servers, s2)
+	c.opts = append(c.opts, o)
+
+	storeDir3 := t.TempDir()
+	s3, o := RunServerWithConfig(createConfFile(t, []byte(fmt.Sprintf(serverConfig3, storeDir3))))
+	c.servers = append(c.servers, s3)
+	c.opts = append(c.opts, o)
+
+	c.t = t
+
+	// Wait til we are formed and have a leader.
+	c.checkClusterFormed()
+	c.waitOnClusterReady()
+
+	return c
+}
+
+// This test is being skipped by CI as it takes too long to run and is meant to test the scalability of sourcing
+// rather than being a unit test.
+func TestStreamSourcingScalingSourcingManyBenchmark(t *testing.T) {
+	t.Skip()
+
+	var numSourced = 1000
+	var numMsgPerSource = 10_000
+	var batchSize = 200
+	var retries int
+
+	var err error
+
+	c := createMyLocalCluster(t)
+	defer c.shutdown()
+
+	nc, err := nats.Connect(connectURL)
+	if err != nil {
+		t.Fatalf("Failed to create client: %v", err)
+	}
+	js, err := nc.JetStream(nats.MaxWait(10 * time.Second))
+	if err != nil {
+		t.Fatalf("Unexpected error getting JetStream context: %v", err)
+	}
+	defer nc.Close()
+
+	// create n streams to source from
+	for i := 0; i < numSourced; i++ {
+		_, err := js.AddStream(&nats.StreamConfig{
+			Name:                 fmt.Sprintf("sourced-%d", i),
+			Subjects:             []string{strconv.Itoa(i)},
+			Retention:            nats.LimitsPolicy,
+			Storage:              nats.FileStorage,
+			Discard:              nats.DiscardOld,
+			Replicas:             1,
+			AllowDirect:          true,
+			MirrorDirect:         false,
+			DiscardNewPerSubject: false,
+		})
+		require_NoError(t, err)
+	}
+
+	fmt.Printf("Streams created\n")
+
+	// publish n messages for each sourced stream
+	for j := 0; j < numMsgPerSource; j++ {
+		start := time.Now()
+		var pafs = make([]nats.PubAckFuture, numSourced)
+		for i := 0; i < numSourced; i++ {
+			var err error
+
+			for {
+				pafs[i], err = js.PublishAsync(strconv.Itoa(i), []byte(strconv.Itoa(j)))
+				if err != nil {
+					fmt.Printf("Error async publishing: %v, retrying\n", err)
+					retries++
+					time.Sleep(10 * time.Millisecond)
+				} else {
+					break
+				}
+			}
+
+			if i != 0 && i%batchSize == 0 {
+				<-js.PublishAsyncComplete()
+			}
+
+		}
+
+		<-js.PublishAsyncComplete()
+
+		for i := 0; i < numSourced; i++ {
+			select {
+			case <-pafs[i].Ok():
+			case psae := <-pafs[i].Err():
+				fmt.Printf("Error on PubAckFuture: %v, retrying sync...\n", psae)
+				retries++
+				_, err = js.Publish(strconv.Itoa(i), []byte(strconv.Itoa(j)))
+				require_NoError(t, err)
+			}
+		}
+
+		end := time.Now()
+		if j%1000 == 0 {
+			fmt.Printf("Published round %d, avg pub latency %v\n", j, end.Sub(start)/time.Duration(numSourced))
+		}
+	}
+
+	fmt.Printf("Messages published\n")
+
+	// create the StreamSources
+	streamSources := make([]*nats.StreamSource, numSourced)
+
+	for i := 0; i < numSourced; i++ {
+		streamSources[i] = &nats.StreamSource{Name: fmt.Sprintf("sourced-%d", i), FilterSubject: strconv.Itoa(i)}
+	}
+
+	// create a stream that sources from them
+	_, err = js.AddStream(&nats.StreamConfig{
+		Name:                 "sourcing",
+		Subjects:             []string{"foo"},
+		Sources:              streamSources,
+		Retention:            nats.LimitsPolicy,
+		Storage:              nats.FileStorage,
+		Discard:              nats.DiscardOld,
+		Replicas:             3,
+		AllowDirect:          true,
+		MirrorDirect:         false,
+		DiscardNewPerSubject: false,
+	})
+	require_NoError(t, err)
+	c.waitOnStreamLeader(globalAccountName, "sourcing")
+	sl := c.streamLeader(globalAccountName, "sourcing")
+	fmt.Printf("Sourcing stream created leader is *** %s ***\n", sl.Name())
+
+	expectedSeq := make([]int, numSourced)
+
+	start := time.Now()
+
+	var lastMsgs uint64
+	mset, err := sl.GlobalAccount().lookupStream("sourcing")
+	require_NoError(t, err)
+
+	checkFor(t, 5*time.Minute, 1000*time.Millisecond, func() error {
+		mset.mu.RLock()
+		var state StreamState
+		mset.store.FastState(&state)
+		mset.mu.RUnlock()
+
+		if state.Msgs == uint64(numMsgPerSource*numSourced) {
+			fmt.Printf("👍 Test passed: expected %d messages, got %d and took %v\n", uint64(numMsgPerSource*numSourced), state.Msgs, time.Since(start))
+			return nil
+		} else if state.Msgs < uint64(numMsgPerSource*numSourced) {
+			fmt.Printf("Current Rate %d per second - Received %d\n", state.Msgs-lastMsgs, state.Msgs)
+			lastMsgs = state.Msgs
+			return fmt.Errorf("Expected %d messages, got %d", uint64(numMsgPerSource*numSourced), state.Msgs)
+		} else {
+			fmt.Printf("Too many messages! expected %d (retries=%d), got %d\n", uint64(numMsgPerSource*numSourced), retries, state.Msgs)
+			return fmt.Errorf("Too many messages: expected %d (retries=%d), got %d", uint64(numMsgPerSource*numSourced), retries, state.Msgs)
+		}
+	})
+
+	// Check that all the messages sourced in the stream are correct
+	// Note: expects to see exactly increasing matching sequence numbers, so could theoretically fail if some messages
+	// get recorded 'out of order' (according to the payload value) because asynchronous JS publication is used to
+	// publish the messages.
+	// However, that should not happen if the publish 'batch size' is not more than the number of streams being sourced.
+
+	// create a consumer on sourcing
+	_, err = js.AddConsumer("sourcing", &nats.ConsumerConfig{AckPolicy: nats.AckExplicitPolicy})
+	require_NoError(t, err)
+	syncSub, err := js.SubscribeSync("", nats.BindStream("sourcing"))
+	require_NoError(t, err)
+
+	start = time.Now()
+
+	print("Checking the messages\n")
+	for i := 0; i < numSourced*numMsgPerSource; i++ {
+		msg, err := syncSub.NextMsg(30 * time.Second)
+		require_NoError(t, err)
+		sId, err := strconv.Atoi(msg.Subject)
+		require_NoError(t, err)
+		seq, err := strconv.Atoi(string(msg.Data))
+		require_NoError(t, err)
+		if expectedSeq[sId] == seq {
+			expectedSeq[sId]++
+		} else {
+			t.Fatalf("Expected seq number %d got %d for source %d\n", expectedSeq[sId], seq, sId)
+		}
+		msg.Ack()
+		if i%100_000 == 0 {
+			now := time.Now()
+			fmt.Printf("[%v] Checked %d messages: %f msgs/sec \n", now, i, 100_000/now.Sub(start).Seconds())
+			start = now
+		}
+	}
+	print("👍 Done. \n")
+}

server/jetstream_tpm_test.go

@@ -0,0 +1,248 @@
+// Copyright 2024 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build windows
+
+package server
+
+import (
+	"fmt"
+	"os"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/nats-io/nats.go"
+)
+
+// The tests in this file are not complete, but are a start to test the TPM
+// with default settings. For a complete test we would need to use a TPM
+// configured with an owner password and a SRK password. This is not
+// typically the default or setup in CI/CD environments.
+
+var jsTPMConfigPassword = `
+	listen: 127.0.0.1:-1
+	jetstream: {
+		store_dir: %q
+		tpm {
+			keys_file: %q
+			encryption_password: "s3cr3t!"
+		}
+	}`
+
+var jsTPMConfigBadPassword = `
+    listen: 127.0.0.1:-1
+	jetstream: {
+		store_dir: %q
+		tpm {
+			keys_file: %q
+			encryption_password: "garbage"
+		}
+	}`
+
+var jsTPMConfigPasswordPcr = `
+	listen: 127.0.0.1:-1
+	jetstream: {
+		store_dir: %q
+		tpm {
+			keys_file: %q
+			encryption_password: "s3cr3t!"
+			pcr: %d
+		}
+	}`
+
+var jsTPMConfigAllFields = `
+	listen: 127.0.0.1:-1
+	jetstream: {
+		store_dir: %q
+		tpm {
+			keys_file: %q
+			encryption_password: "s3cr3t!"
+			pcr: %d
+			srk_password: %q
+			cipher: "aes"
+		}
+	}`
+
+var jsTPMConfigInvalidBothOptionsSet = `
+	listen: 127.0.0.1:-1
+	jetstream: {
+		store_dir: %q
+		encryption_key: "foo"
+		tpm {
+			keys_file: "bar.json"
+			encryption_password: "s3cr3t!"
+		}
+	}`
+
+func getTPMFileName(t *testing.T) string {
+	return t.TempDir() + "/" + t.Name() + "/jskeys.json"
+}
+
+func checkSendMessage(t *testing.T, s *Server) {
+	nc, js := jsClientConnect(t, s)
+	defer nc.Close()
+
+	_, err := js.AddStream(&nats.StreamConfig{
+		Name:     "tpm_test",
+		Subjects: []string{"tpm_test"},
+	})
+	require_NoError(t, err)
+
+	_, err = js.Publish("tpm_test", []byte("hello"))
+	require_NoError(t, err)
+}
+
+func checkReceiveMessage(t *testing.T, s *Server) {
+	// reconnect and get the message
+	nc, js := jsClientConnect(t, s)
+	defer nc.Close()
+
+	// get the message
+	sub, err := js.PullSubscribe("tpm_test", "cls")
+	if err != nil {
+		t.Fatalf("Unexpected error: %v", err)
+	}
+	m := fetchMsgs(t, sub, 1, 5*time.Second)
+	if m == nil {
+		t.Fatalf("Did not receive the message")
+	}
+	if len(m) != 1 {
+		t.Fatalf("Expected 1 message, got %d", len(m))
+	}
+}
+
+func TestJetStreamTPMBasic(t *testing.T) {
+	fileName := getTPMFileName(t)
+	cf := createConfFile(t, []byte(fmt.Sprintf(jsTPMConfigPassword, t.TempDir(), fileName)))
+
+	s, _ := RunServerWithConfig(cf)
+	defer s.Shutdown()
+
+	// Note, actual encryption is tested elsewhere. This just tests that the key is generated.
+	key := s.opts.JetStreamKey
+	if !strings.HasPrefix(key, "SU") {
+		t.Fatalf("expected a TPM key to be generated. Key = %q", key)
+	}
+
+	if _, err := os.Stat(fileName); err != nil {
+		t.Fatalf("keys file was not created")
+	}
+
+	checkSendMessage(t, s)
+
+	// restart the server.
+	s.Shutdown()
+	s, _ = RunServerWithConfig(cf)
+	if s.Running() == false {
+		t.Fatalf("Server should be running")
+	}
+	defer s.Shutdown()
+
+	// double check we're encrypted with the same key.
+	if s.opts.JetStreamKey != key {
+		t.Fatalf("expected same key")
+	}
+	checkReceiveMessage(t, s)
+}
+
+// Warning: Running this test too frequently will considerably slow down
+// these tests and eventually lock out the TPM.
+//
+// Depending on the environment this varies from 10 failures per day with
+// a 24 hour lockout (Dell) to 32 failures per day with a 10
+// minute healing time (decrementing one failure every 10 minutes).
+func TestJetStreamTPMKeyBadPassword(t *testing.T) {
+	fileName := getTPMFileName(t)
+	cf := createConfFile(t, []byte(fmt.Sprintf(jsTPMConfigPassword,
+		t.TempDir(), fileName)))
+
+	s, _ := RunServerWithConfig(cf)
+	defer s.Shutdown()
+
+	checkSendMessage(t, s)
+	s.Shutdown()
+
+	// restart the server.
+	defer func() {
+		r := recover()
+		if r == nil {
+			t.Fatalf("Expected server panic, unexpected start.")
+		}
+	}()
+	cf2 := createConfFile(t, []byte(fmt.Sprintf(jsTPMConfigBadPassword,
+		t.TempDir(), fileName)))
+	s, _ = RunServerWithConfig(cf2)
+	time.Sleep(3 * time.Second)
+	if s.Running() {
+		s.Shutdown()
+		t.Fatalf("Server should NOT be running")
+	}
+}
+
+func TestJetStreamTPMKeyWithPCR(t *testing.T) {
+	cf := createConfFile(t, []byte(fmt.Sprintf(jsTPMConfigPasswordPcr,
+		t.TempDir(), getTPMFileName(t), 18)))
+
+	s, _ := RunServerWithConfig(cf)
+	defer s.Shutdown()
+
+	checkSendMessage(t, s)
+	s.Shutdown()
+
+	// restart the server
+	s, _ = RunServerWithConfig(cf)
+	if !s.Running() {
+		t.Fatalf("Server should be running")
+	}
+}
+
+func TestJetStreamTPMAll(t *testing.T) {
+	fileName := getTPMFileName(t)
+
+	// TODO: When the CI/CD environment supports updating the TPM,
+	// expand this test with the SRK password.
+	cf := createConfFile(t, []byte(fmt.Sprintf(jsTPMConfigAllFields,
+		getTPMFileName(t), fileName, 19, "")))
+
+	s, _ := RunServerWithConfig(cf)
+	defer s.Shutdown()
+
+	checkSendMessage(t, s)
+	s.Shutdown()
+
+	// restart the server.
+	s, _ = RunServerWithConfig(cf)
+	if s.Running() == false {
+		t.Fatalf("Server should be running")
+	}
+	defer s.Shutdown()
+	checkReceiveMessage(t, s)
+}
+
+func TestJetStreamInvalidConfig(t *testing.T) {
+	cf := createConfFile(t, []byte(fmt.Sprintf(jsTPMConfigInvalidBothOptionsSet,
+		getTPMFileName(t))))
+
+	defer func() {
+		r := recover()
+		if r == nil {
+			t.Fatalf("Expected server panic, unexpected start.")
+		}
+	}()
+	s, _ := RunServerWithConfig(cf)
+	if s.Running() == true {
+		s.Shutdown()
+		t.Fatalf("Server should NOT be running")
+	}
+}

server/jetstream_versioning.go

@@ -0,0 +1,246 @@
+// Copyright 2024-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import "strconv"
+
+const (
+	// JSApiLevel is the maximum supported JetStream API level for this server.
+	JSApiLevel int = 5
+
+	JSRequiredLevelMetadataKey = "_nats.req.level"
+	JSServerVersionMetadataKey = "_nats.ver"
+	JSServerLevelMetadataKey   = "_nats.level"
+)
+
+// getRequiredApiLevel returns the required API level for the JetStream asset.
+func getRequiredApiLevel(metadata map[string]string) string {
+	if l, ok := metadata[JSRequiredLevelMetadataKey]; ok && l != _EMPTY_ {
+		return l
+	}
+	return _EMPTY_
+}
+
+// supportsRequiredApiLevel returns whether the required API level for the JetStream asset is supported.
+func supportsRequiredApiLevel(metadata map[string]string) bool {
+	if l := getRequiredApiLevel(metadata); l != _EMPTY_ {
+		li, err := strconv.Atoi(l)
+		return err == nil && li <= JSApiLevel
+	}
+	return true
+}
+
+// setStaticStreamMetadata sets JetStream stream metadata, like the server version and API level.
+// Any dynamic metadata is removed, it must not be stored and only be added for responses.
+func setStaticStreamMetadata(cfg *StreamConfig) {
+	if cfg.Metadata == nil {
+		cfg.Metadata = make(map[string]string)
+	} else {
+		deleteDynamicMetadata(cfg.Metadata)
+	}
+
+	var requiredApiLevel int
+	requires := func(level int) {
+		if level > requiredApiLevel {
+			requiredApiLevel = level
+		}
+	}
+
+	// TTLs were added in v2.11 and require API level 1.
+	if cfg.AllowMsgTTL || cfg.SubjectDeleteMarkerTTL > 0 {
+		requires(1)
+	}
+
+	// Counter CRDTs were added in v2.12 and require API level 2.
+	if cfg.AllowMsgCounter {
+		requires(2)
+	}
+
+	// Atomic batch publishing was added in v2.12 and require API level 2.
+	if cfg.AllowAtomicPublish {
+		requires(2)
+	}
+
+	// Message scheduling was added in v2.12 and require API level 2.
+	if cfg.AllowMsgSchedules {
+		requires(2)
+	}
+
+	// Async persist mode was added in v2.12 and requires API level 2.
+	if cfg.PersistMode == AsyncPersistMode {
+		requires(2)
+	}
+
+	// Fast batch publishing was added in v2.14 and requires API level 4.
+	if cfg.AllowBatchPublish {
+		requires(4)
+	}
+
+	cfg.Metadata[JSRequiredLevelMetadataKey] = strconv.Itoa(requiredApiLevel)
+}
+
+// setDynamicStreamMetadata adds dynamic fields into the (copied) metadata.
+func setDynamicStreamMetadata(cfg *StreamConfig) *StreamConfig {
+	var newCfg StreamConfig
+	if cfg != nil {
+		newCfg = *cfg
+	}
+	newCfg.Metadata = make(map[string]string)
+	if cfg != nil {
+		for key, value := range cfg.Metadata {
+			newCfg.Metadata[key] = value
+		}
+	}
+	newCfg.Metadata[JSServerVersionMetadataKey] = VERSION
+	newCfg.Metadata[JSServerLevelMetadataKey] = strconv.Itoa(JSApiLevel)
+	return &newCfg
+}
+
+// copyConsumerMetadata copies versioning fields from metadata of prevCfg into cfg.
+// Removes versioning fields if no previous metadata, updates if set, and removes fields if it doesn't exist in prevCfg.
+// Any dynamic metadata is removed, it must not be stored and only be added for responses.
+//
+// Note: useful when doing equality checks on cfg and prevCfg, but ignoring any versioning metadata differences.
+func copyStreamMetadata(cfg *StreamConfig, prevCfg *StreamConfig) {
+	if cfg.Metadata != nil {
+		deleteDynamicMetadata(cfg.Metadata)
+	}
+	setOrDeleteInStreamMetadata(cfg, prevCfg, JSRequiredLevelMetadataKey)
+}
+
+// setOrDeleteInConsumerMetadata sets field with key/value in metadata of cfg if set, deletes otherwise.
+func setOrDeleteInStreamMetadata(cfg *StreamConfig, prevCfg *StreamConfig, key string) {
+	if prevCfg != nil && prevCfg.Metadata != nil {
+		if value, ok := prevCfg.Metadata[key]; ok {
+			if cfg.Metadata == nil {
+				cfg.Metadata = make(map[string]string)
+			}
+			cfg.Metadata[key] = value
+			return
+		}
+	}
+	delete(cfg.Metadata, key)
+	if len(cfg.Metadata) == 0 {
+		cfg.Metadata = nil
+	}
+}
+
+// setStaticConsumerMetadata sets JetStream consumer metadata, like the server version and API level.
+// Any dynamic metadata is removed, it must not be stored and only be added for responses.
+func setStaticConsumerMetadata(cfg *ConsumerConfig) {
+	if cfg.Metadata == nil {
+		cfg.Metadata = make(map[string]string)
+	} else {
+		deleteDynamicMetadata(cfg.Metadata)
+	}
+
+	var requiredApiLevel int
+	requires := func(level int) {
+		if level > requiredApiLevel {
+			requiredApiLevel = level
+		}
+	}
+
+	// Added in 2.11, absent | zero is the feature is not used.
+	// one could be stricter and say even if its set but the time
+	// has already passed it is also not needed to restore the consumer
+	if cfg.PauseUntil != nil && !cfg.PauseUntil.IsZero() {
+		requires(1)
+	}
+
+	if cfg.PriorityPolicy != PriorityNone || cfg.PinnedTTL != 0 || len(cfg.PriorityGroups) > 0 {
+		requires(1)
+	}
+
+	// Added in 2.14
+	if cfg.AckPolicy == AckFlowControl {
+		requires(4)
+	}
+
+	cfg.Metadata[JSRequiredLevelMetadataKey] = strconv.Itoa(requiredApiLevel)
+}
+
+// setDynamicConsumerMetadata adds dynamic fields into the (copied) metadata.
+func setDynamicConsumerMetadata(cfg *ConsumerConfig) *ConsumerConfig {
+	var newCfg ConsumerConfig
+	if cfg != nil {
+		newCfg = *cfg
+	}
+	newCfg.Metadata = make(map[string]string)
+	if cfg != nil {
+		for key, value := range cfg.Metadata {
+			newCfg.Metadata[key] = value
+		}
+	}
+	newCfg.Metadata[JSServerVersionMetadataKey] = VERSION
+	newCfg.Metadata[JSServerLevelMetadataKey] = strconv.Itoa(JSApiLevel)
+	return &newCfg
+}
+
+// setDynamicConsumerInfoMetadata adds dynamic fields into the (copied) metadata.
+func setDynamicConsumerInfoMetadata(info *ConsumerInfo) *ConsumerInfo {
+	if info == nil {
+		return nil
+	}
+
+	newInfo := *info
+	cfg := setDynamicConsumerMetadata(info.Config)
+	newInfo.Config = cfg
+	return &newInfo
+}
+
+// copyConsumerMetadata copies versioning fields from metadata of prevCfg into cfg.
+// Removes versioning fields if no previous metadata, updates if set, and removes fields if it doesn't exist in prevCfg.
+// Any dynamic metadata is removed, it must not be stored and only be added for responses.
+//
+// Note: useful when doing equality checks on cfg and prevCfg, but ignoring any versioning metadata differences.
+func copyConsumerMetadata(cfg *ConsumerConfig, prevCfg *ConsumerConfig) {
+	if cfg.Metadata != nil {
+		deleteDynamicMetadata(cfg.Metadata)
+	}
+	setOrDeleteInConsumerMetadata(cfg, prevCfg, JSRequiredLevelMetadataKey)
+}
+
+// setOrDeleteInConsumerMetadata sets field with key/value in metadata of cfg if set, deletes otherwise.
+func setOrDeleteInConsumerMetadata(cfg *ConsumerConfig, prevCfg *ConsumerConfig, key string) {
+	if prevCfg != nil && prevCfg.Metadata != nil {
+		if value, ok := prevCfg.Metadata[key]; ok {
+			if cfg.Metadata == nil {
+				cfg.Metadata = make(map[string]string)
+			}
+			cfg.Metadata[key] = value
+			return
+		}
+	}
+	delete(cfg.Metadata, key)
+	if len(cfg.Metadata) == 0 {
+		cfg.Metadata = nil
+	}
+}
+
+// deleteDynamicMetadata deletes dynamic fields from the metadata.
+func deleteDynamicMetadata(metadata map[string]string) {
+	delete(metadata, JSServerVersionMetadataKey)
+	delete(metadata, JSServerLevelMetadataKey)
+}
+
+// errorOnRequiredApiLevel returns whether a request should be rejected based on the JSRequiredApiLevel header.
+func errorOnRequiredApiLevel(hdr []byte) bool {
+	reqApiLevel := sliceHeader(JSRequiredApiLevel, hdr)
+	if len(reqApiLevel) == 0 {
+		return false
+	}
+	minLevel, err := strconv.Atoi(string(reqApiLevel))
+	return err != nil || JSApiLevel < minLevel
+}

server/jetstream_versioning_test.go

@@ -0,0 +1,734 @@
+// Copyright 2024-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build !skip_js_tests
+
+package server
+
+import (
+	"archive/tar"
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"math"
+	"reflect"
+	"strconv"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/klauspost/compress/s2"
+	"github.com/nats-io/nats.go"
+)
+
+func TestGetAndSupportsRequiredApiLevel(t *testing.T) {
+	require_Equal(t, getRequiredApiLevel(nil), _EMPTY_)
+	require_Equal(t, getRequiredApiLevel(map[string]string{}), _EMPTY_)
+	require_Equal(t, getRequiredApiLevel(map[string]string{JSRequiredLevelMetadataKey: "1"}), "1")
+	require_Equal(t, getRequiredApiLevel(map[string]string{JSRequiredLevelMetadataKey: "text"}), "text")
+
+	require_True(t, supportsRequiredApiLevel(nil))
+	require_True(t, supportsRequiredApiLevel(map[string]string{}))
+	require_True(t, supportsRequiredApiLevel(map[string]string{JSRequiredLevelMetadataKey: "1"}))
+	require_True(t, supportsRequiredApiLevel(map[string]string{JSRequiredLevelMetadataKey: strconv.Itoa(JSApiLevel)}))
+	require_False(t, supportsRequiredApiLevel(map[string]string{JSRequiredLevelMetadataKey: "text"}))
+}
+
+func metadataAtLevel(featureLevel string) map[string]string {
+	return map[string]string{
+		JSRequiredLevelMetadataKey: featureLevel,
+	}
+}
+
+func metadataPrevious() map[string]string {
+	return map[string]string{
+		JSRequiredLevelMetadataKey: "previous-level",
+	}
+}
+
+func TestJetStreamSetStaticStreamMetadata(t *testing.T) {
+	for _, test := range []struct {
+		desc             string
+		cfg              *StreamConfig
+		expectedMetadata map[string]string
+	}{
+		{
+			desc:             "empty",
+			cfg:              &StreamConfig{},
+			expectedMetadata: metadataAtLevel("0"),
+		},
+		{
+			desc:             "overwrite-user-provided",
+			cfg:              &StreamConfig{Metadata: metadataPrevious()},
+			expectedMetadata: metadataAtLevel("0"),
+		},
+		{
+			desc:             "empty-prev-metadata/delete-user-provided",
+			cfg:              &StreamConfig{Metadata: metadataPrevious()},
+			expectedMetadata: metadataAtLevel("0"),
+		},
+		{
+			desc:             "AllowMsgTTL",
+			cfg:              &StreamConfig{AllowMsgTTL: true},
+			expectedMetadata: metadataAtLevel("1"),
+		},
+		{
+			desc:             "SubjectDeleteMarkerTTL",
+			cfg:              &StreamConfig{SubjectDeleteMarkerTTL: time.Second},
+			expectedMetadata: metadataAtLevel("1"),
+		},
+		{
+			desc:             "AllowMsgCounter",
+			cfg:              &StreamConfig{AllowMsgCounter: true},
+			expectedMetadata: metadataAtLevel("2"),
+		},
+		{
+			desc:             "AllowAtomicPublish",
+			cfg:              &StreamConfig{AllowAtomicPublish: true},
+			expectedMetadata: metadataAtLevel("2"),
+		},
+		{
+			desc:             "AllowMsgSchedules",
+			cfg:              &StreamConfig{AllowMsgSchedules: true},
+			expectedMetadata: metadataAtLevel("2"),
+		},
+		{
+			desc:             "AsyncPersistMode",
+			cfg:              &StreamConfig{PersistMode: AsyncPersistMode},
+			expectedMetadata: metadataAtLevel("2"),
+		},
+		{
+			desc:             "AllowBatchPublish",
+			cfg:              &StreamConfig{AllowBatchPublish: true},
+			expectedMetadata: metadataAtLevel("4"),
+		},
+	} {
+		t.Run(test.desc, func(t *testing.T) {
+			setStaticStreamMetadata(test.cfg)
+			level := test.cfg.Metadata[JSRequiredLevelMetadataKey]
+			require_Equal(t, level, test.expectedMetadata[JSRequiredLevelMetadataKey])
+
+			// Ensure we up the server API level if we introduced a feature that requires it.
+			l, err := strconv.Atoi(level)
+			require_NoError(t, err)
+			require_True(t, l <= JSApiLevel)
+		})
+	}
+}
+
+func TestJetStreamSetStaticStreamMetadataRemoveDynamicFields(t *testing.T) {
+	dynamicMetadata := func() map[string]string {
+		return map[string]string{
+			JSServerVersionMetadataKey: "dynamic-version",
+			JSServerLevelMetadataKey:   "dynamic-version",
+		}
+	}
+
+	cfg := StreamConfig{Metadata: dynamicMetadata()}
+	setStaticStreamMetadata(&cfg)
+	require_True(t, reflect.DeepEqual(cfg.Metadata, metadataAtLevel("0")))
+}
+
+func TestJetStreamSetDynamicStreamMetadata(t *testing.T) {
+	cfg := StreamConfig{Metadata: metadataAtLevel("0")}
+	newCfg := setDynamicStreamMetadata(&cfg)
+
+	// Only new metadata must contain dynamic fields.
+	metadata := metadataAtLevel("0")
+	require_True(t, reflect.DeepEqual(cfg.Metadata, metadata))
+	metadata[JSServerVersionMetadataKey] = VERSION
+	metadata[JSServerLevelMetadataKey] = strconv.Itoa(JSApiLevel)
+	require_True(t, reflect.DeepEqual(newCfg.Metadata, metadata))
+}
+
+func TestJetStreamCopyStreamMetadata(t *testing.T) {
+	for _, test := range []struct {
+		desc string
+		cfg  *StreamConfig
+		prev *StreamConfig
+	}{
+		{
+			desc: "no-previous-ignore",
+			cfg:  &StreamConfig{Metadata: metadataAtLevel("-1")},
+			prev: nil,
+		},
+		{
+			desc: "nil-previous-metadata-ignore",
+			cfg:  &StreamConfig{Metadata: metadataAtLevel("-1")},
+			prev: &StreamConfig{Metadata: nil},
+		},
+		{
+			desc: "nil-current-metadata-ignore",
+			cfg:  &StreamConfig{Metadata: nil},
+			prev: &StreamConfig{Metadata: metadataPrevious()},
+		},
+		{
+			desc: "copy-previous",
+			cfg:  &StreamConfig{Metadata: metadataAtLevel("-1")},
+			prev: &StreamConfig{Metadata: metadataPrevious()},
+		},
+		{
+			desc: "delete-missing-fields",
+			cfg:  &StreamConfig{Metadata: metadataAtLevel("-1")},
+			prev: &StreamConfig{Metadata: make(map[string]string)},
+		},
+	} {
+		t.Run(test.desc, func(t *testing.T) {
+			copyStreamMetadata(test.cfg, test.prev)
+
+			var expectedMetadata map[string]string
+			if test.prev != nil {
+				expectedMetadata = test.prev.Metadata
+			}
+
+			value, ok := expectedMetadata[JSRequiredLevelMetadataKey]
+			if ok {
+				require_Equal(t, test.cfg.Metadata[JSRequiredLevelMetadataKey], value)
+			} else {
+				// Key shouldn't exist.
+				_, ok = test.cfg.Metadata[JSRequiredLevelMetadataKey]
+				require_False(t, ok)
+			}
+		})
+	}
+}
+
+func TestJetStreamCopyStreamMetadataRemoveDynamicFields(t *testing.T) {
+	dynamicMetadata := func() map[string]string {
+		return map[string]string{
+			JSServerVersionMetadataKey: "dynamic-version",
+			JSServerLevelMetadataKey:   "dynamic-version",
+		}
+	}
+
+	cfg := StreamConfig{Metadata: dynamicMetadata()}
+	copyStreamMetadata(&cfg, nil)
+	require_Equal(t, len(cfg.Metadata), 0)
+
+	cfg = StreamConfig{Metadata: dynamicMetadata()}
+	prevCfg := StreamConfig{Metadata: metadataAtLevel("0")}
+	copyStreamMetadata(&cfg, &prevCfg)
+	require_True(t, reflect.DeepEqual(cfg.Metadata, metadataAtLevel("0")))
+}
+
+func TestJetStreamSetStaticConsumerMetadata(t *testing.T) {
+	pauseUntil := time.Unix(0, 0)
+	pauseUntilZero := time.Time{}
+	for _, test := range []struct {
+		desc             string
+		cfg              *ConsumerConfig
+		expectedMetadata map[string]string
+	}{
+		{
+			desc:             "empty",
+			cfg:              &ConsumerConfig{},
+			expectedMetadata: metadataAtLevel("0"),
+		},
+		{
+			desc:             "overwrite-user-provided",
+			cfg:              &ConsumerConfig{Metadata: metadataPrevious()},
+			expectedMetadata: metadataAtLevel("0"),
+		},
+		{
+			desc:             "PauseUntil/zero",
+			cfg:              &ConsumerConfig{PauseUntil: &pauseUntilZero},
+			expectedMetadata: metadataAtLevel("0"),
+		},
+		{
+			desc:             "PauseUntil",
+			cfg:              &ConsumerConfig{PauseUntil: &pauseUntil},
+			expectedMetadata: metadataAtLevel("1"),
+		},
+		{
+			desc:             "Pinned",
+			cfg:              &ConsumerConfig{PriorityPolicy: PriorityPinnedClient, PriorityGroups: []string{"a"}},
+			expectedMetadata: metadataAtLevel("1"),
+		},
+		{
+			desc:             "AckFlowControl",
+			cfg:              &ConsumerConfig{AckPolicy: AckFlowControl},
+			expectedMetadata: metadataAtLevel("4"),
+		},
+	} {
+		t.Run(test.desc, func(t *testing.T) {
+			setStaticConsumerMetadata(test.cfg)
+			level := test.cfg.Metadata[JSRequiredLevelMetadataKey]
+			require_Equal(t, level, test.expectedMetadata[JSRequiredLevelMetadataKey])
+
+			// Ensure we up the server API level if we introduced a feature that requires it.
+			l, err := strconv.Atoi(level)
+			require_NoError(t, err)
+			require_True(t, l <= JSApiLevel)
+		})
+	}
+}
+
+func TestJetStreamSetStaticConsumerMetadataRemoveDynamicFields(t *testing.T) {
+	dynamicMetadata := func() map[string]string {
+		return map[string]string{
+			JSServerVersionMetadataKey: "dynamic-version",
+			JSServerLevelMetadataKey:   "dynamic-version",
+		}
+	}
+
+	cfg := ConsumerConfig{Metadata: dynamicMetadata()}
+	setStaticConsumerMetadata(&cfg)
+	require_True(t, reflect.DeepEqual(cfg.Metadata, metadataAtLevel("0")))
+}
+
+func TestJetStreamSetDynamicConsumerMetadata(t *testing.T) {
+	cfg := ConsumerConfig{Metadata: metadataAtLevel("0")}
+	newCfg := setDynamicConsumerMetadata(&cfg)
+
+	// Only new metadata must contain dynamic fields.
+	metadata := metadataAtLevel("0")
+	require_True(t, reflect.DeepEqual(cfg.Metadata, metadata))
+	metadata[JSServerVersionMetadataKey] = VERSION
+	metadata[JSServerLevelMetadataKey] = strconv.Itoa(JSApiLevel)
+	require_True(t, reflect.DeepEqual(newCfg.Metadata, metadata))
+}
+
+func TestJetStreamSetDynamicConsumerInfoMetadata(t *testing.T) {
+	ci := ConsumerInfo{Config: &ConsumerConfig{Metadata: metadataAtLevel("0")}}
+	newCi := setDynamicConsumerInfoMetadata(&ci)
+
+	// Configs should not equal, as that would mean we've overwritten the original ConsumerInfo.
+	require_False(t, reflect.DeepEqual(ci, newCi))
+
+	// Only new metadata must contain dynamic fields.
+	metadata := metadataAtLevel("0")
+	require_True(t, reflect.DeepEqual(ci.Config.Metadata, metadata))
+	metadata[JSServerVersionMetadataKey] = VERSION
+	metadata[JSServerLevelMetadataKey] = strconv.Itoa(JSApiLevel)
+	require_True(t, reflect.DeepEqual(newCi.Config.Metadata, metadata))
+}
+
+func TestJetStreamCopyConsumerMetadata(t *testing.T) {
+	for _, test := range []struct {
+		desc string
+		cfg  *ConsumerConfig
+		prev *ConsumerConfig
+	}{
+		{
+			desc: "no-previous-ignore",
+			cfg:  &ConsumerConfig{Metadata: metadataAtLevel("-1")},
+			prev: nil,
+		},
+		{
+			desc: "nil-previous-metadata-ignore",
+			cfg:  &ConsumerConfig{Metadata: metadataAtLevel("-1")},
+			prev: &ConsumerConfig{Metadata: nil},
+		},
+		{
+			desc: "nil-current-metadata-ignore",
+			cfg:  &ConsumerConfig{Metadata: nil},
+			prev: &ConsumerConfig{Metadata: metadataPrevious()},
+		},
+		{
+			desc: "copy-previous",
+			cfg:  &ConsumerConfig{Metadata: metadataAtLevel("-1")},
+			prev: &ConsumerConfig{Metadata: metadataPrevious()},
+		},
+		{
+			desc: "delete-missing-fields",
+			cfg:  &ConsumerConfig{Metadata: metadataAtLevel("-1")},
+			prev: &ConsumerConfig{Metadata: make(map[string]string)},
+		},
+	} {
+		t.Run(test.desc, func(t *testing.T) {
+			copyConsumerMetadata(test.cfg, test.prev)
+
+			var expectedMetadata map[string]string
+			if test.prev != nil {
+				expectedMetadata = test.prev.Metadata
+			}
+
+			value, ok := expectedMetadata[JSRequiredLevelMetadataKey]
+			if ok {
+				require_Equal(t, test.cfg.Metadata[JSRequiredLevelMetadataKey], value)
+			} else {
+				// Key shouldn't exist.
+				_, ok = test.cfg.Metadata[JSRequiredLevelMetadataKey]
+				require_False(t, ok)
+			}
+		})
+	}
+}
+
+func TestJetStreamCopyConsumerMetadataRemoveDynamicFields(t *testing.T) {
+	dynamicMetadata := func() map[string]string {
+		return map[string]string{
+			JSServerVersionMetadataKey: "dynamic-version",
+			JSServerLevelMetadataKey:   "dynamic-version",
+		}
+	}
+
+	cfg := ConsumerConfig{Metadata: dynamicMetadata()}
+	copyConsumerMetadata(&cfg, nil)
+	require_Equal(t, len(cfg.Metadata), 0)
+
+	cfg = ConsumerConfig{Metadata: dynamicMetadata()}
+	prevCfg := ConsumerConfig{Metadata: metadataAtLevel("0")}
+	copyConsumerMetadata(&cfg, &prevCfg)
+	require_True(t, reflect.DeepEqual(cfg.Metadata, metadataAtLevel("0")))
+}
+
+type server struct {
+	replicas int
+	js       nats.JetStreamContext
+	nc       *nats.Conn
+}
+
+const (
+	streamName   = "STREAM"
+	consumerName = "CONSUMER"
+)
+
+func TestJetStreamMetadataMutations(t *testing.T) {
+	single := RunBasicJetStreamServer(t)
+	defer single.Shutdown()
+	nc, js := jsClientConnect(t, single)
+	defer nc.Close()
+
+	cluster := createJetStreamClusterExplicit(t, "R3S", 3)
+	defer cluster.shutdown()
+	cnc, cjs := jsClientConnect(t, cluster.randomServer())
+	defer cnc.Close()
+
+	// Test for both single server and clustered mode.
+	for _, s := range []server{
+		{1, js, nc},
+		{3, cjs, cnc},
+	} {
+		t.Run(fmt.Sprintf("R%d", s.replicas), func(t *testing.T) {
+			streamMetadataChecks(t, s)
+			consumerMetadataChecks(t, s)
+		})
+	}
+}
+
+func validateMetadata(metadata map[string]string, expectedFeatureLevel string) bool {
+	return metadata[JSRequiredLevelMetadataKey] == expectedFeatureLevel ||
+		metadata[JSServerVersionMetadataKey] == VERSION ||
+		metadata[JSServerLevelMetadataKey] == strconv.Itoa(JSApiLevel)
+}
+
+func streamMetadataChecks(t *testing.T, s server) {
+	// Add stream.
+	sc := nats.StreamConfig{Name: streamName, Replicas: s.replicas}
+	si, err := s.js.AddStream(&sc)
+	require_NoError(t, err)
+	require_True(t, validateMetadata(si.Config.Metadata, "0"))
+
+	// (Double) add stream, has different code path for clustered streams.
+	sc = nats.StreamConfig{Name: streamName, Replicas: s.replicas}
+	si, err = s.js.AddStream(&sc)
+	require_NoError(t, err)
+	require_True(t, validateMetadata(si.Config.Metadata, "0"))
+
+	// Stream info.
+	si, err = s.js.StreamInfo(streamName)
+	require_NoError(t, err)
+	require_True(t, validateMetadata(si.Config.Metadata, "0"))
+
+	// Update stream.
+	// Metadata set on creation should be preserved, even if not included in update.
+	si, err = s.js.UpdateStream(&sc)
+	require_NoError(t, err)
+	require_True(t, validateMetadata(si.Config.Metadata, "0"))
+}
+
+func consumerMetadataChecks(t *testing.T, s server) {
+	// Add consumer.
+	cc := nats.ConsumerConfig{Name: consumerName, Replicas: s.replicas}
+	ci, err := s.js.AddConsumer(streamName, &cc)
+	require_NoError(t, err)
+	require_True(t, validateMetadata(ci.Config.Metadata, "0"))
+
+	// Consumer info.
+	ci, err = s.js.ConsumerInfo(streamName, consumerName)
+	require_NoError(t, err)
+	require_True(t, validateMetadata(ci.Config.Metadata, "0"))
+
+	// Update consumer.
+	// Metadata set on creation should be preserved, even if not included in update.
+	ci, err = s.js.UpdateConsumer(streamName, &cc)
+	require_NoError(t, err)
+	require_True(t, validateMetadata(ci.Config.Metadata, "0"))
+
+	// Use pause advisories to know when pause/resume is applied.
+	pauseCh := make(chan *nats.Msg, 10)
+	_, err = s.nc.ChanSubscribe(JSAdvisoryConsumerPausePre+".STREAM.CONSUMER", pauseCh)
+	require_NoError(t, err)
+
+	// Pause consumer, should up required API level.
+	jsTestPause_PauseConsumer(t, s.nc, streamName, consumerName, time.Now().Add(time.Second*3))
+	require_ChanRead(t, pauseCh, time.Second*2)
+	require_Len(t, len(pauseCh), 0)
+
+	ci, err = s.js.ConsumerInfo(streamName, consumerName)
+	require_NoError(t, err)
+	require_True(t, validateMetadata(ci.Config.Metadata, "1"))
+
+	// Unpause consumer, should lower required API level.
+	subj := fmt.Sprintf("$JS.API.CONSUMER.PAUSE.%s.%s", streamName, consumerName)
+	_, err = s.nc.Request(subj, nil, time.Second)
+	require_NoError(t, err)
+	require_ChanRead(t, pauseCh, time.Second*2)
+	require_Len(t, len(pauseCh), 0)
+
+	ci, err = s.js.ConsumerInfo(streamName, consumerName)
+	require_NoError(t, err)
+	require_True(t, validateMetadata(ci.Config.Metadata, "0"))
+
+	// Test scaling up/down.
+	if s.replicas == 3 {
+		// Scale down.
+		cc.Replicas = 1
+		ci, err = s.js.UpdateConsumer(streamName, &cc)
+		require_NoError(t, err)
+		require_True(t, validateMetadata(ci.Config.Metadata, "0"))
+
+		ci, err = s.js.ConsumerInfo(streamName, consumerName)
+		require_NoError(t, err)
+		require_True(t, validateMetadata(ci.Config.Metadata, "0"))
+
+		// Scale up.
+		cc.Replicas = 3
+		ci, err = s.js.UpdateConsumer(streamName, &cc)
+		require_NoError(t, err)
+		require_True(t, validateMetadata(ci.Config.Metadata, "0"))
+
+		ci, err = s.js.ConsumerInfo(streamName, consumerName)
+		require_NoError(t, err)
+		require_True(t, validateMetadata(ci.Config.Metadata, "0"))
+	}
+}
+
+func TestJetStreamMetadataStreamRestoreAndRestart(t *testing.T) {
+	s := RunBasicJetStreamServer(t)
+	defer s.Shutdown()
+	nc, js := jsClientConnect(t, s)
+	defer nc.Close()
+
+	restoreEmptyStream(t, nc, 1)
+
+	expectedMetadata := map[string]string{
+		JSServerVersionMetadataKey: VERSION,
+		JSServerLevelMetadataKey:   strconv.Itoa(JSApiLevel),
+	}
+
+	// Stream restore should result in empty metadata to be preserved, only adding dynamic metadata.
+	si, err := js.StreamInfo(streamName)
+	require_NoError(t, err)
+	require_True(t, reflect.DeepEqual(si.Config.Metadata, expectedMetadata))
+
+	// Restart server.
+	port := s.opts.Port
+	sd := s.StoreDir()
+	nc.Close()
+	s.Shutdown()
+	s = RunJetStreamServerOnPort(port, sd)
+	defer s.Shutdown()
+	nc, js = jsClientConnect(t, s)
+	defer nc.Close()
+
+	// After restart (or upgrade) metadata data should remain empty, only adding dynamic metadata.
+	si, err = js.StreamInfo(streamName)
+	require_NoError(t, err)
+	require_True(t, reflect.DeepEqual(si.Config.Metadata, expectedMetadata))
+}
+
+func TestJetStreamMetadataStreamRestoreAndRestartCluster(t *testing.T) {
+	c := createJetStreamClusterExplicit(t, "R3S", 3)
+	defer c.shutdown()
+	nc, js := jsClientConnect(t, c.randomServer())
+	defer nc.Close()
+
+	restoreEmptyStream(t, nc, 3)
+
+	expectedMetadata := map[string]string{
+		JSServerVersionMetadataKey: VERSION,
+		JSServerLevelMetadataKey:   strconv.Itoa(JSApiLevel),
+	}
+
+	// Stream restore should result in empty metadata to be preserved, only adding dynamic metadata.
+	si, err := js.StreamInfo(streamName)
+	require_NoError(t, err)
+	require_True(t, reflect.DeepEqual(si.Config.Metadata, expectedMetadata))
+
+	// Restart cluster.
+	c.stopAll()
+	c.restartAllSamePorts()
+	defer c.shutdown()
+	c.waitOnAllCurrent()
+	c.waitOnStreamLeader("$G", streamName)
+	nc, js = jsClientConnect(t, c.randomServer())
+	defer nc.Close()
+
+	// After restart (or upgrade) metadata data should remain empty, only adding dynamic metadata.
+	si, err = js.StreamInfo(streamName)
+	require_NoError(t, err)
+	require_True(t, reflect.DeepEqual(si.Config.Metadata, expectedMetadata))
+}
+
+func restoreEmptyStream(t *testing.T, nc *nats.Conn, replicas int) {
+	rreq := JSApiStreamRestoreRequest{
+		Config: StreamConfig{
+			Name:      "STREAM",
+			Retention: LimitsPolicy,
+			Storage:   FileStorage,
+			Replicas:  replicas,
+		},
+	}
+	buf, err := json.Marshal(rreq)
+	require_NoError(t, err)
+
+	var rresp JSApiStreamRestoreResponse
+	msg, err := nc.Request(fmt.Sprintf(JSApiStreamRestoreT, rreq.Config.Name), buf, 5*time.Second)
+	require_NoError(t, err)
+	json.Unmarshal(msg.Data, &rresp)
+	if rresp.Error != nil {
+		t.Fatalf("Error on restore: %+v", rresp.Error)
+	}
+
+	// Construct empty stream.tar.s2 (only containing meta.inf).
+	fsi := FileStreamInfo{StreamConfig: rreq.Config}
+	fsij, err := json.Marshal(fsi)
+	require_NoError(t, err)
+
+	hdr := &tar.Header{
+		Name:   JetStreamMetaFile,
+		Mode:   0600,
+		Uname:  "nats",
+		Gname:  "nats",
+		Size:   int64(len(fsij)),
+		Format: tar.FormatPAX,
+	}
+	var buffer bytes.Buffer
+	enc := s2.NewWriter(&buffer)
+	tw := tar.NewWriter(enc)
+	err = tw.WriteHeader(hdr)
+	require_NoError(t, err)
+	_, err = tw.Write(fsij)
+	require_NoError(t, err)
+	err = tw.Close()
+	require_NoError(t, err)
+	err = enc.Close()
+	require_NoError(t, err)
+
+	data := buffer.Bytes()
+	msg, err = nc.Request(rresp.DeliverSubject, data, 5*time.Second)
+	require_NoError(t, err)
+	json.Unmarshal(msg.Data, &rresp)
+	if rresp.Error != nil {
+		t.Fatalf("Error on restore: %+v", rresp.Error)
+	}
+
+	msg, err = nc.Request(rresp.DeliverSubject, nil, 5*time.Second)
+	require_NoError(t, err)
+
+	expectedMetadata := map[string]string{
+		JSServerVersionMetadataKey: VERSION,
+		JSServerLevelMetadataKey:   strconv.Itoa(JSApiLevel),
+	}
+
+	var cresp JSApiStreamCreateResponse
+	err = json.Unmarshal(msg.Data, &cresp)
+	require_NoError(t, err)
+	require_True(t, reflect.DeepEqual(cresp.Config.Metadata, expectedMetadata))
+}
+
+func TestJetStreamApiErrorOnRequiredApiLevel(t *testing.T) {
+	s := RunBasicJetStreamServer(t)
+	defer s.Shutdown()
+	nc := clientConnectToServer(t, s)
+	defer nc.Close()
+
+	var subs []*subscription
+	s.getJetStream().apiSubs.All(&subs)
+	require_True(t, len(subs) > 0)
+	for _, sub := range subs {
+		apiSubject := string(sub.subject)
+		t.Run(apiSubject, func(t *testing.T) {
+			req := nats.NewMsg(apiSubject)
+			req.Header.Set("Nats-Required-Api-Level", strconv.Itoa(math.MaxInt))
+			msg, err := nc.RequestMsg(req, time.Second)
+			require_NoError(t, err)
+
+			var resp ApiResponse
+			require_NoError(t, json.Unmarshal(msg.Data, &resp))
+			require_True(t, resp.Error != nil)
+			// Peer remove, stepdown or cancel move is not supported if not clustered.
+			if strings.Contains(apiSubject, ".STEPDOWN.") ||
+				strings.Contains(apiSubject, ".PEER.") ||
+				strings.Contains(apiSubject, ".CANCEL_MOVE.") {
+				require_Error(t, resp.Error, NewJSClusterRequiredError())
+			} else {
+				require_Error(t, resp.Error, NewJSRequiredApiLevelError())
+			}
+		})
+	}
+}
+
+func TestJetStreamApiErrorOnRequiredApiLevelDirectGet(t *testing.T) {
+	s := RunBasicJetStreamServer(t)
+	defer s.Shutdown()
+	nc, js := jsClientConnect(t, s)
+	defer nc.Close()
+
+	_, err := js.AddStream(&nats.StreamConfig{
+		Name:        "TEST",
+		Subjects:    []string{"foo"},
+		AllowDirect: true,
+	})
+	require_NoError(t, err)
+
+	req := nats.NewMsg(fmt.Sprintf(JSDirectMsgGetT, "TEST"))
+	req.Header.Set("Nats-Required-Api-Level", strconv.Itoa(math.MaxInt))
+	msg, err := nc.RequestMsg(req, time.Second)
+	require_NoError(t, err)
+	require_Equal(t, msg.Header.Get("Status"), "412")
+	require_Equal(t, msg.Header.Get("Description"), "Required Api Level")
+
+	req = nats.NewMsg(fmt.Sprintf(JSDirectGetLastBySubjectT, "TEST", "foo"))
+	req.Header.Set("Nats-Required-Api-Level", strconv.Itoa(math.MaxInt))
+	msg, err = nc.RequestMsg(req, time.Second)
+	require_NoError(t, err)
+	require_Equal(t, msg.Header.Get("Status"), "412")
+	require_Equal(t, msg.Header.Get("Description"), "Required Api Level")
+}
+
+func TestJetStreamApiErrorOnRequiredApiLevelPullConsumerNextMsg(t *testing.T) {
+	s := RunBasicJetStreamServer(t)
+	defer s.Shutdown()
+	nc, js := jsClientConnect(t, s)
+	defer nc.Close()
+
+	_, err := js.AddStream(&nats.StreamConfig{
+		Name:        "TEST",
+		Subjects:    []string{"foo"},
+		AllowDirect: true,
+	})
+	require_NoError(t, err)
+
+	_, err = js.AddConsumer("TEST", &nats.ConsumerConfig{Durable: "CONSUMER"})
+	require_NoError(t, err)
+
+	req := nats.NewMsg(fmt.Sprintf(JSApiRequestNextT, "TEST", "CONSUMER"))
+	req.Header.Set("Nats-Required-Api-Level", strconv.Itoa(math.MaxInt))
+	msg, err := nc.RequestMsg(req, time.Second)
+	require_NoError(t, err)
+	require_Equal(t, msg.Header.Get("Status"), "412")
+	require_Equal(t, msg.Header.Get("Description"), "Required Api Level")
+}

server/jwt.go

@@ -0,0 +1,275 @@
+// Copyright 2018-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"errors"
+	"fmt"
+	"net"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/nats-io/jwt/v2"
+	"github.com/nats-io/nkeys"
+)
+
+// All JWTs once encoded start with this
+const jwtPrefix = "eyJ"
+
+// ReadOperatorJWT will read a jwt file for an operator claim. This can be a decorated file.
+func ReadOperatorJWT(jwtfile string) (*jwt.OperatorClaims, error) {
+	_, claim, err := readOperatorJWT(jwtfile)
+	return claim, err
+}
+
+func readOperatorJWT(jwtfile string) (string, *jwt.OperatorClaims, error) {
+	contents, err := os.ReadFile(jwtfile)
+	if err != nil {
+		// Check to see if the JWT has been inlined.
+		if !strings.HasPrefix(jwtfile, jwtPrefix) {
+			return "", nil, err
+		}
+		// We may have an inline jwt here.
+		contents = []byte(jwtfile)
+	}
+	defer wipeSlice(contents)
+
+	theJWT, err := jwt.ParseDecoratedJWT(contents)
+	if err != nil {
+		return "", nil, err
+	}
+	opc, err := jwt.DecodeOperatorClaims(theJWT)
+	if err != nil {
+		return "", nil, err
+	}
+	return theJWT, opc, nil
+}
+
+// Just wipe slice with 'x', for clearing contents of nkey seed file.
+func wipeSlice(buf []byte) {
+	for i := range buf {
+		buf[i] = 'x'
+	}
+}
+
+// validateTrustedOperators will check that we do not have conflicts with
+// assigned trusted keys and trusted operators. If operators are defined we
+// will expand the trusted keys in options.
+func validateTrustedOperators(o *Options) error {
+	if len(o.TrustedOperators) == 0 {
+		// if we have no operator, default sentinel shouldn't be set
+		if o.DefaultSentinel != _EMPTY_ {
+			return fmt.Errorf("default sentinel requires operators and accounts")
+		}
+		return nil
+	}
+	if o.DefaultSentinel != _EMPTY_ {
+		juc, err := jwt.DecodeUserClaims(o.DefaultSentinel)
+		if err != nil {
+			return fmt.Errorf("default sentinel JWT not valid")
+		}
+
+		if !juc.BearerToken && juc.IssuerAccount != "" && juc.HasEmptyPermissions() {
+			// we cannot resolve the account yet - but this looks like a scoped user
+			// it will be rejected at runtime if not valid
+		} else if !juc.BearerToken {
+			return fmt.Errorf("default sentinel must be a bearer token")
+		}
+	}
+	if o.AccountResolver == nil {
+		return fmt.Errorf("operators require an account resolver to be configured")
+	}
+	if len(o.Accounts) > 0 {
+		return fmt.Errorf("operators do not allow Accounts to be configured directly")
+	}
+	if len(o.Users) > 0 || len(o.Nkeys) > 0 {
+		return fmt.Errorf("operators do not allow users to be configured directly")
+	}
+	if len(o.TrustedOperators) > 0 && len(o.TrustedKeys) > 0 {
+		return fmt.Errorf("conflicting options for 'TrustedKeys' and 'TrustedOperators'")
+	}
+	if o.SystemAccount != _EMPTY_ {
+		foundSys := false
+		foundNonEmpty := false
+		for _, op := range o.TrustedOperators {
+			if op.SystemAccount != _EMPTY_ {
+				foundNonEmpty = true
+			}
+			if op.SystemAccount == o.SystemAccount {
+				foundSys = true
+				break
+			}
+		}
+		if foundNonEmpty && !foundSys {
+			return fmt.Errorf("system_account in config and operator JWT must be identical")
+		}
+	} else if o.TrustedOperators[0].SystemAccount == _EMPTY_ {
+		// In case the system account is neither defined in config nor in the first operator.
+		// If it would be needed due to the nats account resolver, raise an error.
+		switch o.AccountResolver.(type) {
+		case *DirAccResolver, *CacheDirAccResolver:
+			return fmt.Errorf("using nats based account resolver - the system account needs to be specified in configuration or the operator jwt")
+		}
+	}
+
+	srvMajor, srvMinor, srvUpdate, _ := versionComponents(VERSION)
+	for _, opc := range o.TrustedOperators {
+		if major, minor, update, err := jwt.ParseServerVersion(opc.AssertServerVersion); err != nil {
+			return fmt.Errorf("operator %s expects version %s got error instead: %s",
+				opc.Subject, opc.AssertServerVersion, err)
+		} else if major > srvMajor {
+			return fmt.Errorf("operator %s expected major version %d > server major version %d",
+				opc.Subject, major, srvMajor)
+		} else if srvMajor > major {
+		} else if minor > srvMinor {
+			return fmt.Errorf("operator %s expected minor version %d > server minor version %d",
+				opc.Subject, minor, srvMinor)
+		} else if srvMinor > minor {
+		} else if update > srvUpdate {
+			return fmt.Errorf("operator %s expected update version %d > server update version %d",
+				opc.Subject, update, srvUpdate)
+		}
+	}
+	// If we have operators, fill in the trusted keys.
+	// FIXME(dlc) - We had TrustedKeys before TrustedOperators. The jwt.OperatorClaims
+	// has a DidSign(). Use that longer term. For now we can expand in place.
+	for _, opc := range o.TrustedOperators {
+		if o.TrustedKeys == nil {
+			o.TrustedKeys = make([]string, 0, 4)
+		}
+		if !opc.StrictSigningKeyUsage {
+			o.TrustedKeys = append(o.TrustedKeys, opc.Subject)
+		}
+		o.TrustedKeys = append(o.TrustedKeys, opc.SigningKeys...)
+	}
+	for _, key := range o.TrustedKeys {
+		if !nkeys.IsValidPublicOperatorKey(key) {
+			return fmt.Errorf("trusted Keys %q are required to be a valid public operator nkey", key)
+		}
+	}
+	if len(o.resolverPinnedAccounts) > 0 {
+		for key := range o.resolverPinnedAccounts {
+			if !nkeys.IsValidPublicAccountKey(key) {
+				return fmt.Errorf("pinned account key %q is not a valid public account nkey", key)
+			}
+		}
+		// ensure the system account (belonging to the operator can always connect)
+		if o.SystemAccount != _EMPTY_ {
+			o.resolverPinnedAccounts[o.SystemAccount] = struct{}{}
+		}
+	}
+
+	// If we have an auth callout defined make sure we are not in operator mode.
+	if o.AuthCallout != nil {
+		return errors.New("operators do not allow authorization callouts to be configured directly")
+	}
+
+	return nil
+}
+
+func validateSrc(claims *jwt.UserClaims, host string) bool {
+	if claims == nil {
+		return false
+	} else if len(claims.Src) == 0 {
+		return true
+	} else if host == "" {
+		return false
+	}
+	ip := net.ParseIP(host)
+	if ip == nil {
+		return false
+	}
+	for _, cidr := range claims.Src {
+		if _, net, err := net.ParseCIDR(cidr); err != nil {
+			return false // should not happen as this jwt is invalid
+		} else if net.Contains(ip) {
+			return true
+		}
+	}
+	return false
+}
+
+func validateTimes(claims *jwt.UserClaims) (bool, time.Duration) {
+	return validateTimesAt(claims, time.Now())
+}
+
+func validateTimesAt(claims *jwt.UserClaims, now time.Time) (bool, time.Duration) {
+	if claims == nil {
+		return false, time.Duration(0)
+	} else if len(claims.Times) == 0 {
+		return true, time.Duration(0)
+	}
+	loc := time.Local
+	if claims.Locale != "" {
+		var err error
+		if loc, err = time.LoadLocation(claims.Locale); err != nil {
+			return false, time.Duration(0) // parsing not expected to fail at this point
+		}
+		now = now.In(loc)
+	}
+
+	var ok bool
+	var validFor time.Duration
+
+	for _, timeRange := range claims.Times {
+		start, err := time.ParseInLocation("15:04:05", timeRange.Start, loc)
+		if err != nil {
+			return false, time.Duration(0) // parsing not expected to fail at this point
+		}
+		end, err := time.ParseInLocation("15:04:05", timeRange.End, loc)
+		if err != nil {
+			return false, time.Duration(0) // parsing not expected to fail at this point
+		}
+
+		y, m, d := now.Date()
+		start = time.Date(y, m, d, start.Hour(), start.Minute(), start.Second(), 0, loc)
+		end = time.Date(y, m, d, end.Hour(), end.Minute(), end.Second(), 0, loc)
+
+		inRange, expires := validateTimeRangeAt(start, end, now)
+		if inRange && (!ok || expires > validFor) {
+			ok = true
+			validFor = expires
+		}
+	}
+	return ok, validFor
+}
+
+// Returns true if now is within `start` and `end`, and
+// how much time is left until `end`.
+// False if `now` is not within range.
+func validateTimeRangeAt(start, end, now time.Time) (bool, time.Duration) {
+	// Now falls within range.
+	// For example 11:00-22:00 at 13:00
+	if start.Before(now) && end.After(now) {
+		return true, end.Sub(now)
+	}
+
+	// Range crosses midnight.
+	if start.After(end) {
+		// Now is after midnight.
+		// For example 22:00-06:00 at 05:00.
+		if end.After(now) {
+			return true, end.Sub(now)
+		}
+
+		// Now is before midnight.
+		// For example 22:00-06:00 at 23:30.
+		end = end.AddDate(0, 0, 1)
+		if start.Before(now) && end.After(now) {
+			return true, end.Sub(now)
+		}
+	}
+	return false, time.Duration(0)
+}

server/leafnode_proxy_test.go

@@ -0,0 +1,929 @@
+// Copyright 2025-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"errors"
+	"fmt"
+	"net"
+	"net/url"
+	"strings"
+	"testing"
+	"time"
+)
+
+// Basic HTTP proxy for testing
+type testHTTPProxy struct {
+	listener    net.Listener
+	port        int
+	username    string
+	password    string
+	started     bool
+	closeDelay  time.Duration // Delay before closing connections for robustness
+	connections []net.Conn    // Track connections for cleanup
+}
+
+func createTestHTTPProxy(username, password string) *testHTTPProxy {
+	l, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		panic(err)
+	}
+	port := l.Addr().(*net.TCPAddr).Port
+
+	proxy := &testHTTPProxy{
+		listener:    l,
+		port:        port,
+		username:    username,
+		password:    password,
+		closeDelay:  100 * time.Millisecond, // Default delay for test robustness
+		connections: make([]net.Conn, 0),
+	}
+
+	return proxy
+}
+
+func (p *testHTTPProxy) setCloseDelay(delay time.Duration) {
+	p.closeDelay = delay
+}
+
+func (p *testHTTPProxy) start() {
+	if p.started {
+		return
+	}
+	p.started = true
+
+	go func() {
+		for {
+			conn, err := p.listener.Accept()
+			if err != nil {
+				return
+			}
+			p.connections = append(p.connections, conn)
+			go p.handleConnection(conn)
+		}
+	}()
+}
+
+func (p *testHTTPProxy) handleConnection(conn net.Conn) {
+	defer func() {
+		if p.closeDelay > 0 {
+			time.Sleep(p.closeDelay)
+		}
+		conn.Close()
+	}()
+
+	// Set read timeout to prevent hanging on malformed requests
+	conn.SetReadDeadline(time.Now().Add(10 * time.Second))
+
+	// Read the CONNECT request
+	buffer := make([]byte, 4096)
+	n, err := conn.Read(buffer)
+	if err != nil {
+		return
+	}
+
+	request := string(buffer[:n])
+	lines := strings.Split(request, "\r\n")
+
+	if len(lines) == 0 || !strings.HasPrefix(lines[0], "CONNECT ") {
+		conn.Write([]byte("HTTP/1.1 400 Bad Request\r\n\r\n"))
+		return
+	}
+
+	// Check authentication if required
+	if p.username != _EMPTY_ || p.password != _EMPTY_ {
+		authFound := false
+		for _, line := range lines {
+			if strings.HasPrefix(line, "Proxy-Authorization: Basic ") {
+				authFound = true
+				break
+			}
+		}
+		if !authFound {
+			conn.Write([]byte("HTTP/1.1 407 Proxy Authentication Required\r\n\r\n"))
+			return
+		}
+	}
+
+	// Extract target host from CONNECT line
+	parts := strings.Fields(lines[0])
+	if len(parts) < 3 {
+		conn.Write([]byte("HTTP/1.1 400 Bad Request\r\n\r\n"))
+		return
+	}
+
+	targetHost := parts[1]
+
+	// Connect to target with timeout
+	target, err := net.DialTimeout("tcp", targetHost, 5*time.Second)
+	if err != nil {
+		conn.Write([]byte("HTTP/1.1 502 Bad Gateway\r\n\r\n"))
+		return
+	}
+	defer target.Close()
+
+	// Send success response
+	conn.Write([]byte("HTTP/1.1 200 Connection established\r\n\r\n"))
+
+	// Clear read deadline for ongoing connection
+	conn.SetReadDeadline(time.Time{})
+
+	// Relay data between client and target with proper error handling
+	done := make(chan bool, 2)
+
+	// Client to target
+	go func() {
+		defer func() {
+			done <- true
+			target.Close()
+		}()
+		buffer := make([]byte, 32*1024)
+		for {
+			conn.SetReadDeadline(time.Now().Add(30 * time.Second))
+			n, err := conn.Read(buffer)
+			if err != nil {
+				return
+			}
+			target.SetWriteDeadline(time.Now().Add(10 * time.Second))
+			_, err = target.Write(buffer[:n])
+			if err != nil {
+				return
+			}
+		}
+	}()
+
+	// Target to client
+	go func() {
+		defer func() {
+			done <- true
+			conn.Close()
+		}()
+		buffer := make([]byte, 32*1024)
+		for {
+			target.SetReadDeadline(time.Now().Add(30 * time.Second))
+			n, err := target.Read(buffer)
+			if err != nil {
+				return
+			}
+			conn.SetWriteDeadline(time.Now().Add(10 * time.Second))
+			_, err = conn.Write(buffer[:n])
+			if err != nil {
+				return
+			}
+		}
+	}()
+
+	// Wait for either direction to finish
+	<-done
+}
+
+func (p *testHTTPProxy) stop() {
+	if p.listener != nil {
+		p.listener.Close()
+	}
+	// Close all tracked connections with delay for robustness
+	for _, conn := range p.connections {
+		go func(c net.Conn) {
+			if p.closeDelay > 0 {
+				time.Sleep(p.closeDelay)
+			}
+			c.Close()
+		}(conn)
+	}
+}
+
+func (p *testHTTPProxy) url() string {
+	return fmt.Sprintf("http://127.0.0.1:%d", p.port)
+}
+
+func TestLeafNodeHttpProxyConfigParsing(t *testing.T) {
+	// Test valid proxy configuration
+	conf := `
+		leafnodes {
+			remotes = [
+				{
+					url: "ws://127.0.0.1:7422"
+					proxy {
+						url: "http://proxy.example.com:8080"
+						username: "user"
+						password: "pass"
+						timeout: "10s"
+					}
+				}
+			]
+		}
+	`
+
+	configFile := createConfFile(t, []byte(conf))
+
+	opts, err := ProcessConfigFile(configFile)
+	if err != nil {
+		t.Fatalf("Error parsing config: %v", err)
+	}
+
+	if len(opts.LeafNode.Remotes) != 1 {
+		t.Fatalf("Expected 1 remote, got %d", len(opts.LeafNode.Remotes))
+	}
+
+	remote := opts.LeafNode.Remotes[0]
+	if remote.Proxy.URL != "http://proxy.example.com:8080" {
+		t.Errorf("Expected proxy URL 'http://proxy.example.com:8080', got '%s'", remote.Proxy.URL)
+	}
+	if remote.Proxy.Username != "user" {
+		t.Errorf("Expected proxy username 'user', got '%s'", remote.Proxy.Username)
+	}
+	if remote.Proxy.Password != "pass" {
+		t.Errorf("Expected proxy password 'pass', got '%s'", remote.Proxy.Password)
+	}
+	if remote.Proxy.Timeout != 10*time.Second {
+		t.Errorf("Expected proxy timeout 10s, got %v", remote.Proxy.Timeout)
+	}
+}
+
+func TestLeafNodeHttpProxyConfigWarnings(t *testing.T) {
+	testCases := []struct {
+		name          string
+		config        string
+		expectWarning bool
+		warningMatch  string
+	}{
+		{
+			name: "proxy with only TCP URLs",
+			config: `
+				leafnodes {
+					remotes = [
+						{
+							url: "nats://127.0.0.1:7422"
+							proxy {
+								url: "http://proxy.example.com:8080"
+							}
+						}
+					]
+				}
+			`,
+			expectWarning: true,
+			warningMatch:  "proxy configuration will be ignored",
+		},
+		{
+			name: "proxy with mixed TCP and WebSocket URLs",
+			config: `
+				leafnodes {
+					remotes = [
+						{
+							urls: ["nats://127.0.0.1:7422", "ws://127.0.0.1:8080"]
+							proxy {
+								url: "http://proxy.example.com:8080"
+							}
+						}
+					]
+				}
+			`,
+			expectWarning: true,
+			warningMatch:  "proxy configuration will only be used for WebSocket URLs",
+		},
+		{
+			name: "proxy with only WebSocket URLs",
+			config: `
+				leafnodes {
+					remotes = [
+						{
+							url: "ws://127.0.0.1:7422"
+							proxy {
+								url: "http://proxy.example.com:8080"
+							}
+						}
+					]
+				}
+			`,
+			expectWarning: false,
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			configFile := createConfFile(t, []byte(tc.config))
+
+			opts, err := ProcessConfigFile(configFile)
+
+			if tc.expectWarning {
+				// With ProcessConfigFile, warnings don't cause errors
+				// The configuration should be valid but might log warnings
+				if err != nil {
+					t.Fatalf("Expected valid configuration with warnings, but got error: %v", err)
+				}
+				if opts == nil {
+					t.Fatal("Expected valid options but got nil")
+				}
+				// Note: With ProcessConfigFile, warnings are filtered out and not returned as errors
+				// The test verifies that the configuration is valid despite having warning conditions
+			} else {
+				// No warnings expected - should parse successfully
+				if err != nil {
+					t.Fatalf("Expected no error but got: %v", err)
+				}
+				if opts == nil {
+					t.Fatal("Expected valid options but got nil")
+				}
+			}
+		})
+	}
+}
+
+func TestLeafNodeHttpProxyConnection(t *testing.T) {
+	// Create a hub server with WebSocket support using config file
+	hubConfig := createConfFile(t, []byte(`
+		listen: "127.0.0.1:-1"
+		websocket {
+			listen: "127.0.0.1:-1"
+			no_tls: true
+		}
+		leafnodes {
+			listen: "127.0.0.1:-1"
+		}
+	`))
+
+	hub, hubOpts := RunServerWithConfig(hubConfig)
+	defer hub.Shutdown()
+
+	// Create HTTP proxy
+	proxy := createTestHTTPProxy(_EMPTY_, _EMPTY_)
+	proxy.start()
+	defer proxy.stop()
+
+	// Create spoke server with proxy configuration via config file
+	configContent := fmt.Sprintf(`
+		listen: "127.0.0.1:-1"
+		leafnodes {
+			reconnect_interval: "50ms"
+			remotes = [
+				{
+					url: "ws://127.0.0.1:%d"
+					proxy {
+						url: "%s"
+						timeout: 5s
+					}
+				}
+			]
+		}
+	`, hubOpts.Websocket.Port, proxy.url())
+
+	configFile := createConfFile(t, []byte(configContent))
+
+	spoke, _ := RunServerWithConfig(configFile)
+	defer spoke.Shutdown()
+
+	// Verify leafnode connections are established
+	checkLeafNodeConnected(t, spoke)
+	checkLeafNodeConnected(t, hub)
+}
+
+func TestLeafNodeHttpProxyConnectionToTCP(t *testing.T) {
+	// Create a hub server using config file
+	hubConfig := createConfFile(t, []byte(`
+		listen: "127.0.0.1:-1"
+		leafnodes {
+			listen: "127.0.0.1:-1"
+		}
+	`))
+
+	hub, hubOpts := RunServerWithConfig(hubConfig)
+	defer hub.Shutdown()
+
+	// Create HTTP proxy
+	proxy := createTestHTTPProxy(_EMPTY_, _EMPTY_)
+	proxy.start()
+	defer proxy.stop()
+
+	// Create spoke server with proxy configuration via config file
+	configContent := fmt.Sprintf(`
+		listen: "127.0.0.1:-1"
+		leafnodes {
+			reconnect_interval: "50ms"
+			remotes = [
+				{
+					url: "nats://127.0.0.1:%d"
+					proxy {
+						url: "%s"
+						timeout: 5s
+					}
+				}
+			]
+		}
+	`, hubOpts.LeafNode.Port, proxy.url())
+
+	configFile := createConfFile(t, []byte(configContent))
+
+	spoke, _ := RunServerWithConfig(configFile)
+	defer spoke.Shutdown()
+
+	// Verify leafnode connections are established
+	checkLeafNodeConnected(t, spoke)
+	checkLeafNodeConnected(t, hub)
+}
+
+func TestLeafNodeHttpProxyWithAuthentication(t *testing.T) {
+	// Create a hub server with WebSocket support using config file
+	hubConfig := createConfFile(t, []byte(`
+		listen: "127.0.0.1:-1"
+		websocket {
+			listen: "127.0.0.1:-1"
+			no_tls: true
+		}
+		leafnodes {
+			listen: "127.0.0.1:-1"
+		}
+	`))
+
+	hub, hubOpts := RunServerWithConfig(hubConfig)
+	defer hub.Shutdown()
+
+	// Create HTTP proxy with authentication
+	proxy := createTestHTTPProxy("testuser", "testpass")
+	proxy.start()
+	defer proxy.stop()
+
+	// Create spoke server with proxy configuration including auth via config file
+	configContent := fmt.Sprintf(`
+		listen: "127.0.0.1:-1"
+		leafnodes {
+			reconnect_interval: "50ms"
+			remotes = [
+				{
+					url: "ws://127.0.0.1:%d"
+					proxy {
+						url: "%s"
+						username: "testuser"
+						password: "testpass"
+						timeout: 5s
+					}
+				}
+			]
+		}
+	`, hubOpts.Websocket.Port, proxy.url())
+
+	configFile := createConfFile(t, []byte(configContent))
+
+	spoke, _ := RunServerWithConfig(configFile)
+	defer spoke.Shutdown()
+
+	// Verify leafnode connections are established
+	checkLeafNodeConnected(t, spoke)
+	checkLeafNodeConnected(t, hub)
+}
+
+func TestLeafNodeHttpProxyTLSMismatchDetection(t *testing.T) {
+	// This test simulates the TLS mismatch scenario described in the feedback:
+	// - Leafnode configured with proxy but no TLS
+	// - Hub requires TLS
+	// - Connection should fail with appropriate error message
+
+	// Create hub server with TLS required using config file
+	hubConfig := createConfFile(t, []byte(`
+		listen: "127.0.0.1:-1"
+		leafnodes {
+			listen: "127.0.0.1:-1"
+			tls: {
+				cert_file: "../test/configs/certs/server-cert.pem"
+				key_file: "../test/configs/certs/server-key.pem"
+			}
+		}
+	`))
+
+	hub, hubOpts := RunServerWithConfig(hubConfig)
+	defer hub.Shutdown()
+
+	// Create HTTP proxy
+	proxy := createTestHTTPProxy(_EMPTY_, _EMPTY_)
+	proxy.start()
+	defer proxy.stop()
+
+	// Create spoke server with proxy but no TLS configuration (intentional mismatch)
+	spokeConfigContent := fmt.Sprintf(`
+		listen: "127.0.0.1:-1"
+		leafnodes {
+			reconnect_interval: "50ms"
+			remotes = [
+				{
+					url: "ws://127.0.0.1:%d"
+					proxy {
+						url: "%s"
+						timeout: 5s
+					}
+					# Intentionally no TLS configuration to create mismatch
+				}
+			]
+		}
+	`, hubOpts.LeafNode.Port, proxy.url())
+
+	spokeConfig := createConfFile(t, []byte(spokeConfigContent))
+	spoke, _ := RunServerWithConfig(spokeConfig)
+	defer spoke.Shutdown()
+
+	// Wait and verify that connection was NOT established due to TLS mismatch
+	// First attempt happens during RunServerWithConfig(), then retries every 50ms
+	time.Sleep(250 * time.Millisecond)
+
+	if spoke.NumLeafNodes() != 0 {
+		t.Errorf("Expected 0 leafnode connections due to TLS mismatch, got %d", spoke.NumLeafNodes())
+	}
+}
+
+func TestLeafNodeHttpProxyTunnelBasic(t *testing.T) {
+	// Create HTTP proxy with longer delay for robustness
+	proxy := createTestHTTPProxy(_EMPTY_, _EMPTY_)
+	proxy.setCloseDelay(200 * time.Millisecond)
+	proxy.start()
+	defer proxy.stop()
+
+	// Create a simple TCP server to connect to through proxy
+	listener, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatalf("Failed to create test server: %v", err)
+	}
+	defer listener.Close()
+
+	targetPort := listener.Addr().(*net.TCPAddr).Port
+	targetHost := fmt.Sprintf("127.0.0.1:%d", targetPort)
+
+	errCh := make(chan error, 1)
+
+	// Accept one connection with proper timeout handling
+	go func() {
+		conn, err := listener.Accept()
+		if err != nil {
+			errCh <- fmt.Errorf("unable to accept: %v", err)
+			return
+		}
+		defer conn.Close()
+
+		// Set read deadline to prevent hanging forever
+		conn.SetReadDeadline(time.Now().Add(5 * time.Second))
+
+		// Read the incoming data first
+		buffer := make([]byte, 1024)
+		n, err := conn.Read(buffer)
+		if err != nil {
+			errCh <- fmt.Errorf("server failed to read: %v", err)
+			return
+		}
+
+		receivedMsg := string(buffer[:n])
+		if receivedMsg != "Hello" {
+			errCh <- fmt.Errorf("server expected 'Hello', got '%s'", receivedMsg)
+			return
+		}
+
+		// Send response
+		conn.SetWriteDeadline(time.Now().Add(5 * time.Second))
+		_, err = conn.Write([]byte("Hello from target server"))
+		if err != nil {
+			errCh <- fmt.Errorf("server failed to write: %v", err)
+			return
+		}
+
+		// Wait a bit to ensure the client has time to read before closing
+		time.Sleep(50 * time.Millisecond)
+		errCh <- nil
+	}()
+
+	// Test establishing proxy tunnel with timeout
+	conn, err := establishHTTPProxyTunnel(proxy.url(), targetHost, 10*time.Second, _EMPTY_, _EMPTY_)
+	if err != nil {
+		t.Fatalf("Failed to establish proxy tunnel: %v", err)
+	}
+	defer conn.Close()
+
+	// Test that we can communicate through the tunnel with deadlines
+	conn.SetWriteDeadline(time.Now().Add(5 * time.Second))
+	_, err = conn.Write([]byte("Hello"))
+	if err != nil {
+		t.Fatalf("Failed to write to proxy tunnel: %v", err)
+	}
+
+	conn.SetReadDeadline(time.Now().Add(5 * time.Second))
+	buffer := make([]byte, 1024)
+	n, err := conn.Read(buffer)
+	if err != nil {
+		t.Fatalf("Failed to read from proxy tunnel: %v", err)
+	}
+
+	response := string(buffer[:n])
+	if response != "Hello from target server" {
+		t.Errorf("Unexpected response: '%s', expected 'Hello from target server'", response)
+	}
+
+	select {
+	case err := <-errCh:
+		if err != nil {
+			t.Fatalf("%v", err)
+		}
+	case <-time.After(3 * time.Second):
+		t.Fatal("Server goroutine didn't complete in time, but test data was exchanged successfully")
+	}
+}
+
+func TestLeafNodeHttpProxyTunnelWithAuth(t *testing.T) {
+	// Create HTTP proxy with authentication and delay for robustness
+	proxy := createTestHTTPProxy("testuser", "testpass")
+	proxy.setCloseDelay(200 * time.Millisecond)
+	proxy.start()
+	defer proxy.stop()
+
+	// Create a simple TCP server to connect to through proxy
+	listener, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatalf("Failed to create test server: %v", err)
+	}
+	defer listener.Close()
+
+	targetPort := listener.Addr().(*net.TCPAddr).Port
+	targetHost := fmt.Sprintf("127.0.0.1:%d", targetPort)
+
+	errCh := make(chan error, 1)
+
+	// Accept one connection with proper timeout handling
+	go func() {
+		conn, err := listener.Accept()
+		if err != nil {
+			errCh <- fmt.Errorf("unable to accept: %v", err)
+			return
+		}
+		defer conn.Close()
+
+		// Set read deadline to prevent hanging
+		conn.SetReadDeadline(time.Now().Add(5 * time.Second))
+
+		// Read the incoming data first
+		buffer := make([]byte, 1024)
+		_, err = conn.Read(buffer)
+		if err != nil {
+			errCh <- fmt.Errorf("unable to read: %v", err)
+			return
+		}
+
+		// Send response
+		conn.SetWriteDeadline(time.Now().Add(5 * time.Second))
+		if _, err := conn.Write([]byte("Hello from authenticated server")); err != nil {
+			errCh <- fmt.Errorf("unable to write: %v", err)
+			return
+		}
+
+		errCh <- nil
+	}()
+
+	// Test establishing proxy tunnel with authentication and timeout
+	conn, err := establishHTTPProxyTunnel(proxy.url(), targetHost, 10*time.Second, "testuser", "testpass")
+	if err != nil {
+		t.Fatalf("Failed to establish proxy tunnel with auth: %v", err)
+	}
+	defer conn.Close()
+
+	// Test that we can communicate through the tunnel with deadlines
+	conn.SetWriteDeadline(time.Now().Add(5 * time.Second))
+	_, err = conn.Write([]byte("Hello"))
+	if err != nil {
+		t.Fatalf("Failed to write to proxy tunnel: %v", err)
+	}
+
+	conn.SetReadDeadline(time.Now().Add(5 * time.Second))
+	buffer := make([]byte, 1024)
+	n, err := conn.Read(buffer)
+	if err != nil {
+		t.Fatalf("Failed to read from proxy tunnel: %v", err)
+	}
+
+	response := string(buffer[:n])
+	if response != "Hello from authenticated server" {
+		t.Errorf("Unexpected response: %s", response)
+	}
+
+	select {
+	case err := <-errCh:
+		if err != nil {
+			t.Fatalf("%v", err)
+		}
+	case <-time.After(2 * time.Second):
+		t.Fatal("Server goroutine didn't complete in time, but test data was exchanged successfully")
+	}
+}
+
+func TestLeafNodeHttpProxyTunnelFailsWithoutAuth(t *testing.T) {
+	// Create HTTP proxy with authentication required and delay for robustness
+	proxy := createTestHTTPProxy("testuser", "testpass")
+	proxy.setCloseDelay(200 * time.Millisecond)
+	proxy.start()
+	defer proxy.stop()
+
+	// Try to establish tunnel without providing credentials (should fail quickly)
+	_, err := establishHTTPProxyTunnel(proxy.url(), "127.0.0.1:80", 10*time.Second, _EMPTY_, _EMPTY_)
+	if err == nil {
+		t.Fatal("Expected error when connecting without authentication")
+	}
+
+	if !strings.Contains(err.Error(), "proxy CONNECT failed") {
+		t.Errorf("Expected proxy authentication error, got: %v", err)
+	}
+
+	// Verify the error contains the expected HTTP response code
+	if !strings.Contains(err.Error(), "407") {
+		t.Errorf("Expected HTTP 407 error in response, got: %v", err)
+	}
+}
+
+// TestLeafNodeProxyValidationProgrammatic tests proxy validation when configuring server programmatically
+func TestLeafNodeHttpProxyValidationProgrammatic(t *testing.T) {
+	tests := []struct {
+		// name is the name of the test.
+		name string
+
+		// setupOptions creates the Options configuration for the test.
+		setupOptions func() *Options
+
+		// err is the expected error. nil means no error expected.
+		err error
+	}{
+		{
+			name: "invalid proxy scheme",
+			setupOptions: func() *Options {
+				opts := &Options{}
+				opts.LeafNode.Remotes = []*RemoteLeafOpts{
+					{
+						URLs: []*url.URL{{Scheme: wsSchemePrefix, Host: "127.0.0.1:7422"}},
+					},
+				}
+				opts.LeafNode.Remotes[0].Proxy.URL = "ftp://proxy.example.com:21"
+				return opts
+			},
+			err: errors.New("proxy URL scheme must be http or https"),
+		},
+		{
+			name: "empty proxy URL - no validation performed",
+			setupOptions: func() *Options {
+				opts := &Options{}
+				opts.LeafNode.Remotes = []*RemoteLeafOpts{
+					{
+						URLs: []*url.URL{{Scheme: wsSchemePrefix, Host: "127.0.0.1:7422"}},
+					},
+				}
+				opts.LeafNode.Remotes[0].Proxy.URL = _EMPTY_
+				return opts
+			},
+			err: nil, // No error expected for empty URL
+		},
+		{
+			name: "invalid proxy URL parse failure",
+			setupOptions: func() *Options {
+				opts := &Options{}
+				opts.LeafNode.Remotes = []*RemoteLeafOpts{
+					{
+						URLs: []*url.URL{{Scheme: wsSchemePrefix, Host: "127.0.0.1:7422"}},
+					},
+				}
+				opts.LeafNode.Remotes[0].Proxy.URL = "ht!tp://invalid-url-with-bad-characters"
+				return opts
+			},
+			err: errors.New("invalid proxy URL"),
+		},
+		{
+			name: "missing proxy host",
+			setupOptions: func() *Options {
+				opts := &Options{}
+				opts.LeafNode.Remotes = []*RemoteLeafOpts{
+					{
+						URLs: []*url.URL{{Scheme: wsSchemePrefix, Host: "127.0.0.1:7422"}},
+					},
+				}
+				opts.LeafNode.Remotes[0].Proxy.URL = "http://"
+				return opts
+			},
+			err: errors.New("proxy URL must specify a host"),
+		},
+		{
+			name: "username without password",
+			setupOptions: func() *Options {
+				opts := &Options{}
+				opts.LeafNode.Remotes = []*RemoteLeafOpts{
+					{
+						URLs: []*url.URL{{Scheme: wsSchemePrefix, Host: "127.0.0.1:7422"}},
+					},
+				}
+				opts.LeafNode.Remotes[0].Proxy.URL = "http://proxy.example.com:8080"
+				opts.LeafNode.Remotes[0].Proxy.Username = "user"
+				return opts
+			},
+			err: errors.New("proxy username and password must both be specified"),
+		},
+		{
+			name: "password without username",
+			setupOptions: func() *Options {
+				opts := &Options{}
+				opts.LeafNode.Remotes = []*RemoteLeafOpts{
+					{
+						URLs: []*url.URL{{Scheme: wsSchemePrefix, Host: "127.0.0.1:7422"}},
+					},
+				}
+				opts.LeafNode.Remotes[0].Proxy.URL = "http://proxy.example.com:8080"
+				opts.LeafNode.Remotes[0].Proxy.Password = "pass"
+				return opts
+			},
+			err: errors.New("proxy username and password must both be specified"),
+		},
+		{
+			name: "negative timeout value",
+			setupOptions: func() *Options {
+				opts := &Options{}
+				opts.LeafNode.Remotes = []*RemoteLeafOpts{
+					{
+						URLs: []*url.URL{{Scheme: wsSchemePrefix, Host: "127.0.0.1:7422"}},
+					},
+				}
+				opts.LeafNode.Remotes[0].Proxy.URL = "http://proxy.example.com:8080"
+				opts.LeafNode.Remotes[0].Proxy.Timeout = -5 * time.Second
+				return opts
+			},
+			err: errors.New("proxy timeout must be >= 0"),
+		},
+		{
+			name: "zero timeout value - valid",
+			setupOptions: func() *Options {
+				opts := &Options{}
+				opts.LeafNode.Remotes = []*RemoteLeafOpts{
+					{
+						URLs: []*url.URL{{Scheme: wsSchemePrefix, Host: "127.0.0.1:7422"}},
+					},
+				}
+				opts.LeafNode.Remotes[0].Proxy.URL = "http://proxy.example.com:8080"
+				opts.LeafNode.Remotes[0].Proxy.Timeout = 0
+				return opts
+			},
+			err: nil, // No error expected for zero timeout
+		},
+		{
+			name: "valid proxy configuration with authentication",
+			setupOptions: func() *Options {
+				opts := &Options{}
+				opts.LeafNode.Remotes = []*RemoteLeafOpts{
+					{
+						URLs: []*url.URL{{Scheme: wsSchemePrefix, Host: "127.0.0.1:7422"}},
+					},
+				}
+				opts.LeafNode.Remotes[0].Proxy.URL = "http://proxy.example.com:8080"
+				opts.LeafNode.Remotes[0].Proxy.Username = "user"
+				opts.LeafNode.Remotes[0].Proxy.Password = "pass"
+				opts.LeafNode.Remotes[0].Proxy.Timeout = 10 * time.Second
+				return opts
+			},
+			err: nil, // No error expected
+		},
+		{
+			name: "valid proxy configuration without authentication",
+			setupOptions: func() *Options {
+				opts := &Options{}
+				opts.LeafNode.Remotes = []*RemoteLeafOpts{
+					{
+						URLs: []*url.URL{{Scheme: wsSchemePrefix, Host: "127.0.0.1:7422"}},
+					},
+				}
+				opts.LeafNode.Remotes[0].Proxy.URL = "https://proxy.example.com:3128"
+				opts.LeafNode.Remotes[0].Proxy.Timeout = 30 * time.Second
+				return opts
+			},
+			err: nil, // No error expected
+		},
+	}
+
+	checkErr := func(t *testing.T, err, expectedErr error) {
+		t.Helper()
+		switch {
+		case err == nil && expectedErr == nil:
+			// OK
+		case err != nil && expectedErr == nil:
+			t.Errorf("Unexpected error after validating options: %s", err)
+		case err == nil && expectedErr != nil:
+			t.Errorf("Expected %q error after validating invalid options but got nothing", expectedErr)
+		case err != nil && expectedErr != nil:
+			if !strings.Contains(err.Error(), expectedErr.Error()) {
+				t.Errorf("Expected error containing %q, got: %q", expectedErr.Error(), err.Error())
+			}
+		}
+	}
+
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			opts := test.setupOptions()
+			err := validateLeafNode(opts)
+			checkErr(t, err, test.err)
+		})
+	}
+}

server/log.go

@@ -0,0 +1,298 @@
+// Copyright 2012-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"fmt"
+	"io"
+	"os"
+	"sync/atomic"
+	"time"
+
+	srvlog "github.com/nats-io/nats-server/v2/logger"
+)
+
+// Logger interface of the NATS Server
+type Logger interface {
+
+	// Log a notice statement
+	Noticef(format string, v ...any)
+
+	// Log a warning statement
+	Warnf(format string, v ...any)
+
+	// Log a fatal error
+	Fatalf(format string, v ...any)
+
+	// Log an error
+	Errorf(format string, v ...any)
+
+	// Log a debug statement
+	Debugf(format string, v ...any)
+
+	// Log a trace statement
+	Tracef(format string, v ...any)
+}
+
+// ConfigureLogger configures and sets the logger for the server.
+func (s *Server) ConfigureLogger() {
+	var (
+		log Logger
+
+		// Snapshot server options.
+		opts = s.getOpts()
+	)
+
+	if opts.NoLog {
+		return
+	}
+
+	syslog := opts.Syslog
+	if isWindowsService() && opts.LogFile == "" {
+		// Enable syslog if no log file is specified and we're running as a
+		// Windows service so that logs are written to the Windows event log.
+		syslog = true
+	}
+
+	if opts.LogFile != "" {
+		log = srvlog.NewFileLogger(opts.LogFile, opts.Logtime, opts.Debug, opts.Trace, true, srvlog.LogUTC(opts.LogtimeUTC))
+		if opts.LogSizeLimit > 0 {
+			if l, ok := log.(*srvlog.Logger); ok {
+				l.SetSizeLimit(opts.LogSizeLimit)
+			}
+		}
+		if opts.LogMaxFiles > 0 {
+			if l, ok := log.(*srvlog.Logger); ok {
+				al := int(opts.LogMaxFiles)
+				if int64(al) != opts.LogMaxFiles {
+					// set to default (no max) on overflow
+					al = 0
+				}
+				l.SetMaxNumFiles(al)
+			}
+		}
+	} else if opts.RemoteSyslog != "" {
+		log = srvlog.NewRemoteSysLogger(opts.RemoteSyslog, opts.Debug, opts.Trace)
+	} else if syslog {
+		log = srvlog.NewSysLogger(opts.Debug, opts.Trace)
+	} else {
+		colors := true
+		// Check to see if stderr is being redirected and if so turn off color
+		// Also turn off colors if we're running on Windows where os.Stderr.Stat() returns an invalid handle-error
+		stat, err := os.Stderr.Stat()
+		if err != nil || (stat.Mode()&os.ModeCharDevice) == 0 {
+			colors = false
+		}
+		log = srvlog.NewStdLogger(opts.Logtime, opts.Debug, opts.Trace, colors, true, srvlog.LogUTC(opts.LogtimeUTC))
+	}
+
+	s.SetLoggerV2(log, opts.Debug, opts.Trace, opts.TraceVerbose)
+}
+
+// Returns our current logger.
+func (s *Server) Logger() Logger {
+	s.logging.Lock()
+	defer s.logging.Unlock()
+	return s.logging.logger
+}
+
+// SetLogger sets the logger of the server
+func (s *Server) SetLogger(logger Logger, debugFlag, traceFlag bool) {
+	s.SetLoggerV2(logger, debugFlag, traceFlag, false)
+}
+
+// SetLogger sets the logger of the server
+func (s *Server) SetLoggerV2(logger Logger, debugFlag, traceFlag, sysTrace bool) {
+	if debugFlag {
+		atomic.StoreInt32(&s.logging.debug, 1)
+	} else {
+		atomic.StoreInt32(&s.logging.debug, 0)
+	}
+	if traceFlag {
+		atomic.StoreInt32(&s.logging.trace, 1)
+	} else {
+		atomic.StoreInt32(&s.logging.trace, 0)
+	}
+	if sysTrace {
+		atomic.StoreInt32(&s.logging.traceSysAcc, 1)
+	} else {
+		atomic.StoreInt32(&s.logging.traceSysAcc, 0)
+	}
+	var prevLoggerErr error
+	s.logging.Lock()
+	if s.logging.logger != nil {
+		// Check to see if the logger implements io.Closer.  This could be a
+		// logger from another process embedding the NATS server or a dummy
+		// test logger that may not implement that interface.
+		if l, ok := s.logging.logger.(io.Closer); ok {
+			prevLoggerErr = l.Close()
+		}
+	}
+	s.logging.logger = logger
+	s.logging.Unlock()
+
+	if prevLoggerErr != nil {
+		s.Errorf("Error closing logger: %v", prevLoggerErr)
+	}
+}
+
+// ReOpenLogFile if the logger is a file based logger, close and re-open the file.
+// This allows for file rotation by 'mv'ing the file then signaling
+// the process to trigger this function.
+func (s *Server) ReOpenLogFile() {
+	// Check to make sure this is a file logger.
+	s.logging.RLock()
+	ll := s.logging.logger
+	s.logging.RUnlock()
+
+	if ll == nil {
+		s.Noticef("File log re-open ignored, no logger")
+		return
+	}
+
+	// Snapshot server options.
+	opts := s.getOpts()
+
+	if opts.LogFile == "" {
+		s.Noticef("File log re-open ignored, not a file logger")
+	} else {
+		fileLog := srvlog.NewFileLogger(
+			opts.LogFile, opts.Logtime,
+			opts.Debug, opts.Trace, true,
+			srvlog.LogUTC(opts.LogtimeUTC),
+		)
+		s.SetLogger(fileLog, opts.Debug, opts.Trace)
+		if opts.LogSizeLimit > 0 {
+			fileLog.SetSizeLimit(opts.LogSizeLimit)
+		}
+		s.Noticef("File log re-opened")
+	}
+}
+
+// Noticef logs a notice statement
+func (s *Server) Noticef(format string, v ...any) {
+	s.executeLogCall(func(logger Logger, format string, v ...any) {
+		logger.Noticef(format, v...)
+	}, format, v...)
+}
+
+// Errorf logs an error
+func (s *Server) Errorf(format string, v ...any) {
+	s.executeLogCall(func(logger Logger, format string, v ...any) {
+		logger.Errorf(format, v...)
+	}, format, v...)
+}
+
+// Error logs an error with a scope
+func (s *Server) Errors(scope any, e error) {
+	s.executeLogCall(func(logger Logger, format string, v ...any) {
+		logger.Errorf(format, v...)
+	}, "%s - %s", scope, UnpackIfErrorCtx(e))
+}
+
+// Error logs an error with a context
+func (s *Server) Errorc(ctx string, e error) {
+	s.executeLogCall(func(logger Logger, format string, v ...any) {
+		logger.Errorf(format, v...)
+	}, "%s: %s", ctx, UnpackIfErrorCtx(e))
+}
+
+// Error logs an error with a scope and context
+func (s *Server) Errorsc(scope any, ctx string, e error) {
+	s.executeLogCall(func(logger Logger, format string, v ...any) {
+		logger.Errorf(format, v...)
+	}, "%s - %s: %s", scope, ctx, UnpackIfErrorCtx(e))
+}
+
+// Warnf logs a warning error
+func (s *Server) Warnf(format string, v ...any) {
+	s.executeLogCall(func(logger Logger, format string, v ...any) {
+		logger.Warnf(format, v...)
+	}, format, v...)
+}
+
+func (s *Server) rateLimitFormatWarnf(format string, v ...any) {
+	if _, loaded := s.rateLimitLogging.LoadOrStore(format, time.Now()); loaded {
+		return
+	}
+	statement := fmt.Sprintf(format, v...)
+	s.Warnf("%s", statement)
+}
+
+func (s *Server) RateLimitErrorf(format string, v ...any) {
+	statement := fmt.Sprintf(format, v...)
+	if _, loaded := s.rateLimitLogging.LoadOrStore(statement, time.Now()); loaded {
+		return
+	}
+	s.Errorf("%s", statement)
+}
+
+func (s *Server) RateLimitWarnf(format string, v ...any) {
+	statement := fmt.Sprintf(format, v...)
+	if _, loaded := s.rateLimitLogging.LoadOrStore(statement, time.Now()); loaded {
+		return
+	}
+	s.Warnf("%s", statement)
+}
+
+func (s *Server) RateLimitDebugf(format string, v ...any) {
+	statement := fmt.Sprintf(format, v...)
+	if _, loaded := s.rateLimitLogging.LoadOrStore(statement, time.Now()); loaded {
+		return
+	}
+	s.Debugf("%s", statement)
+}
+
+// Fatalf logs a fatal error
+func (s *Server) Fatalf(format string, v ...any) {
+	if s.isShuttingDown() {
+		s.Errorf(format, v)
+		return
+	}
+	s.executeLogCall(func(logger Logger, format string, v ...any) {
+		logger.Fatalf(format, v...)
+	}, format, v...)
+}
+
+// Debugf logs a debug statement
+func (s *Server) Debugf(format string, v ...any) {
+	if atomic.LoadInt32(&s.logging.debug) == 0 {
+		return
+	}
+
+	s.executeLogCall(func(logger Logger, format string, v ...any) {
+		logger.Debugf(format, v...)
+	}, format, v...)
+}
+
+// Tracef logs a trace statement
+func (s *Server) Tracef(format string, v ...any) {
+	if atomic.LoadInt32(&s.logging.trace) == 0 {
+		return
+	}
+
+	s.executeLogCall(func(logger Logger, format string, v ...any) {
+		logger.Tracef(format, v...)
+	}, format, v...)
+}
+
+func (s *Server) executeLogCall(f func(logger Logger, format string, v ...any), format string, args ...any) {
+	s.logging.RLock()
+	defer s.logging.RUnlock()
+	if s.logging.logger == nil {
+		return
+	}
+
+	f(s.logging.logger, format, args...)
+}

server/log_test.go

@@ -0,0 +1,536 @@
+// Copyright 2012-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"bytes"
+	"os"
+	"path/filepath"
+	"runtime"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/nats-io/nats-server/v2/internal/testhelper"
+	"github.com/nats-io/nats-server/v2/logger"
+)
+
+func TestSetLogger(t *testing.T) {
+	server := &Server{}
+	defer server.SetLogger(nil, false, false)
+	dl := &DummyLogger{}
+	server.SetLogger(dl, true, true)
+
+	// We assert that the logger has change to the DummyLogger
+	_ = server.logging.logger.(*DummyLogger)
+
+	if server.logging.debug != 1 {
+		t.Fatalf("Expected debug 1, received value %d\n", server.logging.debug)
+	}
+
+	if server.logging.trace != 1 {
+		t.Fatalf("Expected trace 1, received value %d\n", server.logging.trace)
+	}
+
+	// Check traces
+	expectedStr := "This is a Notice"
+	server.Noticef(expectedStr)
+	dl.CheckContent(t, expectedStr)
+	expectedStr = "This is an Error"
+	server.Errorf(expectedStr)
+	dl.CheckContent(t, expectedStr)
+	expectedStr = "This is a Fatal"
+	server.Fatalf(expectedStr)
+	dl.CheckContent(t, expectedStr)
+	expectedStr = "This is a Debug"
+	server.Debugf(expectedStr)
+	dl.CheckContent(t, expectedStr)
+	expectedStr = "This is a Trace"
+	server.Tracef(expectedStr)
+	dl.CheckContent(t, expectedStr)
+	expectedStr = "This is a Warning"
+	server.Tracef(expectedStr)
+	dl.CheckContent(t, expectedStr)
+
+	// Make sure that we can reset to fal
+	server.SetLogger(dl, false, false)
+	if server.logging.debug != 0 {
+		t.Fatalf("Expected debug 0, got %v", server.logging.debug)
+	}
+	if server.logging.trace != 0 {
+		t.Fatalf("Expected trace 0, got %v", server.logging.trace)
+	}
+	// Now, Debug and Trace should not produce anything
+	dl.Msg = ""
+	server.Debugf("This Debug should not be traced")
+	dl.CheckContent(t, "")
+	server.Tracef("This Trace should not be traced")
+	dl.CheckContent(t, "")
+}
+
+type DummyLogger = testhelper.DummyLogger
+
+func TestReOpenLogFile(t *testing.T) {
+	// We can't rename the file log when still opened on Windows, so skip
+	if runtime.GOOS == "windows" {
+		t.SkipNow()
+	}
+	s := &Server{opts: &Options{}}
+	defer s.SetLogger(nil, false, false)
+
+	// First check with no logger
+	s.SetLogger(nil, false, false)
+	s.ReOpenLogFile()
+
+	// Then when LogFile is not provided.
+	dl := &DummyLogger{}
+	s.SetLogger(dl, false, false)
+	s.ReOpenLogFile()
+	dl.CheckContent(t, "File log re-open ignored, not a file logger")
+
+	// Set a File log
+	s.opts.LogFile = filepath.Join(t.TempDir(), "test.log")
+	fileLog := logger.NewFileLogger(s.opts.LogFile, s.opts.Logtime, s.opts.Debug, s.opts.Trace, true, logger.LogUTC(s.opts.LogtimeUTC))
+	s.SetLogger(fileLog, false, false)
+	// Add some log
+	expectedStr := "This is a Notice"
+	s.Noticef(expectedStr)
+	// Check content of log
+	buf, err := os.ReadFile(s.opts.LogFile)
+	if err != nil {
+		t.Fatalf("Error reading file: %v", err)
+	}
+	if !strings.Contains(string(buf), expectedStr) {
+		t.Fatalf("Expected log to contain: %q, got %q", expectedStr, string(buf))
+	}
+	// Close the file and rename it
+	if err := os.Rename(s.opts.LogFile, s.opts.LogFile+".bak"); err != nil {
+		t.Fatalf("Unable to rename log file: %v", err)
+	}
+	// Now re-open LogFile
+	s.ReOpenLogFile()
+	// Content should indicate that we have re-opened the log
+	buf, err = os.ReadFile(s.opts.LogFile)
+	if err != nil {
+		t.Fatalf("Error reading file: %v", err)
+	}
+	if strings.HasSuffix(string(buf), "File log-reopened") {
+		t.Fatalf("File should indicate that file log was re-opened, got: %v", string(buf))
+	}
+	// Make sure we can append to the log
+	s.Noticef("New message")
+	buf, err = os.ReadFile(s.opts.LogFile)
+	if err != nil {
+		t.Fatalf("Error reading file: %v", err)
+	}
+	if strings.HasSuffix(string(buf), "New message") {
+		t.Fatalf("New message was not appended after file was re-opened, got: %v", string(buf))
+	}
+}
+
+func TestFileLoggerSizeLimitAndReopen(t *testing.T) {
+	file := createTempFile(t, "log_")
+	file.Close()
+
+	s := &Server{opts: &Options{}}
+	defer s.SetLogger(nil, false, false)
+
+	// Set a File log
+	s.opts.LogFile = file.Name()
+	s.opts.Logtime = true
+	s.opts.LogSizeLimit = 1000
+	s.ConfigureLogger()
+
+	// Add a trace
+	s.Noticef("this is a notice")
+
+	// Do a re-open...
+	s.ReOpenLogFile()
+
+	// Content should indicate that we have re-opened the log
+	buf, err := os.ReadFile(s.opts.LogFile)
+	if err != nil {
+		t.Fatalf("Error reading file: %v", err)
+	}
+	if strings.HasSuffix(string(buf), "File log-reopened") {
+		t.Fatalf("File should indicate that file log was re-opened, got: %v", string(buf))
+	}
+
+	// Now make sure that the limit is still honored.
+	txt := make([]byte, 800)
+	for i := 0; i < len(txt); i++ {
+		txt[i] = 'A'
+	}
+	s.Noticef(string(txt))
+	for i := 0; i < len(txt); i++ {
+		txt[i] = 'B'
+	}
+	s.Noticef(string(txt))
+
+	buf, err = os.ReadFile(s.opts.LogFile)
+	if err != nil {
+		t.Fatalf("Error reading file: %v", err)
+	}
+	sbuf := string(buf)
+	if strings.Contains(sbuf, "AAAAA") || strings.Contains(sbuf, "BBBBB") {
+		t.Fatalf("Looks like file was not rotated: %s", sbuf)
+	}
+	if !strings.Contains(sbuf, "Rotated log, backup saved") {
+		t.Fatalf("File should have been rotated, was not: %s", sbuf)
+	}
+}
+
+func TestNoPasswordsFromConnectTrace(t *testing.T) {
+	opts := DefaultOptions()
+	opts.NoLog = false
+	opts.Trace = true
+	opts.Username = "derek"
+	opts.Password = "s3cr3t"
+	opts.PingInterval = 2 * time.Minute
+	setBaselineOptions(opts)
+	s := &Server{opts: opts}
+	dl := testhelper.NewDummyLogger(100)
+	s.SetLogger(dl, false, true)
+
+	_ = s.logging.logger.(*DummyLogger)
+	if s.logging.trace != 1 {
+		t.Fatalf("Expected trace 1, received value %d\n", s.logging.trace)
+	}
+	defer s.SetLogger(nil, false, false)
+
+	c, _, _ := newClientForServer(s)
+	defer c.close()
+
+	connectOp := []byte("CONNECT {\"user\":\"derek\",\"pass\":\"s3cr3t\"}\r\n")
+	err := c.parse(connectOp)
+	if err != nil {
+		t.Fatalf("Received error: %v\n", err)
+	}
+
+	dl.CheckForProhibited(t, "password found", "s3cr3t")
+}
+
+func TestRemovePassFromTrace(t *testing.T) {
+	tests := []struct {
+		name     string
+		input    string
+		expected string
+	}{
+		{
+			"user and pass",
+			"CONNECT {\"user\":\"derek\",\"pass\":\"s3cr3t\"}\r\n",
+			"CONNECT {\"user\":\"derek\",\"pass\":\"[REDACTED]\"}\r\n",
+		},
+		{
+			"user and pass extra space",
+			"CONNECT {\"user\":\"derek\",\"pass\":  \"s3cr3t\"}\r\n",
+			"CONNECT {\"user\":\"derek\",\"pass\":  \"[REDACTED]\"}\r\n",
+		},
+		{
+			"user and pass is empty",
+			"CONNECT {\"user\":\"derek\",\"pass\":\"\"}\r\n",
+			"CONNECT {\"user\":\"derek\",\"pass\":\"[REDACTED]\"}\r\n",
+		},
+		{
+			"user and pass is empty whitespace",
+			"CONNECT {\"user\":\"derek\",\"pass\":\"               \"}\r\n",
+			"CONNECT {\"user\":\"derek\",\"pass\":\"[REDACTED]\"}\r\n",
+		},
+		{
+			"user and pass whitespace",
+			"CONNECT {\"user\":\"derek\",\"pass\":    \"s3cr3t\"     }\r\n",
+			"CONNECT {\"user\":\"derek\",\"pass\":    \"[REDACTED]\"     }\r\n",
+		},
+		{
+			"only pass",
+			"CONNECT {\"pass\":\"s3cr3t\",}\r\n",
+			"CONNECT {\"pass\":\"[REDACTED]\",}\r\n",
+		},
+		{
+			"invalid json",
+			"CONNECT {pass:s3cr3t ,   password =  s3cr3t}",
+			"CONNECT {pass:[REDACTED],   password =  [REDACTED]}",
+		},
+		{
+			"invalid json no whitespace after key",
+			"CONNECT {pass:s3cr3t ,   password=  s3cr3t}",
+			"CONNECT {pass:[REDACTED],   password=  [REDACTED]}",
+		},
+		{
+			"both pass and wrong password key",
+			`CONNECT {"pass":"s3cr3t4", "password": "s3cr3t4"}`,
+			`CONNECT {"pass":"[REDACTED]", "password": "[REDACTED]"}`,
+		},
+		{
+			"invalid json",
+			"CONNECT {user = hello, password =  s3cr3t}",
+			"CONNECT {user = hello, password =  [REDACTED]}",
+		},
+		{
+			"complete connect",
+			"CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"foo\",\"pass\":\"s3cr3t\",\"tls_required\":false,\"name\":\"APM7JU94z77YzP6WTBEiuw\"}\r\n",
+			"CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"foo\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"name\":\"APM7JU94z77YzP6WTBEiuw\"}\r\n",
+		},
+		{
+			"invalid json with only pass key",
+			"CONNECT {pass:s3cr3t\r\n",
+			"CONNECT {pass:[REDACTED]\r\n",
+		},
+		{
+			"invalid password key also filtered",
+			"CONNECT {\"password\":\"s3cr3t\",}\r\n",
+			"CONNECT {\"password\":\"[REDACTED]\",}\r\n",
+		},
+		{
+			"single long password with whitespace",
+			"CONNECT {\"pass\":\"secret password which is very long\",}\r\n",
+			"CONNECT {\"pass\":\"[REDACTED]\",}\r\n",
+		},
+		{
+			"single long pass key is filtered",
+			"CONNECT {\"pass\":\"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\"}\r\n",
+			"CONNECT {\"pass\":\"[REDACTED]\"}\r\n",
+		},
+		{
+			"duplicate keys all filtered",
+			"CONNECT {\"pass\":\"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\",\"pass\":\"BBBBBBBBBBBBBBBBBBBB\",\"password\":\"CCCCCCCCCCCCCCCC\"}\r\n",
+			"CONNECT {\"pass\":\"[REDACTED]\",\"pass\":\"[REDACTED]\",\"password\":\"[REDACTED]\"}\r\n",
+		},
+		{
+			"invalid json with multiple keys all filtered",
+			"CONNECT {pass = \"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\",pass= \"BBBBBBBBBBBBBBBBBBBB\",password =\"CCCCCCCCCCCCCCCC\"}\r\n",
+			"CONNECT {pass = \"[REDACTED]\",pass= \"[REDACTED]\",password =\"[REDACTED]\"}\r\n",
+		},
+		{
+			"complete connect protocol",
+			"CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"foo\",\"pass\":\"s3cr3t\",\"tls_required\":false,\"name\":\"APM7JU94z77YzP6WTBEiuw\"}\r\n",
+			"CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"foo\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"name\":\"APM7JU94z77YzP6WTBEiuw\"}\r\n",
+		},
+		{
+			"user and pass are filterered",
+			"CONNECT {\"user\":\"s3cr3t\",\"pass\":\"s3cr3t\"}\r\n",
+			"CONNECT {\"user\":\"s3cr3t\",\"pass\":\"[REDACTED]\"}\r\n",
+		},
+		{
+			"complete connect using password key with user and password being the same",
+			"CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"s3cr3t\",\"pass\":\"s3cr3t\",\"tls_required\":false,\"name\":\"...\"}\r\n",
+			"CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"s3cr3t\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"name\":\"...\"}\r\n",
+		},
+		{
+			"complete connect with user password and name all the same",
+			"CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"s3cr3t\",\"pass\":\"s3cr3t\",\"tls_required\":false,\"name\":\"s3cr3t\"}\r\n",
+			"CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"s3cr3t\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"name\":\"s3cr3t\"}\r\n",
+		},
+		{
+			"complete connect extra white space at the beginning",
+			"CONNECT 	 {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"s3cr3t\",\"pass\":\"s3cr3t\",\"tls_required\":false,\"name\":\"foo\"}\r\n",
+			"CONNECT 	 {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"s3cr3t\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"name\":\"foo\"}\r\n",
+		},
+	}
+
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			output := removeSecretsFromTrace([]byte(test.input))
+			if !bytes.Equal(output, []byte(test.expected)) {
+				t.Errorf("\nExpected %q\n    got: %q", test.expected, string(output))
+			}
+		})
+	}
+}
+
+func TestRemoveAuthTokenFromTrace(t *testing.T) {
+	tests := []struct {
+		name     string
+		input    string
+		expected string
+	}{
+		{
+			"user and auth_token",
+			"CONNECT {\"user\":\"derek\",\"auth_token\":\"s3cr3t\"}\r\n",
+			"CONNECT {\"user\":\"derek\",\"auth_token\":\"[REDACTED]\"}\r\n",
+		},
+		{
+			"user and pass extra space",
+			"CONNECT {\"user\":\"derek\",\"auth_token\":  \"s3cr3t\"}\r\n",
+			"CONNECT {\"user\":\"derek\",\"auth_token\":  \"[REDACTED]\"}\r\n",
+		},
+		{
+			"user and pass is empty",
+			"CONNECT {\"user\":\"derek\",\"auth_token\":\"\"}\r\n",
+			"CONNECT {\"user\":\"derek\",\"auth_token\":\"[REDACTED]\"}\r\n",
+		},
+		{
+			"user and pass is empty whitespace",
+			"CONNECT {\"user\":\"derek\",\"auth_token\":\"               \"}\r\n",
+			"CONNECT {\"user\":\"derek\",\"auth_token\":\"[REDACTED]\"}\r\n",
+		},
+		{
+			"user and pass whitespace",
+			"CONNECT {\"user\":\"derek\",\"auth_token\":    \"s3cr3t\"     }\r\n",
+			"CONNECT {\"user\":\"derek\",\"auth_token\":    \"[REDACTED]\"     }\r\n",
+		},
+		{
+			"only pass",
+			"CONNECT {\"auth_token\":\"s3cr3t\",}\r\n",
+			"CONNECT {\"auth_token\":\"[REDACTED]\",}\r\n",
+		},
+		{
+			"invalid json",
+			"CONNECT {auth_token:s3cr3t ,   password =  s3cr3t}",
+			"CONNECT {auth_token:[REDACTED],   password =  [REDACTED]}",
+		},
+		{
+			"invalid json no whitespace after key",
+			"CONNECT {auth_token:s3cr3t ,   password=  s3cr3t}",
+			"CONNECT {auth_token:[REDACTED],   password=  [REDACTED]}",
+		},
+		{
+			"both pass and wrong password key",
+			`CONNECT {"auth_token":"s3cr3t4", "password": "s3cr3t4"}`,
+			`CONNECT {"auth_token":"[REDACTED]", "password": "[REDACTED]"}`,
+		},
+		{
+			"invalid json",
+			"CONNECT {user = hello, auth_token =  s3cr3t}",
+			"CONNECT {user = hello, auth_token =  [REDACTED]}",
+		},
+		{
+			"complete connect",
+			"CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"auth_token\":\"s3cr3t\",\"tls_required\":false,\"name\":\"APM7JU94z77YzP6WTBEiuw\"}\r\n",
+			"CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"auth_token\":\"[REDACTED]\",\"tls_required\":false,\"name\":\"APM7JU94z77YzP6WTBEiuw\"}\r\n",
+		},
+		{
+			"invalid json with only pass key",
+			"CONNECT {auth_token:s3cr3t\r\n",
+			"CONNECT {auth_token:[REDACTED]\r\n",
+		},
+		{
+			"invalid password key also filtered",
+			"CONNECT {\"auth_token\":\"s3cr3t\",}\r\n",
+			"CONNECT {\"auth_token\":\"[REDACTED]\",}\r\n",
+		},
+		{
+			"single long password with whitespace",
+			"CONNECT {\"auth_token\":\"secret password which is very long\",}\r\n",
+			"CONNECT {\"auth_token\":\"[REDACTED]\",}\r\n",
+		},
+		{
+			"single long pass key is filtered",
+			"CONNECT {\"auth_token\":\"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\"}\r\n",
+			"CONNECT {\"auth_token\":\"[REDACTED]\"}\r\n",
+		},
+		{
+			"duplicate keys all filtered",
+			"CONNECT {\"auth_token\":\"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\",\"auth_token\":\"BBB\",\"pass\":\"BBBBBBBBBBBBBBBBBBBB\",\"password\":\"CCCCCCCCCCCCCCCC\"}\r\n",
+			"CONNECT {\"auth_token\":\"[REDACTED]\",\"auth_token\":\"[REDACTED]\",\"pass\":\"[REDACTED]\",\"password\":\"[REDACTED]\"}\r\n",
+		},
+		{
+			"invalid json with multiple keys all filtered",
+			"CONNECT {auth_token = \"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\",pass= \"BBBBBBBBBBBBBBBBBBBB\",password =\"CCCCCCCCCCCCCCCC\"}\r\n",
+			"CONNECT {auth_token = \"[REDACTED]\",pass= \"[REDACTED]\",password =\"[REDACTED]\"}\r\n",
+		},
+		{
+			"complete connect protocol",
+			"CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"foo\",\"auth_token\":\"s3cr3t\",\"tls_required\":false,\"name\":\"APM7JU94z77YzP6WTBEiuw\"}\r\n",
+			"CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"foo\",\"auth_token\":\"[REDACTED]\",\"tls_required\":false,\"name\":\"APM7JU94z77YzP6WTBEiuw\"}\r\n",
+		},
+		{
+			"user and token are filterered",
+			"CONNECT {\"user\":\"s3cr3t\",\"auth_token\":\"s3cr3t\"}\r\n",
+			"CONNECT {\"user\":\"s3cr3t\",\"auth_token\":\"[REDACTED]\"}\r\n",
+		},
+		{
+			"complete connect using token key with user and token being the same",
+			"CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"s3cr3t\",\"auth_token\":\"s3cr3t\",\"tls_required\":false,\"name\":\"...\"}\r\n",
+			"CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"s3cr3t\",\"auth_token\":\"[REDACTED]\",\"tls_required\":false,\"name\":\"...\"}\r\n",
+		},
+		{
+			"complete connect with user, token and name all the same",
+			"CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"s3cr3t\",\"auth_token\":\"s3cr3t\",\"tls_required\":false,\"name\":\"s3cr3t\"}\r\n",
+			"CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"s3cr3t\",\"auth_token\":\"[REDACTED]\",\"tls_required\":false,\"name\":\"s3cr3t\"}\r\n",
+		},
+		{
+			"complete connect extra white space at the beginning",
+			"CONNECT 	 {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"s3cr3t\",\"auth_token\":\"s3cr3t\",\"tls_required\":false,\"name\":\"foo\"}\r\n",
+			"CONNECT 	 {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"s3cr3t\",\"auth_token\":\"[REDACTED]\",\"tls_required\":false,\"name\":\"foo\"}\r\n",
+		},
+	}
+
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			output := removeSecretsFromTrace([]byte(test.input))
+			if !bytes.Equal(output, []byte(test.expected)) {
+				t.Errorf("\nExpected %q\n    got: %q", test.expected, string(output))
+			}
+		})
+	}
+}
+
+func TestRemoveAdditionalAuthFieldsFromTrace(t *testing.T) {
+	tests := []struct {
+		name     string
+		input    string
+		expected string
+	}{
+		{
+			name:     "sig",
+			input:    "CONNECT {\"sig\":\"signature\",\"user\":\"foo\"}\r\n",
+			expected: "CONNECT {\"sig\":\"[REDACTED]\",\"user\":\"foo\"}\r\n",
+		},
+		{
+			name:     "nkey",
+			input:    "CONNECT {\"nkey\":\"nkey\",\"user\":\"foo\"}\r\n",
+			expected: "CONNECT {\"nkey\":\"[REDACTED]\",\"user\":\"foo\"}\r\n",
+		},
+		{
+			name:     "proxy sig",
+			input:    "CONNECT {\"proxy_sig\":\"proxy-signature\",\"user\":\"foo\"}\r\n",
+			expected: "CONNECT {\"proxy_sig\":\"[REDACTED]\",\"user\":\"foo\"}\r\n",
+		},
+		{
+			name:     "proxy sig followed by sig",
+			input:    "CONNECT {\"proxy_sig\":\"proxy-signature\",\"sig\":\"sig\",\"user\":\"foo\"}\r\n",
+			expected: "CONNECT {\"proxy_sig\":\"[REDACTED]\",\"sig\":\"[REDACTED]\",\"user\":\"foo\"}\r\n",
+		},
+		{
+			name:     "sig followed by proxy sig",
+			input:    "CONNECT {\"sig\":\"sig\",\"proxy_sig\":\"proxy-signature\",\"user\":\"foo\"}\r\n",
+			expected: "CONNECT {\"sig\":\"[REDACTED]\",\"proxy_sig\":\"[REDACTED]\",\"user\":\"foo\"}\r\n",
+		},
+		{
+			name:     "sig marker inside another string still redacts real sig",
+			input:    "CONNECT {\"name\":\"foo sig:bar\",\"sig\":\"REAL\",\"user\":\"foo\"}\r\n",
+			expected: "CONNECT {\"name\":\"foo sig:[REDACTED]\",\"sig\":\"[REDACTED]\",\"user\":\"foo\"}\r\n",
+		},
+		{
+			name:     "malformed sig without object delimiters",
+			input:    "CONNECT sig=abc\r\n",
+			expected: "CONNECT sig=[REDACTED]\r\n",
+		},
+		{
+			name:     "sig after malformed quoted field is redacted",
+			input:    "CONNECT {\"name\":\"unterminated,\"sig\":\"abc\"}\r\n",
+			expected: "CONNECT {\"name\":\"unterminated,\"sig\":\"[REDACTED]\"}\r\n",
+		},
+	}
+
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			output := removeSecretsFromTrace([]byte(test.input))
+			if !bytes.Equal(output, []byte(test.expected)) {
+				t.Errorf("\nExpected %q\n    got: %q", test.expected, string(output))
+			}
+		})
+	}
+}

server/monitor_sort_opts.go

@@ -0,0 +1,156 @@
+// Copyright 2013-2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"time"
+)
+
+// ConnInfos represents a connection info list. We use pointers since it will be sorted.
+type ConnInfos []*ConnInfo
+
+// For sorting
+// Len returns length for sorting.
+func (cl ConnInfos) Len() int { return len(cl) }
+
+// Swap will sawap the elements.
+func (cl ConnInfos) Swap(i, j int) { cl[i], cl[j] = cl[j], cl[i] }
+
+// SortOpt is a helper type to sort clients
+type SortOpt string
+
+// Possible sort options
+const (
+	ByCid      SortOpt = "cid"        // By connection ID
+	ByStart    SortOpt = "start"      // By connection start time, same as CID
+	BySubs     SortOpt = "subs"       // By number of subscriptions
+	ByPending  SortOpt = "pending"    // By amount of data in bytes waiting to be sent to client
+	ByOutMsgs  SortOpt = "msgs_to"    // By number of messages sent
+	ByInMsgs   SortOpt = "msgs_from"  // By number of messages received
+	ByOutBytes SortOpt = "bytes_to"   // By amount of bytes sent
+	ByInBytes  SortOpt = "bytes_from" // By amount of bytes received
+	ByLast     SortOpt = "last"       // By the last activity
+	ByIdle     SortOpt = "idle"       // By the amount of inactivity
+	ByUptime   SortOpt = "uptime"     // By the amount of time connections exist
+	ByStop     SortOpt = "stop"       // By the stop time for a closed connection
+	ByReason   SortOpt = "reason"     // By the reason for a closed connection
+	ByRTT      SortOpt = "rtt"        // By the round trip time
+)
+
+// Individual sort options provide the Less for sort.Interface. Len and Swap are on cList.
+// CID
+type SortByCid struct{ ConnInfos }
+
+func (l SortByCid) Less(i, j int) bool { return l.ConnInfos[i].Cid < l.ConnInfos[j].Cid }
+
+// Number of Subscriptions
+type SortBySubs struct{ ConnInfos }
+
+func (l SortBySubs) Less(i, j int) bool { return l.ConnInfos[i].NumSubs < l.ConnInfos[j].NumSubs }
+
+// Pending Bytes
+type SortByPending struct{ ConnInfos }
+
+func (l SortByPending) Less(i, j int) bool { return l.ConnInfos[i].Pending < l.ConnInfos[j].Pending }
+
+// Outbound Msgs
+type SortByOutMsgs struct{ ConnInfos }
+
+func (l SortByOutMsgs) Less(i, j int) bool { return l.ConnInfos[i].OutMsgs < l.ConnInfos[j].OutMsgs }
+
+// Inbound Msgs
+type SortByInMsgs struct{ ConnInfos }
+
+func (l SortByInMsgs) Less(i, j int) bool { return l.ConnInfos[i].InMsgs < l.ConnInfos[j].InMsgs }
+
+// Outbound Bytes
+type SortByOutBytes struct{ ConnInfos }
+
+func (l SortByOutBytes) Less(i, j int) bool { return l.ConnInfos[i].OutBytes < l.ConnInfos[j].OutBytes }
+
+// Inbound Bytes
+type SortByInBytes struct{ ConnInfos }
+
+func (l SortByInBytes) Less(i, j int) bool { return l.ConnInfos[i].InBytes < l.ConnInfos[j].InBytes }
+
+// Last Activity
+type SortByLast struct{ ConnInfos }
+
+func (l SortByLast) Less(i, j int) bool {
+	return l.ConnInfos[i].LastActivity.UnixNano() < l.ConnInfos[j].LastActivity.UnixNano()
+}
+
+// Idle time
+type SortByIdle struct {
+	ConnInfos
+	now time.Time
+}
+
+func (l SortByIdle) Less(i, j int) bool {
+	return l.now.Sub(l.ConnInfos[i].LastActivity) < l.now.Sub(l.ConnInfos[j].LastActivity)
+}
+
+// Uptime
+type SortByUptime struct {
+	ConnInfos
+	now time.Time
+}
+
+func (l SortByUptime) Less(i, j int) bool {
+	ci := l.ConnInfos[i]
+	cj := l.ConnInfos[j]
+	var upi, upj time.Duration
+	if ci.Stop == nil || ci.Stop.IsZero() {
+		upi = l.now.Sub(ci.Start)
+	} else {
+		upi = ci.Stop.Sub(ci.Start)
+	}
+	if cj.Stop == nil || cj.Stop.IsZero() {
+		upj = l.now.Sub(cj.Start)
+	} else {
+		upj = cj.Stop.Sub(cj.Start)
+	}
+	return upi < upj
+}
+
+// Stop
+type SortByStop struct{ ConnInfos }
+
+func (l SortByStop) Less(i, j int) bool {
+	ciStop := l.ConnInfos[i].Stop
+	cjStop := l.ConnInfos[j].Stop
+	return ciStop.Before(*cjStop)
+}
+
+// Reason
+type SortByReason struct{ ConnInfos }
+
+func (l SortByReason) Less(i, j int) bool {
+	return l.ConnInfos[i].Reason < l.ConnInfos[j].Reason
+}
+
+// RTT - Default is descending
+type SortByRTT struct{ ConnInfos }
+
+func (l SortByRTT) Less(i, j int) bool { return l.ConnInfos[i].rtt < l.ConnInfos[j].rtt }
+
+// IsValid determines if a sort option is valid
+func (s SortOpt) IsValid() bool {
+	switch s {
+	case _EMPTY_, ByCid, ByStart, BySubs, ByPending, ByOutMsgs, ByInMsgs, ByOutBytes, ByInBytes, ByLast, ByIdle, ByUptime, ByStop, ByReason, ByRTT:
+		return true
+	default:
+		return false
+	}
+}

server/mqtt_ex_bench_test.go

@@ -0,0 +1,353 @@
+// Copyright 2024-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build !skip_mqtt_tests
+
+package server
+
+import (
+	"fmt"
+	"strconv"
+	"testing"
+	"time"
+)
+
+const (
+	KB = 1024
+)
+
+type mqttBenchMatrix struct {
+	QOS         []int
+	MessageSize []int
+	Topics      []int
+	Publishers  []int
+	Subscribers []int
+}
+
+type mqttBenchContext struct {
+	QOS         int
+	MessageSize int
+	Topics      int
+	Publishers  int
+	Subscribers int
+
+	Host string
+	Port int
+}
+
+var mqttBenchDefaultMatrix = mqttBenchMatrix{
+	QOS:         []int{0, 1, 2},
+	MessageSize: []int{100, 1 * KB, 10 * KB},
+	Topics:      []int{100},
+	Publishers:  []int{1},
+	Subscribers: []int{1},
+}
+
+type MQTTBenchmarkResult struct {
+	Ops   int                      `json:"ops"`
+	NS    map[string]time.Duration `json:"ns"`
+	Bytes int64                    `json:"bytes"`
+}
+
+func BenchmarkXMQTT(b *testing.B) {
+	if mqttTestCommandPath == "" {
+		b.Skip(`"mqtt-test" command is not found in $PATH.`)
+	}
+
+	bc := mqttBenchContext{}
+	b.Run("Server", func(b *testing.B) {
+		b.Cleanup(bc.startServer(b, false))
+		bc.runAll(b)
+	})
+
+	b.Run("Cluster", func(b *testing.B) {
+		b.Cleanup(bc.startCluster(b, false))
+		bc.runAll(b)
+	})
+
+	b.Run("Server-no-RMSCache", func(b *testing.B) {
+		b.Cleanup(bc.startServer(b, true))
+
+		bc.benchmarkSubRet(b)
+	})
+
+	b.Run("Cluster-no-RMSCache", func(b *testing.B) {
+		b.Cleanup(bc.startCluster(b, true))
+
+		bc.benchmarkSubRet(b)
+	})
+}
+
+func (bc mqttBenchContext) runAll(b *testing.B) {
+	bc.benchmarkPub(b)
+	bc.benchmarkPubPipelined(b)
+	bc.benchmarkPubRetained(b)
+	bc.benchmarkPubSub(b)
+	bc.benchmarkSubRet(b)
+}
+
+// makes a copy of bc
+func (bc mqttBenchContext) benchmarkPub(b *testing.B) {
+	m := mqttBenchDefaultMatrix.
+		NoSubscribers().
+		NoTopics()
+
+	b.Run("PUB", func(b *testing.B) {
+		m.runMatrix(b, bc, func(b *testing.B, bc *mqttBenchContext) {
+			bc.runAndReport(b, "pub",
+				"--qos", strconv.Itoa(bc.QOS),
+				"--messages", strconv.Itoa(b.N),
+				"--size", strconv.Itoa(bc.MessageSize),
+				"--publishers", strconv.Itoa(bc.Publishers),
+				"--mps", "0", // no rate throttle
+			)
+		})
+	})
+}
+
+// makes a copy of bc
+//
+// Like PUB, but with up to 256 messages in flight: measures per-connection
+// throughput rather than round-trip latency.
+func (bc mqttBenchContext) benchmarkPubPipelined(b *testing.B) {
+	m := mqttBenchDefaultMatrix.
+		NoSubscribers().
+		NoTopics().
+		QOS1Only()
+
+	b.Run("PUBX", func(b *testing.B) {
+		m.runMatrix(b, bc, func(b *testing.B, bc *mqttBenchContext) {
+			bc.runAndReport(b, "pub",
+				"--qos", strconv.Itoa(bc.QOS),
+				"--messages", strconv.Itoa(b.N),
+				"--size", strconv.Itoa(bc.MessageSize),
+				"--publishers", strconv.Itoa(bc.Publishers),
+				"--mps", "0", // no rate throttle
+				"--pipeline", "256",
+			)
+		})
+	})
+}
+
+// makes a copy of bc
+func (bc mqttBenchContext) benchmarkPubRetained(b *testing.B) {
+	// This bench is meaningless for QOS0 since the client considers the message
+	// sent as soon as it's written out. It is also useless for QOS2 since the
+	// flow takes a lot longer, and the difference of publishing as retained or
+	// not is lost in the noise.
+	m := mqttBenchDefaultMatrix.
+		NoSubscribers().
+		NoTopics().
+		QOS1Only()
+
+	b.Run("PUBRET", func(b *testing.B) {
+		m.runMatrix(b, bc, func(b *testing.B, bc *mqttBenchContext) {
+			bc.runAndReport(b, "pub", "--retain",
+				"--qos", strconv.Itoa(bc.QOS),
+				"--messages", strconv.Itoa(b.N),
+				"--size", strconv.Itoa(bc.MessageSize),
+				"--publishers", strconv.Itoa(bc.Publishers),
+			)
+		})
+	})
+}
+
+// makes a copy of bc
+func (bc mqttBenchContext) benchmarkPubSub(b *testing.B) {
+	// This test uses a single built-in topic, and a built-in publisher, so no
+	// reason to run it for topics and publishers.
+	m := mqttBenchDefaultMatrix.
+		NoTopics().
+		NoPublishers()
+
+	b.Run("PUBSUB", func(b *testing.B) {
+		m.runMatrix(b, bc, func(b *testing.B, bc *mqttBenchContext) {
+			bc.runAndReport(b, "pubsub",
+				"--qos", strconv.Itoa(bc.QOS),
+				"--messages", strconv.Itoa(b.N),
+				"--size", strconv.Itoa(bc.MessageSize),
+				"--subscribers", strconv.Itoa(bc.Subscribers),
+			)
+		})
+	})
+}
+
+// makes a copy of bc
+func (bc mqttBenchContext) benchmarkSubRet(b *testing.B) {
+	// This test uses a built-in publisher, and it makes most sense to measure
+	// the retained message delivery "overhead" on a QoS0 subscription; without
+	// the extra time involved in actually subscribing.
+	m := mqttBenchDefaultMatrix.
+		NoPublishers().
+		QOS0Only()
+
+	b.Run("SUBRET", func(b *testing.B) {
+		m.runMatrix(b, bc, func(b *testing.B, bc *mqttBenchContext) {
+			bc.runAndReport(b, "subret",
+				"--qos", strconv.Itoa(bc.QOS),
+				"--topics", strconv.Itoa(bc.Topics), // number of retained messages
+				"--subscribers", strconv.Itoa(bc.Subscribers),
+				"--size", strconv.Itoa(bc.MessageSize),
+				"--repeat", strconv.Itoa(b.N), // number of subscribe requests
+			)
+		})
+	})
+}
+
+func (bc mqttBenchContext) runAndReport(b *testing.B, name string, extraArgs ...string) {
+	b.Helper()
+	r := mqttRunExCommandTest(b, name, mqttNewDial("", "", bc.Host, bc.Port, ""), extraArgs...)
+	r.report(b)
+}
+
+func (bc *mqttBenchContext) startServer(b *testing.B, disableRMSCache bool) func() {
+	b.Helper()
+	b.StopTimer()
+	o := testMQTTDefaultOptions()
+	s := testMQTTRunServer(b, o)
+
+	o = s.getOpts()
+	bc.Host = o.MQTT.Host
+	bc.Port = o.MQTT.Port
+	mqttInitTestServer(b, mqttNewDial("", "", bc.Host, bc.Port, ""))
+	return func() {
+		testMQTTShutdownServer(s)
+	}
+}
+
+func (bc *mqttBenchContext) startCluster(b *testing.B, disableRMSCache bool) func() {
+	b.Helper()
+	b.StopTimer()
+	conf := `
+		listen: 127.0.0.1:-1
+		server_name: %s
+		jetstream: {max_mem_store: 256MB, max_file_store: 2GB, store_dir: '%s'}
+
+		cluster {
+			name: %s
+			listen: 127.0.0.1:%d
+			routes = [%s]
+		}
+
+		mqtt {
+			listen: 127.0.0.1:-1
+			stream_replicas: 3
+		}
+
+		# For access to system account.
+		accounts { $SYS { users = [ { user: "admin", pass: "s3cr3t!" } ] } }
+	`
+
+	cl := createJetStreamClusterWithTemplate(b, conf, "MQTT", 3)
+	o := cl.randomNonLeader().getOpts()
+	bc.Host = o.MQTT.Host
+	bc.Port = o.MQTT.Port
+	mqttInitTestServer(b, mqttNewDial("", "", bc.Host, bc.Port, ""))
+	return func() {
+		cl.shutdown()
+	}
+}
+
+func mqttBenchWrapForMatrixField(
+	vFieldPtr *int,
+	arr []int,
+	f func(b *testing.B, bc *mqttBenchContext),
+	namef func(int) string,
+) func(b *testing.B, bc *mqttBenchContext) {
+	if len(arr) == 0 {
+		return f
+	}
+	return func(b *testing.B, bc *mqttBenchContext) {
+		for _, value := range arr {
+			*vFieldPtr = value
+			b.Run(namef(value), func(b *testing.B) {
+				f(b, bc)
+			})
+		}
+	}
+}
+
+func (m mqttBenchMatrix) runMatrix(b *testing.B, bc mqttBenchContext, f func(*testing.B, *mqttBenchContext)) {
+	b.Helper()
+	f = mqttBenchWrapForMatrixField(&bc.MessageSize, m.MessageSize, f, func(size int) string {
+		return sizeKB(size)
+	})
+	f = mqttBenchWrapForMatrixField(&bc.Topics, m.Topics, f, func(n int) string {
+		return fmt.Sprintf("%dtopics", n)
+	})
+	f = mqttBenchWrapForMatrixField(&bc.Publishers, m.Publishers, f, func(n int) string {
+		return fmt.Sprintf("%dpubc", n)
+	})
+	f = mqttBenchWrapForMatrixField(&bc.Subscribers, m.Subscribers, f, func(n int) string {
+		return fmt.Sprintf("%dsubc", n)
+	})
+	f = mqttBenchWrapForMatrixField(&bc.QOS, m.QOS, f, func(qos int) string {
+		return fmt.Sprintf("QOS%d", qos)
+	})
+	b.ResetTimer()
+	b.StartTimer()
+	f(b, &bc)
+}
+
+func (m mqttBenchMatrix) NoSubscribers() mqttBenchMatrix {
+	m.Subscribers = nil
+	return m
+}
+
+func (m mqttBenchMatrix) NoTopics() mqttBenchMatrix {
+	m.Topics = nil
+	return m
+}
+
+func (m mqttBenchMatrix) NoPublishers() mqttBenchMatrix {
+	m.Publishers = nil
+	return m
+}
+
+func (m mqttBenchMatrix) QOS0Only() mqttBenchMatrix {
+	m.QOS = []int{0}
+	return m
+}
+
+func (m mqttBenchMatrix) QOS1Only() mqttBenchMatrix {
+	m.QOS = []int{1}
+	return m
+}
+
+func sizeKB(size int) string {
+	unit := "B"
+	N := size
+	if size >= KB {
+		unit = "KB"
+		N = (N + KB/2) / KB
+	}
+	return fmt.Sprintf("%d%s", N, unit)
+}
+
+func (r MQTTBenchmarkResult) report(b *testing.B) {
+	// Disable the default ns metric in favor of custom X_ms/op.
+	b.ReportMetric(0, "ns/op")
+
+	// Disable MB/s since the github benchmarking action misinterprets the sign
+	// of the result (treats it as less is better).
+	b.SetBytes(0)
+	// b.SetBytes(r.Bytes)
+
+	for unit, ns := range r.NS {
+		nsOp := float64(ns) / float64(r.Ops)
+		b.ReportMetric(nsOp/1000000, unit+"_ms/op")
+	}
+	// Diable ReportAllocs() since it confuses the github benchmarking action
+	// with the noise.
+	// b.ReportAllocs()
+}

server/mqtt_ex_test_test.go

@@ -0,0 +1,382 @@
+// Copyright 2024-2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build !skip_mqtt_tests
+
+package server
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"io"
+	"os"
+	"os/exec"
+	"strconv"
+	"strings"
+	"testing"
+
+	"github.com/nats-io/nuid"
+)
+
+type mqttDial string
+
+type mqttTarget struct {
+	singleServers []*Server
+	clusters      []*cluster
+	configs       []mqttTestConfig
+	all           []mqttDial
+}
+
+type mqttTestConfig struct {
+	name string
+	pub  []mqttDial
+	sub  []mqttDial
+}
+
+func TestXMQTTCompliance(t *testing.T) {
+	if mqttCLICommandPath == _EMPTY_ {
+		t.Skip(`"mqtt" command is not found in $PATH nor $MQTT_CLI. See https://hivemq.github.io/mqtt-cli/docs/installation/#debian-package for installation instructions`)
+	}
+
+	o := testMQTTDefaultOptions()
+	s := testMQTTRunServer(t, o)
+	o = s.getOpts()
+	defer testMQTTShutdownServer(s)
+
+	cmd := exec.Command(mqttCLICommandPath, "test", "-V", "3", "-p", strconv.Itoa(o.MQTT.Port))
+
+	output, err := cmd.CombinedOutput()
+	t.Log(string(output))
+	if err != nil {
+		if exitError, ok := err.(*exec.ExitError); ok {
+			t.Fatalf("mqtt cli exited with error: %v", exitError)
+		}
+	}
+}
+
+func TestXMQTTRetainedMessages(t *testing.T) {
+	if mqttTestCommandPath == _EMPTY_ {
+		t.Skip(`"mqtt-test" command is not found in $PATH.`)
+	}
+
+	for _, topo := range []struct {
+		name  string
+		makef func(testing.TB) *mqttTarget
+	}{
+		{
+			name:  "single server",
+			makef: mqttMakeTestServer,
+		},
+		{
+			name:  "cluster",
+			makef: mqttMakeTestCluster(4, ""),
+		},
+	} {
+		t.Run(topo.name, func(t *testing.T) {
+			target := topo.makef(t)
+			t.Cleanup(target.Shutdown)
+
+			// initialize the MQTT assets by "touching" all nodes in the
+			// cluster, but then reload to start with fresh server state.
+			for _, dial := range target.all {
+				mqttInitTestServer(t, dial)
+			}
+
+			numRMS := 100
+			strNumRMS := strconv.Itoa(numRMS)
+			topics := make([]string, len(target.configs))
+
+			for i, tc := range target.configs {
+				// Publish numRMS retained messages one at a time,
+				// round-robin across pub nodes. Remember the topic for each
+				// test config to check the subs after reload.
+				topic := "subret_" + nuid.Next()
+				topics[i] = topic
+				iNode := 0
+				for i := 0; i < numRMS; i++ {
+					pubTopic := fmt.Sprintf("%s/%d", topic, i)
+					dial := tc.pub[iNode%len(tc.pub)]
+					mqttRunExCommandTest(t, "pub", dial,
+						"--retain",
+						"--topic", pubTopic,
+						"--qos", "0",
+						"--size", "128", // message size 128 bytes
+					)
+					iNode++
+				}
+			}
+
+			// Check all sub nodes for retained messages
+			for i, tc := range target.configs {
+				for _, dial := range tc.sub {
+					mqttRunExCommandTest(t, "sub", dial,
+						"--retained", strNumRMS,
+						"--qos", "0",
+						"--topic", topics[i],
+					)
+				}
+			}
+
+			// Reload the target
+			target.Reload(t)
+
+			// Now check again
+			for i, tc := range target.configs {
+				for _, dial := range tc.sub {
+					mqttRunExCommandTestRetry(t, 1, "sub", dial,
+						"--retained", strNumRMS,
+						"--qos", "0",
+						"--topic", topics[i],
+					)
+				}
+			}
+		})
+	}
+}
+
+func mqttInitTestServer(tb testing.TB, dial mqttDial) {
+	tb.Helper()
+	mqttRunExCommandTestRetry(tb, 5, "pub", dial)
+}
+
+func mqttNewDialForServer(s *Server, username, password string) mqttDial {
+	o := s.getOpts().MQTT
+	return mqttNewDial(username, password, o.Host, o.Port, s.Name())
+}
+
+func mqttNewDial(username, password, host string, port int, comment string) mqttDial {
+	d := ""
+	switch {
+	case username != "" && password != "":
+		d = fmt.Sprintf("%s:%s@%s:%d", username, password, host, port)
+	case username != "":
+		d = fmt.Sprintf("%s@%s:%d", username, host, port)
+	default:
+		d = fmt.Sprintf("%s:%d", host, port)
+	}
+	if comment != "" {
+		d += "#" + comment
+	}
+	return mqttDial(d)
+}
+
+func (d mqttDial) Get() (u, p, s, c string) {
+	if d == "" {
+		return "", "", "127.0.0.1:1883", ""
+	}
+	in := string(d)
+	if i := strings.LastIndex(in, "#"); i != -1 {
+		c = in[i+1:]
+		in = in[:i]
+	}
+	if i := strings.LastIndex(in, "@"); i != -1 {
+		up := in[:i]
+		in = in[i+1:]
+		u = up
+		if i := strings.Index(up, ":"); i != -1 {
+			u = up[:i]
+			p = up[i+1:]
+		}
+	}
+	s = in
+	return u, p, s, c
+}
+
+func (d mqttDial) Name() string {
+	_, _, _, c := d.Get()
+	return c
+}
+
+func (t *mqttTarget) Reload(tb testing.TB) {
+	tb.Helper()
+
+	for _, c := range t.clusters {
+		c.stopAll()
+		c.restartAllSamePorts()
+	}
+
+	for i, s := range t.singleServers {
+		o := s.getOpts()
+		s.Shutdown()
+		t.singleServers[i] = testMQTTRunServer(tb, o)
+	}
+
+	for _, dial := range t.all {
+		mqttInitTestServer(tb, dial)
+	}
+}
+
+func (t *mqttTarget) Shutdown() {
+	for _, c := range t.clusters {
+		c.shutdown()
+	}
+	for _, s := range t.singleServers {
+		testMQTTShutdownServer(s)
+	}
+}
+
+func mqttMakeTestServer(tb testing.TB) *mqttTarget {
+	tb.Helper()
+	o := testMQTTDefaultOptions()
+	s := testMQTTRunServer(tb, o)
+	all := []mqttDial{mqttNewDialForServer(s, "", "")}
+	return &mqttTarget{
+		singleServers: []*Server{s},
+		all:           all,
+		configs: []mqttTestConfig{
+			{
+				name: "single server",
+				pub:  all,
+				sub:  all,
+			},
+		},
+	}
+}
+
+func mqttMakeTestCluster(size int, domain string) func(tb testing.TB) *mqttTarget {
+	return func(tb testing.TB) *mqttTarget {
+		tb.Helper()
+		if size < 3 {
+			tb.Fatal("cluster size must be at least 3")
+		}
+
+		if domain != "" {
+			domain = "domain: " + domain + ", "
+		}
+		clusterConf := `
+	listen: 127.0.0.1:-1
+
+	server_name: %s
+	jetstream: {max_mem_store: 256MB, max_file_store: 2GB, ` + domain + `store_dir: '%s'}
+
+	leafnodes {
+		listen: 127.0.0.1:-1
+	}
+
+	cluster {
+		name: %s
+		listen: 127.0.0.1:%d
+		routes = [%s]
+	}
+
+	mqtt {
+		listen: 127.0.0.1:-1
+		stream_replicas: 3
+	}
+
+	accounts {
+		ONE { users = [ { user: "one", pass: "p" } ]; jetstream: enabled }
+		$SYS { users = [ { user: "admin", pass: "s3cr3t!" } ] }
+	}
+`
+		cl := createJetStreamClusterWithTemplate(tb, clusterConf, "MQTT", size)
+		cl.waitOnLeader()
+
+		all := []mqttDial{}
+		for _, s := range cl.servers {
+			all = append(all, mqttNewDialForServer(s, "one", "p"))
+		}
+
+		return &mqttTarget{
+			clusters: []*cluster{cl},
+			all:      all,
+			configs: []mqttTestConfig{
+				{
+					name: "publish to one",
+					pub: []mqttDial{
+						mqttNewDialForServer(cl.randomServer(), "one", "p"),
+					},
+					sub: all,
+				},
+				{
+					name: "publish to all",
+					pub:  all,
+					sub:  all,
+				},
+			},
+		}
+	}
+}
+
+var mqttCLICommandPath = func() string {
+	p := os.Getenv("MQTT_CLI")
+	if p == "" {
+		p, _ = exec.LookPath("mqtt")
+	}
+	return p
+}()
+
+var mqttTestCommandPath = func() string {
+	p, _ := exec.LookPath("mqtt-test")
+	return p
+}()
+
+func mqttRunExCommandTest(tb testing.TB, subCommand string, dial mqttDial, extraArgs ...string) *MQTTBenchmarkResult {
+	tb.Helper()
+	return mqttRunExCommandTestRetry(tb, 1, subCommand, dial, extraArgs...)
+}
+
+func mqttRunExCommandTestRetry(tb testing.TB, n int, subCommand string, dial mqttDial, extraArgs ...string) (r *MQTTBenchmarkResult) {
+	tb.Helper()
+	var err error
+	for i := 0; i < n; i++ {
+		if r, err = mqttTryExCommandTest(tb, subCommand, dial, extraArgs...); err == nil {
+			return r
+		}
+
+		if i < (n - 1) {
+			tb.Logf("failed to %q %s to %q on attempt %v, will retry.", subCommand, extraArgs, dial.Name(), i)
+		} else {
+			tb.Fatal(err)
+		}
+	}
+	return nil
+}
+
+func mqttTryExCommandTest(tb testing.TB, subCommand string, dial mqttDial, extraArgs ...string) (r *MQTTBenchmarkResult, err error) {
+	tb.Helper()
+	if mqttTestCommandPath == "" {
+		tb.Skip(`"mqtt-test" command is not found in $PATH.`)
+	}
+
+	args := []string{subCommand, // "-q",
+		"-s", string(dial),
+	}
+	args = append(args, extraArgs...)
+	cmd := exec.Command(mqttTestCommandPath, args...)
+
+	stdout, err := cmd.StdoutPipe()
+	if err != nil {
+		return nil, fmt.Errorf("error executing %q: %v", cmd.String(), err)
+	}
+	defer stdout.Close()
+	errbuf := bytes.Buffer{}
+	cmd.Stderr = &errbuf
+	if err = cmd.Start(); err != nil {
+		return nil, fmt.Errorf("error executing %q: %v", cmd.String(), err)
+	}
+	out, err := io.ReadAll(stdout)
+	if err != nil {
+		return nil, fmt.Errorf("error executing %q: failed to read output: %v", cmd.String(), err)
+	}
+	if err = cmd.Wait(); err != nil {
+		return nil, fmt.Errorf("error executing %q: %v\n\n%s\n\n%s", cmd.String(), err, string(out), errbuf.String())
+	}
+
+	r = &MQTTBenchmarkResult{}
+	if err := json.Unmarshal(out, r); err != nil {
+		tb.Fatalf("error executing %q: failed to decode output: %v\n\n%s\n\n%s", cmd.String(), err, string(out), errbuf.String())
+	}
+	return r, nil
+}

server/msgtrace.go

@@ -0,0 +1,903 @@
+// Copyright 2024-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"math/rand/v2"
+	"strconv"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"time"
+)
+
+const (
+	MsgTraceDest          = "Nats-Trace-Dest"
+	MsgTraceDestDisabled  = "trace disabled" // This must be an invalid NATS subject
+	MsgTraceHop           = "Nats-Trace-Hop"
+	MsgTraceOriginAccount = "Nats-Trace-Origin-Account"
+	MsgTraceOnly          = "Nats-Trace-Only"
+
+	// External trace header. Note that this header is normally in lower
+	// case (https://www.w3.org/TR/trace-context/#header-name). Vendors
+	// MUST expect the header in any case (upper, lower, mixed), and
+	// SHOULD send the header name in lowercase. We used to change it
+	// to lower case, but no longer do that in 2.14.
+	traceParentHdr = "traceparent"
+)
+
+var (
+	traceDestHdrAsBytes      = stringToBytes(MsgTraceDest)
+	traceDestDisabledAsBytes = stringToBytes(MsgTraceDestDisabled)
+	traceParentHdrAsBytes    = stringToBytes(traceParentHdr)
+	crLFAsBytes              = stringToBytes(CR_LF)
+	dashAsBytes              = stringToBytes("-")
+)
+
+type MsgTraceType string
+
+// Type of message trace events in the MsgTraceEvents list.
+// This is needed to unmarshal the list.
+const (
+	MsgTraceIngressType        = "in"
+	MsgTraceSubjectMappingType = "sm"
+	MsgTraceStreamExportType   = "se"
+	MsgTraceServiceImportType  = "si"
+	MsgTraceJetStreamType      = "js"
+	MsgTraceEgressType         = "eg"
+)
+
+type MsgTraceEvent struct {
+	Server  ServerInfo      `json:"server"`
+	Request MsgTraceRequest `json:"request"`
+	Hops    int             `json:"hops,omitempty"`
+	Events  MsgTraceEvents  `json:"events"`
+}
+
+type MsgTraceRequest struct {
+	// We are not making this an http.Header so that header name case is preserved.
+	Header  map[string][]string `json:"header,omitempty"`
+	MsgSize int                 `json:"msgsize,omitempty"`
+}
+
+type MsgTraceEvents []MsgTrace
+
+type MsgTrace interface {
+	new() MsgTrace
+	typ() MsgTraceType
+}
+
+type MsgTraceBase struct {
+	Type      MsgTraceType `json:"type"`
+	Timestamp time.Time    `json:"ts"`
+}
+
+type MsgTraceIngress struct {
+	MsgTraceBase
+	Kind    int    `json:"kind"`
+	CID     uint64 `json:"cid"`
+	Name    string `json:"name,omitempty"`
+	Account string `json:"acc"`
+	Subject string `json:"subj"`
+	Error   string `json:"error,omitempty"`
+}
+
+type MsgTraceSubjectMapping struct {
+	MsgTraceBase
+	MappedTo string `json:"to"`
+}
+
+type MsgTraceStreamExport struct {
+	MsgTraceBase
+	Account string `json:"acc"`
+	To      string `json:"to"`
+}
+
+type MsgTraceServiceImport struct {
+	MsgTraceBase
+	Account string `json:"acc"`
+	From    string `json:"from"`
+	To      string `json:"to"`
+}
+
+type MsgTraceJetStream struct {
+	MsgTraceBase
+	Stream     string `json:"stream"`
+	Subject    string `json:"subject,omitempty"`
+	NoInterest bool   `json:"nointerest,omitempty"`
+	Error      string `json:"error,omitempty"`
+}
+
+type MsgTraceEgress struct {
+	MsgTraceBase
+	Kind         int    `json:"kind"`
+	CID          uint64 `json:"cid"`
+	Name         string `json:"name,omitempty"`
+	Hop          string `json:"hop,omitempty"`
+	Account      string `json:"acc,omitempty"`
+	Subscription string `json:"sub,omitempty"`
+	Queue        string `json:"queue,omitempty"`
+	Error        string `json:"error,omitempty"`
+
+	// This is for applications that unmarshal the trace events
+	// and want to link an egress to route/leaf/gateway with
+	// the MsgTraceEvent from that server.
+	Link *MsgTraceEvent `json:"-"`
+}
+
+// -------------------------------------------------------------
+
+func (t MsgTraceBase) typ() MsgTraceType     { return t.Type }
+func (MsgTraceIngress) new() MsgTrace        { return &MsgTraceIngress{} }
+func (MsgTraceSubjectMapping) new() MsgTrace { return &MsgTraceSubjectMapping{} }
+func (MsgTraceStreamExport) new() MsgTrace   { return &MsgTraceStreamExport{} }
+func (MsgTraceServiceImport) new() MsgTrace  { return &MsgTraceServiceImport{} }
+func (MsgTraceJetStream) new() MsgTrace      { return &MsgTraceJetStream{} }
+func (MsgTraceEgress) new() MsgTrace         { return &MsgTraceEgress{} }
+
+var msgTraceInterfaces = map[MsgTraceType]MsgTrace{
+	MsgTraceIngressType:        MsgTraceIngress{},
+	MsgTraceSubjectMappingType: MsgTraceSubjectMapping{},
+	MsgTraceStreamExportType:   MsgTraceStreamExport{},
+	MsgTraceServiceImportType:  MsgTraceServiceImport{},
+	MsgTraceJetStreamType:      MsgTraceJetStream{},
+	MsgTraceEgressType:         MsgTraceEgress{},
+}
+
+func (t *MsgTraceEvents) UnmarshalJSON(data []byte) error {
+	var raw []json.RawMessage
+	err := json.Unmarshal(data, &raw)
+	if err != nil {
+		return err
+	}
+	*t = make(MsgTraceEvents, len(raw))
+	var tt MsgTraceBase
+	for i, r := range raw {
+		if err = json.Unmarshal(r, &tt); err != nil {
+			return err
+		}
+		tr, ok := msgTraceInterfaces[tt.Type]
+		if !ok {
+			return fmt.Errorf("unknown trace type %v", tt.Type)
+		}
+		te := tr.new()
+		if err := json.Unmarshal(r, te); err != nil {
+			return err
+		}
+		(*t)[i] = te
+	}
+	return nil
+}
+
+func getTraceAs[T MsgTrace](e any) *T {
+	v, ok := e.(*T)
+	if ok {
+		return v
+	}
+	return nil
+}
+
+func (t *MsgTraceEvent) Ingress() *MsgTraceIngress {
+	if len(t.Events) < 1 {
+		return nil
+	}
+	return getTraceAs[MsgTraceIngress](t.Events[0])
+}
+
+func (t *MsgTraceEvent) SubjectMapping() *MsgTraceSubjectMapping {
+	for _, e := range t.Events {
+		if e.typ() == MsgTraceSubjectMappingType {
+			return getTraceAs[MsgTraceSubjectMapping](e)
+		}
+	}
+	return nil
+}
+
+func (t *MsgTraceEvent) StreamExports() []*MsgTraceStreamExport {
+	var se []*MsgTraceStreamExport
+	for _, e := range t.Events {
+		if e.typ() == MsgTraceStreamExportType {
+			se = append(se, getTraceAs[MsgTraceStreamExport](e))
+		}
+	}
+	return se
+}
+
+func (t *MsgTraceEvent) ServiceImports() []*MsgTraceServiceImport {
+	var si []*MsgTraceServiceImport
+	for _, e := range t.Events {
+		if e.typ() == MsgTraceServiceImportType {
+			si = append(si, getTraceAs[MsgTraceServiceImport](e))
+		}
+	}
+	return si
+}
+
+func (t *MsgTraceEvent) JetStream() *MsgTraceJetStream {
+	for _, e := range t.Events {
+		if e.typ() == MsgTraceJetStreamType {
+			return getTraceAs[MsgTraceJetStream](e)
+		}
+	}
+	return nil
+}
+
+func (t *MsgTraceEvent) Egresses() []*MsgTraceEgress {
+	var eg []*MsgTraceEgress
+	for _, e := range t.Events {
+		if e.typ() == MsgTraceEgressType {
+			eg = append(eg, getTraceAs[MsgTraceEgress](e))
+		}
+	}
+	return eg
+}
+
+const (
+	errMsgTraceOnlyNoSupport   = "Not delivered because remote does not support message tracing"
+	errMsgTraceNoSupport       = "Message delivered but remote does not support message tracing so no trace event generated from there"
+	errMsgTraceNoEcho          = "Not delivered because of no echo"
+	errMsgTracePubViolation    = "Not delivered because publish denied for this subject"
+	errMsgTraceSubDeny         = "Not delivered because subscription denies this subject"
+	errMsgTraceSubClosed       = "Not delivered because subscription is closed"
+	errMsgTraceClientClosed    = "Not delivered because client is closed"
+	errMsgTraceAutoSubExceeded = "Not delivered because auto-unsubscribe exceeded"
+	errMsgTraceFastProdNoStall = "Not delivered because fast producer not stalled and consumer is slow"
+)
+
+type msgTrace struct {
+	kind  int
+	ready int32
+	srv   *Server
+	acc   *Account
+	dest  string
+	event *MsgTraceEvent
+	js    *MsgTraceJetStream
+	siMu  sync.RWMutex
+	si    []*msgTraceServiceImport
+	rsi   *msgTraceRespServiceImport
+	hop   string
+	nhop  string
+	tonly bool // Will only trace the message, not do delivery.
+	ct    compressionType
+}
+
+type msgTraceServiceImport struct {
+	si  *serviceImport
+	acc *Account
+}
+
+type msgTraceRespServiceImport struct {
+	hops     int
+	received int
+	tree     map[string]*msgTraceRespServiceImport
+}
+
+// This will be false outside of the tests, so when building the server binary,
+// any code where you see `if msgTraceRunInTests` statement will be compiled
+// out, so this will have no performance penalty.
+var (
+	msgTraceRunInTests   bool
+	msgTraceCheckSupport bool
+)
+
+// Returns the message trace object, if message is being traced,
+// and `true` if we want to only trace, not actually deliver the message.
+func (c *client) isMsgTraceEnabled() (*msgTrace, bool) {
+	t := c.pa.trace
+	if t == nil {
+		return nil, false
+	}
+	return t, t.tonly
+}
+
+// For LEAF/ROUTER/GATEWAY, return false if the remote does not support
+// message tracing (important if the tracing requests trace-only).
+func (c *client) msgTraceSupport() bool {
+	// Exclude client connection from the protocol check.
+	return c.kind == CLIENT || c.opts.Protocol >= MsgTraceProto
+}
+
+func getConnName(c *client) string {
+	switch c.kind {
+	case ROUTER:
+		if n := c.route.remoteName; n != _EMPTY_ {
+			return n
+		}
+	case GATEWAY:
+		if n := c.gw.remoteName; n != _EMPTY_ {
+			return n
+		}
+	case LEAF:
+		if n := c.leaf.remoteServer; n != _EMPTY_ {
+			return n
+		}
+	}
+	return c.opts.Name
+}
+
+func getCompressionType(cts string) compressionType {
+	if cts == _EMPTY_ {
+		return noCompression
+	}
+	cts = strings.ToLower(cts)
+	if strings.Contains(cts, "snappy") || strings.Contains(cts, "s2") {
+		return snappyCompression
+	}
+	if strings.Contains(cts, "gzip") {
+		return gzipCompression
+	}
+	return unsupportedCompression
+}
+
+// Possibly initialize message tracing if appropriate headers are found in the
+// given `hdr` byte slice.
+// If there is an error and the header `MsgTraceOnly` is set to `true` (indicating
+// that it should be a "trace only" message, without message delivery), then this
+// function will return `true, nil`, so that the caller can skip message processing.
+func (c *client) initMsgTrace(hdr []byte, ingressError error) (bool, *msgTrace) {
+	// The code in the "if" statement is only running in test mode.
+	if msgTraceRunInTests {
+		// Check the type of client that tries to initialize a trace struct.
+		if !(c.kind == CLIENT || c.kind == ROUTER || c.kind == GATEWAY || c.kind == LEAF) {
+			panic(fmt.Sprintf("Unexpected client type %q trying to initialize msgTrace", c.kindString()))
+		}
+		// In some tests, we want to make a server behave like an old server
+		// and so even if a trace header is received, we want the server to
+		// simply ignore it.
+		if msgTraceCheckSupport {
+			if c.srv == nil || c.srv.getServerProto() < MsgTraceProto {
+				return false, nil
+			}
+		}
+	}
+	// Get the headers if we find `Nats-Trace-Dest` or `traceparent` header.
+	// For `traceparent`, `external` will be true indicating that we need
+	// to get the destination and sampling from the account.
+	headers, external := genHeaderMapIfTraceHeadersPresent(hdr)
+	if len(headers) == 0 {
+		return false, nil
+	}
+	// Little helper to give us the first value of a given header, or _EMPTY_
+	// if key is not present.
+	getHdrVal := func(key string) string {
+		vv, ok := headers[key]
+		if !ok {
+			return _EMPTY_
+		}
+		return vv[0]
+	}
+	var (
+		dest      string   // The destination the trace message should be sent to
+		traceOnly bool     // True if this is "trace only" and no message delivery should occur
+		hop       string   // The hop "id", taken from headers only when not from CLIENT
+		kind      = c.kind // The type of connection this originates from
+	)
+	// Check for traceOnly only if not external.
+	if !external {
+		if to := getHdrVal(MsgTraceOnly); to != _EMPTY_ {
+			tos := strings.ToLower(to)
+			switch tos {
+			case "1", "true", "on":
+				traceOnly = true
+			}
+		}
+		dest = getHdrVal(MsgTraceDest)
+		// If no dest, bail out.
+		if dest == _EMPTY_ {
+			return traceOnly, nil
+		}
+	}
+	// Now that we know if there is `traceOnly` set or not, if we were to fail
+	// here by returning `nil` for `*msgTrace`, we will return `true, nil` to
+	// indicate to the caller that it should skip message processing (so that
+	// the failed trace message is not delivered to regular subscriptions).
+
+	// First, disable tracing if the message contains the old MsgTraceOriginAccount header.
+	if v := getHdrVal(MsgTraceOriginAccount); v != _EMPTY_ {
+		c.Errorf("Message tracing disabled because of header %q with value=%q",
+			MsgTraceOriginAccount, v)
+		// Skip this if we are invoked with ingressError, or if we are going to skip
+		// message processing anyway (if `traceOnly` is true).
+		if ingressError == nil && !traceOnly {
+			// Disable tracing by changing the `MsgTraceDest` header to `MsgTraceDestDisabled`.
+			c.msgBuf = c.setHeader(MsgTraceDest, MsgTraceDestDisabled, c.msgBuf)
+		}
+		// If `traceOnly` is false, the message will be delivered and possibly routed,
+		// but it will not be traced. If it is `true`, then there will be no tracing
+		// and the message will not be processed (so no chance to be routed).
+		return traceOnly, nil
+	}
+
+	// Get the account and server.
+	c.mu.Lock()
+	acc := c.acc
+	srv := c.srv
+	c.mu.Unlock()
+	// There should always be a server object, and for CLIENT and LEAF, the
+	// account should be available.
+	if srv == nil || (acc == nil && (kind == CLIENT || kind == LEAF)) {
+		// Make the caller skip message procesing if `traceOnly` is true.
+		return traceOnly, nil
+	}
+	// For non CLIENT connection, get the account and hop header.
+	if kind != CLIENT {
+		hop = getHdrVal(MsgTraceHop)
+		if kind != LEAF {
+			// Lookup the account, if not present, bail out.
+			if acci, ok := c.srv.accounts.Load(string(c.pa.account)); ok {
+				acc = acci.(*Account)
+			} else {
+				c.Errorf("Account %q was not found, won't be able to trace events", c.pa.account)
+				return traceOnly, nil
+			}
+		}
+	}
+	// If external, we need to have the account's trace destination set,
+	// otherwise, we are not enabling tracing.
+	if external {
+		var sampling int
+		dest, sampling = acc.getTraceDestAndSampling()
+		if dest == _EMPTY_ {
+			// No account destination, no tracing for external trace headers.
+			// This is not an error, so return false so that we don't skip
+			// message processing.
+			return false, nil
+		}
+		// Check sampling, but only from origin server.
+		if kind == CLIENT && !sample(sampling) {
+			// Need to disable tracing so that if the message is routed, it won't
+			// trigger a trace there. Skip this if we have an ingress error.
+			if ingressError == nil {
+				c.msgBuf = c.setHeader(MsgTraceDest, MsgTraceDestDisabled, c.msgBuf)
+			}
+			return false, nil
+		}
+	}
+	// Check the destination to see if this is a valid publish subject.
+	if !IsValidPublishSubject(dest) {
+		c.Errorf("Destination %q is not valid, won't be able to trace events", dest)
+		// Return `traceOnly` to cause the caller to skip message processing if
+		// this was supposed to be a trace only message.
+		return traceOnly, nil
+	}
+	// Now that we have a valid `dest`, make sure this connection is allowed
+	// to publish to it.
+	if !c.allowedToPublishOnMsgTraceDest(srv, acc, dest) {
+		// Send the error back only if CLIENT or LEAF, otherwise just log.
+		if kind == CLIENT || kind == LEAF {
+			c.pubPermissionViolation(stringToBytes(dest))
+		} else {
+			c.Errorf("Publish Violation - Subject %q", dest)
+		}
+		// Return `true, nil` to force skipping of message processing.
+		return true, nil
+	}
+	c.pa.trace = &msgTrace{
+		kind: kind,
+		srv:  c.srv,
+		acc:  acc,
+		dest: dest,
+		ct:   getCompressionType(getHdrVal(acceptEncodingHeader)),
+		hop:  hop,
+		event: &MsgTraceEvent{
+			Request: MsgTraceRequest{
+				Header:  headers,
+				MsgSize: c.pa.size,
+			},
+			Events: append(MsgTraceEvents(nil), &MsgTraceIngress{
+				MsgTraceBase: MsgTraceBase{
+					Type:      MsgTraceIngressType,
+					Timestamp: time.Now(),
+				},
+				Kind:    kind,
+				CID:     c.cid,
+				Name:    getConnName(c),
+				Account: acc.GetName(),
+				Subject: string(c.pa.subject),
+			}),
+		},
+		tonly: traceOnly,
+	}
+	// If we are invoked with an ingress error, set it now.
+	if ingressError != nil {
+		c.pa.trace.event.Events[0].(*MsgTraceIngress).Error = ingressError.Error()
+	}
+	return false, c.pa.trace
+}
+
+func (c *client) allowedToPublishOnMsgTraceDest(s *Server, acc *Account, dest string) bool {
+	td := stringToBytes(dest)
+	if hasGWRoutedReplyPrefix(td) {
+		return false
+	}
+	if bytes.HasPrefix(td, clientNRGPrefix) && acc != s.SystemAccount() {
+		return false
+	}
+	allowed := true
+	c.mu.Lock()
+	if c.kind == LEAF {
+		if c.isSpokeLeafNode() {
+			allowed = c.leafReceiveAllowed(td)
+		} else {
+			allowed = c.leafSendAllowed(td)
+		}
+	} else if c.perms != nil && (c.perms.pub.allow != nil || c.perms.pub.deny != nil) && !c.pubAllowedFullCheck(dest, false, true) {
+		allowed = false
+	}
+	c.mu.Unlock()
+	return allowed
+}
+
+func sample(sampling int) bool {
+	// Option parsing should ensure that sampling is [1..100], but consider
+	// any value outside of this range to be 100%.
+	if sampling <= 0 || sampling >= 100 {
+		return true
+	}
+	return rand.Int32N(100) <= int32(sampling)
+}
+
+// This function will return the header as a map (instead of http.Header because
+// we want to preserve the header names' case) and a boolean that indicates if
+// the headers have been lifted due to the presence of the external trace header
+// only.
+// Note that because of the traceParentHdr, the search is done in a case
+// insensitive way. We used to rewrite it in lower case but no longer do since v2.14.
+func genHeaderMapIfTraceHeadersPresent(hdr []byte) (map[string][]string, bool) {
+
+	var (
+		_keys               = [64][]byte{}
+		_vals               = [64][]byte{}
+		m                   map[string][]string
+		traceDestHdrFound   bool
+		traceParentHdrFound bool
+	)
+	// Skip the hdrLine
+	if !bytes.HasPrefix(hdr, stringToBytes(hdrLine)) {
+		return nil, false
+	}
+
+	keys := _keys[:0]
+	vals := _vals[:0]
+
+	for i := len(hdrLine); i < len(hdr); {
+		// Search for key/val delimiter
+		del := bytes.IndexByte(hdr[i:], ':')
+		if del < 0 {
+			break
+		}
+		keyStart := i
+		key := hdr[keyStart : keyStart+del]
+		i += del + 1
+		for i < len(hdr) && (hdr[i] == ' ' || hdr[i] == '\t') {
+			i++
+		}
+		valStart := i
+		nl := bytes.Index(hdr[valStart:], crLFAsBytes)
+		if nl < 0 {
+			break
+		}
+		valEnd := valStart + nl
+		for valEnd > valStart && (hdr[valEnd-1] == ' ' || hdr[valEnd-1] == '\t') {
+			valEnd--
+		}
+		val := hdr[valStart:valEnd]
+		if len(key) > 0 && len(val) > 0 {
+			vals = append(vals, val)
+
+			// We search for our special keys only if not already found.
+
+			// Check for the external trace header.
+			// Search needs to be case insensitive.
+			if !traceParentHdrFound && bytes.EqualFold(key, traceParentHdrAsBytes) {
+				// We will now check if the value has sampling or not.
+				// TODO(ik): Not sure if this header can have multiple values
+				// or not, and if so, what would be the rule to check for
+				// sampling. What is done here is to check them all until we
+				// found one with sampling.
+				tk := bytes.Split(val, dashAsBytes)
+				if len(tk) == 4 && len([]byte(tk[3])) == 2 {
+					if hexVal, err := strconv.ParseInt(bytesToString(tk[3]), 16, 8); err == nil {
+						if hexVal&0x1 == 0x1 {
+							traceParentHdrFound = true
+						}
+					}
+				}
+			} else if !traceDestHdrFound && bytes.Equal(key, traceDestHdrAsBytes) {
+				// This is the Nats-Trace-Dest header, check the value to see
+				// if it indicates that the trace was disabled.
+				if bytes.Equal(val, traceDestDisabledAsBytes) {
+					return nil, false
+				}
+				traceDestHdrFound = true
+			}
+			// Add to the keys and preserve the key's case
+			keys = append(keys, key)
+		}
+		i += nl + 2
+	}
+	if !traceDestHdrFound && !traceParentHdrFound {
+		return nil, false
+	}
+	m = make(map[string][]string, len(keys))
+	for i, k := range keys {
+		hname := string(k)
+		m[hname] = append(m[hname], string(vals[i]))
+	}
+	return m, !traceDestHdrFound && traceParentHdrFound
+}
+
+// Special case where we create a trace event before parsing the message.
+// This is for cases where the connection will be closed when detecting
+// an error during early message processing (for instance max payload).
+func (c *client) sendMsgTraceIngressErrEvent(hdr []byte, ingressError error) {
+	if ingressError == nil {
+		return
+	}
+	if _, t := c.initMsgTrace(hdr, ingressError); t != nil {
+		t.sendEvent()
+	}
+}
+
+// Returns `true` if message tracing is enabled and we are tracing only,
+// that is, we are not going to deliver the inbound message, returns
+// `false` otherwise (no tracing, or tracing and message delivery).
+func (t *msgTrace) traceOnly() bool {
+	return t != nil && t.tonly
+}
+
+func (t *msgTrace) setHopHeader(c *client, msg []byte) []byte {
+	e := t.event
+	e.Hops++
+	if len(t.hop) > 0 {
+		t.nhop = fmt.Sprintf("%s.%d", t.hop, e.Hops)
+	} else {
+		t.nhop = fmt.Sprintf("%d", e.Hops)
+	}
+	if t.kind == CLIENT && strings.HasPrefix(t.dest, replyPrefix) {
+		t.siMu.Lock()
+		if t.rsi == nil {
+			t.rsi = &msgTraceRespServiceImport{}
+		}
+		t.rsi.hops++
+		t.siMu.Unlock()
+	}
+	return c.setHeader(MsgTraceHop, t.nhop, msg)
+}
+
+func (t *msgTrace) setIngressError(err string) {
+	if i := t.event.Ingress(); i != nil {
+		i.Error = err
+	}
+}
+
+func (t *msgTrace) addSubjectMappingEvent(subj []byte) {
+	if t == nil {
+		return
+	}
+	t.event.Events = append(t.event.Events, &MsgTraceSubjectMapping{
+		MsgTraceBase: MsgTraceBase{
+			Type:      MsgTraceSubjectMappingType,
+			Timestamp: time.Now(),
+		},
+		MappedTo: string(subj),
+	})
+}
+
+func (t *msgTrace) addEgressEvent(dc *client, sub *subscription, err string) {
+	if t == nil {
+		return
+	}
+	e := &MsgTraceEgress{
+		MsgTraceBase: MsgTraceBase{
+			Type:      MsgTraceEgressType,
+			Timestamp: time.Now(),
+		},
+		Kind:  dc.kind,
+		CID:   dc.cid,
+		Name:  getConnName(dc),
+		Hop:   t.nhop,
+		Error: err,
+	}
+	t.nhop = _EMPTY_
+	// Specific to CLIENT connections...
+	if dc.kind == CLIENT {
+		// Set the subscription's subject and possibly queue name.
+		e.Subscription = string(sub.subject)
+		if len(sub.queue) > 0 {
+			e.Queue = string(sub.queue)
+		}
+	}
+	if dc.kind == CLIENT || dc.kind == LEAF {
+		if i := t.event.Ingress(); i != nil {
+			// If the Ingress' account is different from the destination's
+			// account, add the account name into the Egress trace event.
+			// This would happen with service imports.
+			if dcAccName := dc.acc.GetName(); dcAccName != i.Account {
+				e.Account = dcAccName
+			}
+		}
+	}
+	t.event.Events = append(t.event.Events, e)
+}
+
+func (t *msgTrace) addStreamExportEvent(dc *client, to []byte) {
+	if t == nil {
+		return
+	}
+	dc.mu.Lock()
+	accName := dc.acc.GetName()
+	dc.mu.Unlock()
+	t.event.Events = append(t.event.Events, &MsgTraceStreamExport{
+		MsgTraceBase: MsgTraceBase{
+			Type:      MsgTraceStreamExportType,
+			Timestamp: time.Now(),
+		},
+		Account: accName,
+		To:      string(to),
+	})
+}
+
+func (t *msgTrace) addServiceImportEvent(accName, from, to string) {
+	if t == nil {
+		return
+	}
+	t.event.Events = append(t.event.Events, &MsgTraceServiceImport{
+		MsgTraceBase: MsgTraceBase{
+			Type:      MsgTraceServiceImportType,
+			Timestamp: time.Now(),
+		},
+		Account: accName,
+		From:    from,
+		To:      to,
+	})
+}
+
+func (t *msgTrace) addJetStreamEvent(streamName string) {
+	if t == nil {
+		return
+	}
+	t.js = &MsgTraceJetStream{
+		MsgTraceBase: MsgTraceBase{
+			Type:      MsgTraceJetStreamType,
+			Timestamp: time.Now(),
+		},
+		Stream: streamName,
+	}
+	t.event.Events = append(t.event.Events, t.js)
+}
+
+func (t *msgTrace) updateJetStreamEvent(subject string, noInterest bool) {
+	if t == nil {
+		return
+	}
+	// JetStream event should have been created in addJetStreamEvent
+	if t.js == nil {
+		return
+	}
+	t.js.Subject = subject
+	t.js.NoInterest = noInterest
+	// Update the timestamp since this is more accurate than when it
+	// was first added in addJetStreamEvent().
+	t.js.Timestamp = time.Now()
+}
+
+func (t *msgTrace) sendEventFromJetStream(err error) {
+	if t == nil {
+		return
+	}
+	// JetStream event should have been created in addJetStreamEvent
+	if t.js == nil {
+		return
+	}
+	if err != nil {
+		t.js.Error = err.Error()
+	}
+	t.sendEvent()
+}
+
+func (t *msgTrace) sendEvent() {
+	if t == nil {
+		return
+	}
+	if t.js != nil {
+		ready := atomic.AddInt32(&t.ready, 1) == 2
+		if !ready {
+			return
+		}
+	}
+	t.srv.sendInternalAccountSysMsg(t.acc, t.dest, &t.event.Server, t.event, t.ct)
+}
+
+func (t *msgTrace) setupResponseServiceImport(c *client, acc *Account, si *serviceImport, msg []byte) (*serviceImport, []byte) {
+	rsi := si.acc.addRespServiceImport(acc, t.dest, si, false, nil, t)
+	t.dest = rsi.from
+	t.siMu.Lock()
+	t.si = append(t.si, &msgTraceServiceImport{rsi, si.acc})
+	t.siMu.Unlock()
+	return rsi, c.setHeader(MsgTraceDest, t.dest, msg)
+}
+
+func (t *msgTrace) handleRespServiceImport(e *MsgTraceEvent) {
+	t.siMu.Lock()
+	defer t.siMu.Unlock()
+
+	// If response service import were created and routed for message traces,
+	// we should have t.rsi created. If it is not, we are done. Note that
+	// if for any reason we bail out because we are not in a state that we
+	// expect, the response service imports will be cleaned-up on a timer based.
+	if t.rsi == nil {
+		return
+	}
+	t.updateRespServiceImport(e)
+
+	if t.allRespServiceImportReceived(t.rsi) {
+		for _, rsi := range t.si {
+			rsi.acc.removeRespServiceImport(rsi.si, rsiOk)
+		}
+		t.rsi, t.si = nil, nil
+	}
+}
+
+// For a given trace message event, update the tree of response service import
+// trace messages.
+// Lock is held on entry.
+func (t *msgTrace) updateRespServiceImport(e *MsgTraceEvent) {
+	// Check for the hop header.
+	hop, ok := e.Request.Header[MsgTraceHop]
+	if !ok || len(hop) != 1 {
+		return
+	}
+	hops := strings.Split(hop[0], ".")
+	prsi := t.rsi
+	for i, h := range hops {
+		rsi := prsi.tree[h]
+		if rsi == nil {
+			if prsi.tree == nil {
+				prsi.tree = make(map[string]*msgTraceRespServiceImport)
+			}
+			// Create an entry and initializes the hops count to -1. It will be set
+			// when receiving the corresponding trace message.
+			rsi = &msgTraceRespServiceImport{hops: -1}
+			// Bind it to the parent tree for this "hop" id.
+			prsi.tree[h] = rsi
+		}
+		// When dealing with the last section of the `hop` string, we set the
+		// expected hops count based on the event's `Hops` field and bump the number
+		// of received trace messages on the parent's node. We do this only once
+		// per event (use rsi.hops == -1 as the indicator).
+		if rsi.hops == -1 && i == len(hops)-1 {
+			rsi.hops = e.Hops
+			prsi.received++
+		}
+		prsi = rsi
+	}
+}
+
+// Determine if all response service import traces have been received.
+// Lock held on entry.
+func (t *msgTrace) allRespServiceImportReceived(prsi *msgTraceRespServiceImport) bool {
+	if prsi.hops != prsi.received {
+		return false
+	}
+	for _, rsi := range prsi.tree {
+		if !t.allRespServiceImportReceived(rsi) {
+			return false
+		}
+	}
+	return true
+}

server/nkey.go

@@ -0,0 +1,47 @@
+// Copyright 2018-2025 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	crand "crypto/rand"
+	"encoding/base64"
+)
+
+// Raw length of the nonce challenge
+const (
+	nonceRawLen = 11
+	nonceLen    = 15 // base64.RawURLEncoding.EncodedLen(nonceRawLen)
+)
+
+// NonceRequired tells us if we should send a nonce.
+func (s *Server) NonceRequired() bool {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	return s.nonceRequired()
+}
+
+// nonceRequired tells us if we should send a nonce.
+// Lock should be held on entry.
+func (s *Server) nonceRequired() bool {
+	return s.getOpts().AlwaysEnableNonce || len(s.nkeys) > 0 || s.trustedKeys != nil || len(s.proxiesKeyPairs) > 0
+}
+
+// Generate a nonce for INFO challenge.
+// Assumes server lock is held
+func (s *Server) generateNonce(n []byte) {
+	var raw [nonceRawLen]byte
+	data := raw[:]
+	crand.Read(data)
+	base64.RawURLEncoding.Encode(n, data)
+}

server/nkey_test.go

@@ -0,0 +1,303 @@
+// Copyright 2018-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"bufio"
+	"encoding/base64"
+	"encoding/json"
+	"fmt"
+	"strings"
+	"testing"
+
+	crand "crypto/rand"
+	mrand "math/rand/v2"
+
+	"github.com/nats-io/nkeys"
+)
+
+// Nonce has to be a string since we used different encoding by default than json.Unmarshal.
+type nonceInfo struct {
+	Id    string `json:"server_id"`
+	CID   uint64 `json:"client_id,omitempty"`
+	Nonce string `json:"nonce,omitempty"`
+}
+
+// This is a seed for a user. We can extract public and private keys from this for testing.
+var seed = []byte("SUAKYRHVIOREXV7EUZTBHUHL7NUMHPMAS7QMDU3GTIUWEI5LDNOXD43IZY")
+
+func nkeyBasicSetup() (*Server, *testAsyncClient, *bufio.Reader, string) {
+	kp, _ := nkeys.FromSeed(seed)
+	pub, _ := kp.PublicKey()
+	opts := defaultServerOptions
+	opts.Nkeys = []*NkeyUser{{Nkey: string(pub)}}
+	return rawSetup(opts)
+}
+
+func mixedSetup() (*Server, *testAsyncClient, *bufio.Reader, string) {
+	kp, _ := nkeys.FromSeed(seed)
+	pub, _ := kp.PublicKey()
+	opts := defaultServerOptions
+	opts.Nkeys = []*NkeyUser{{Nkey: string(pub)}}
+	opts.Users = []*User{{Username: "derek", Password: "foo"}}
+	return rawSetup(opts)
+}
+
+func TestServerInfoNonceAlwaysEnabled(t *testing.T) {
+	opts := defaultServerOptions
+	opts.AlwaysEnableNonce = true
+	s, c, _, l := rawSetup(opts)
+	defer s.WaitForShutdown()
+	defer s.Shutdown()
+	defer c.close()
+
+	if !strings.HasPrefix(l, "INFO ") {
+		t.Fatalf("INFO response incorrect: %s\n", l)
+	}
+
+	var info nonceInfo
+	err := json.Unmarshal([]byte(l[5:]), &info)
+	if err != nil {
+		t.Fatalf("Could not parse INFO json: %v\n", err)
+	}
+	if info.Nonce == "" {
+		t.Fatalf("Expected a non-empty nonce with AlwaysEnableNonce set")
+	}
+}
+
+func TestServerInfoNonce(t *testing.T) {
+	c, l := setUpClientWithResponse()
+	defer c.close()
+	if !strings.HasPrefix(l, "INFO ") {
+		t.Fatalf("INFO response incorrect: %s\n", l)
+	}
+	// Make sure payload is proper json
+	var info nonceInfo
+	err := json.Unmarshal([]byte(l[5:]), &info)
+	if err != nil {
+		t.Fatalf("Could not parse INFO json: %v\n", err)
+	}
+	if info.Nonce != "" {
+		t.Fatalf("Expected an empty nonce with no nkeys defined")
+	}
+
+	// Now setup server with auth and nkeys to trigger nonce generation
+	s, c, _, l := nkeyBasicSetup()
+	defer c.close()
+
+	if !strings.HasPrefix(l, "INFO ") {
+		t.Fatalf("INFO response incorrect: %s\n", l)
+	}
+	// Make sure payload is proper json
+	err = json.Unmarshal([]byte(l[5:]), &info)
+	if err != nil {
+		t.Fatalf("Could not parse INFO json: %v\n", err)
+	}
+	if info.Nonce == "" {
+		t.Fatalf("Expected a non-empty nonce with nkeys defined")
+	}
+
+	// Make sure new clients get new nonces
+	oldNonce := info.Nonce
+
+	c, _, l = newClientForServer(s)
+	defer c.close()
+
+	err = json.Unmarshal([]byte(l[5:]), &info)
+	if err != nil {
+		t.Fatalf("Could not parse INFO json: %v\n", err)
+	}
+	if info.Nonce == "" {
+		t.Fatalf("Expected a non-empty nonce")
+	}
+	if strings.Compare(oldNonce, info.Nonce) == 0 {
+		t.Fatalf("Expected subsequent nonces to be different\n")
+	}
+}
+
+func TestNkeyClientConnect(t *testing.T) {
+	s, c, cr, _ := nkeyBasicSetup()
+	defer c.close()
+	// Send CONNECT with no signature or nkey, should fail.
+	connectOp := "CONNECT {\"verbose\":true,\"pedantic\":true}\r\n"
+	c.parseAsync(connectOp)
+	l, _ := cr.ReadString('\n')
+	if !strings.HasPrefix(l, "-ERR ") {
+		t.Fatalf("Expected an error")
+	}
+
+	kp, _ := nkeys.FromSeed(seed)
+	pubKey, _ := kp.PublicKey()
+
+	// Send nkey but no signature
+	c, cr, _ = newClientForServer(s)
+	defer c.close()
+	cs := fmt.Sprintf("CONNECT {\"nkey\":%q, \"verbose\":true,\"pedantic\":true}\r\n", pubKey)
+	c.parseAsync(cs)
+	l, _ = cr.ReadString('\n')
+	if !strings.HasPrefix(l, "-ERR ") {
+		t.Fatalf("Expected an error")
+	}
+
+	// Now improperly sign etc.
+	c, cr, _ = newClientForServer(s)
+	defer c.close()
+	cs = fmt.Sprintf("CONNECT {\"nkey\":%q,\"sig\":%q,\"verbose\":true,\"pedantic\":true}\r\n", pubKey, "bad_sig")
+	c.parseAsync(cs)
+	l, _ = cr.ReadString('\n')
+	if !strings.HasPrefix(l, "-ERR ") {
+		t.Fatalf("Expected an error")
+	}
+
+	// Now properly sign the nonce
+	c, cr, l = newClientForServer(s)
+	defer c.close()
+	// Check for Nonce
+	var info nonceInfo
+	err := json.Unmarshal([]byte(l[5:]), &info)
+	if err != nil {
+		t.Fatalf("Could not parse INFO json: %v\n", err)
+	}
+	if info.Nonce == "" {
+		t.Fatalf("Expected a non-empty nonce with nkeys defined")
+	}
+	sigraw, err := kp.Sign([]byte(info.Nonce))
+	if err != nil {
+		t.Fatalf("Failed signing nonce: %v", err)
+	}
+	sig := base64.RawURLEncoding.EncodeToString(sigraw)
+
+	// PING needed to flush the +OK to us.
+	cs = fmt.Sprintf("CONNECT {\"nkey\":%q,\"sig\":\"%s\",\"verbose\":true,\"pedantic\":true}\r\nPING\r\n", pubKey, sig)
+	c.parseAsync(cs)
+	l, _ = cr.ReadString('\n')
+	if !strings.HasPrefix(l, "+OK") {
+		t.Fatalf("Expected an OK, got: %v", l)
+	}
+}
+
+func TestMixedClientConnect(t *testing.T) {
+	s, c, cr, _ := mixedSetup()
+	defer c.close()
+	// Normal user/pass
+	c.parseAsync("CONNECT {\"user\":\"derek\",\"pass\":\"foo\",\"verbose\":true,\"pedantic\":true}\r\nPING\r\n")
+	l, _ := cr.ReadString('\n')
+	if !strings.HasPrefix(l, "+OK") {
+		t.Fatalf("Expected an OK, got: %v", l)
+	}
+
+	kp, _ := nkeys.FromSeed(seed)
+	pubKey, _ := kp.PublicKey()
+
+	c, cr, l = newClientForServer(s)
+	defer c.close()
+	// Check for Nonce
+	var info nonceInfo
+	err := json.Unmarshal([]byte(l[5:]), &info)
+	if err != nil {
+		t.Fatalf("Could not parse INFO json: %v\n", err)
+	}
+	if info.Nonce == "" {
+		t.Fatalf("Expected a non-empty nonce with nkeys defined")
+	}
+	sigraw, err := kp.Sign([]byte(info.Nonce))
+	if err != nil {
+		t.Fatalf("Failed signing nonce: %v", err)
+	}
+	sig := base64.RawURLEncoding.EncodeToString(sigraw)
+
+	// PING needed to flush the +OK to us.
+	cs := fmt.Sprintf("CONNECT {\"nkey\":%q,\"sig\":\"%s\",\"verbose\":true,\"pedantic\":true}\r\nPING\r\n", pubKey, sig)
+	c.parseAsync(cs)
+	l, _ = cr.ReadString('\n')
+	if !strings.HasPrefix(l, "+OK") {
+		t.Fatalf("Expected an OK, got: %v", l)
+	}
+}
+
+func TestMixedClientConfig(t *testing.T) {
+	confFileName := createConfFile(t, []byte(`
+    authorization {
+      users = [
+        {nkey: "UDKTV7HZVYJFJN64LLMYQBUR6MTNNYCDC3LAZH4VHURW3GZLL3FULBXV"}
+        {user: alice, password: foo}
+      ]
+    }`))
+	opts, err := ProcessConfigFile(confFileName)
+	if err != nil {
+		t.Fatalf("Received an error processing config file: %v", err)
+	}
+	if len(opts.Nkeys) != 1 {
+		t.Fatalf("Expected 1 nkey, got %d", len(opts.Nkeys))
+	}
+	if len(opts.Users) != 1 {
+		t.Fatalf("Expected 1 user, got %d", len(opts.Users))
+	}
+}
+
+func BenchmarkCryptoRandGeneration(b *testing.B) {
+	data := make([]byte, 16)
+	for i := 0; i < b.N; i++ {
+		crand.Read(data)
+	}
+}
+
+func BenchmarkMathRandGeneration(b *testing.B) {
+	data := make([]byte, 16)
+	prng := mrand.NewChaCha8([32]byte{42})
+	for i := 0; i < b.N; i++ {
+		prng.Read(data)
+	}
+}
+
+func BenchmarkNonceGeneration(b *testing.B) {
+	data := make([]byte, nonceRawLen)
+	b64 := make([]byte, nonceLen)
+	prand := mrand.NewChaCha8([32]byte{42})
+	for i := 0; i < b.N; i++ {
+		prand.Read(data)
+		base64.RawURLEncoding.Encode(b64, data)
+	}
+}
+
+func BenchmarkPublicVerify(b *testing.B) {
+	data := make([]byte, nonceRawLen)
+	nonce := make([]byte, nonceLen)
+	crand.Read(data)
+	base64.RawURLEncoding.Encode(nonce, data)
+
+	user, err := nkeys.CreateUser()
+	if err != nil {
+		b.Fatalf("Error creating User Nkey: %v", err)
+	}
+	sig, err := user.Sign(nonce)
+	if err != nil {
+		b.Fatalf("Error sigining nonce: %v", err)
+	}
+	pk, err := user.PublicKey()
+	if err != nil {
+		b.Fatalf("Could not extract public key from user: %v", err)
+	}
+	pub, err := nkeys.FromPublicKey(pk)
+	if err != nil {
+		b.Fatalf("Could not create public key pair from public key string: %v", err)
+	}
+
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		if err := pub.Verify(nonce, sig); err != nil {
+			b.Fatalf("Error verifying nonce: %v", err)
+		}
+	}
+}

server/ocsp.go

@@ -0,0 +1,997 @@
+// Copyright 2021-2026 The NATS Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package server
+
+import (
+	"bytes"
+	"crypto/sha256"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/asn1"
+	"encoding/base64"
+	"encoding/pem"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"time"
+
+	"golang.org/x/crypto/ocsp"
+
+	"github.com/nats-io/nats-server/v2/server/certidp"
+	"github.com/nats-io/nats-server/v2/server/certstore"
+)
+
+const (
+	defaultOCSPStoreDir      = "ocsp"
+	defaultOCSPCheckInterval = 24 * time.Hour
+	minOCSPCheckInterval     = 2 * time.Minute
+)
+
+type OCSPMode uint8
+
+const (
+	// OCSPModeAuto staples a status, only if "status_request" is set in cert.
+	OCSPModeAuto OCSPMode = iota
+
+	// OCSPModeAlways enforces OCSP stapling for certs and shuts down the server in
+	// case a server is revoked or cannot get OCSP staples.
+	OCSPModeAlways
+
+	// OCSPModeNever disables OCSP stapling even if cert has Must-Staple flag.
+	OCSPModeNever
+
+	// OCSPModeMust honors the Must-Staple flag from a certificate but also causing shutdown
+	// in case the certificate has been revoked.
+	OCSPModeMust
+)
+
+// OCSPMonitor monitors the state of a staple per certificate.
+type OCSPMonitor struct {
+	kind     string
+	mu       sync.Mutex
+	raw      []byte
+	srv      *Server
+	certFile string
+	resp     *ocsp.Response
+	hc       *http.Client
+	stopCh   chan struct{}
+	Leaf     *x509.Certificate
+	Issuer   *x509.Certificate
+
+	shutdownOnRevoke bool
+}
+
+func (oc *OCSPMonitor) getNextRun() time.Duration {
+	oc.mu.Lock()
+	nextUpdate := oc.resp.NextUpdate
+	oc.mu.Unlock()
+
+	now := time.Now()
+	if nextUpdate.IsZero() {
+		// If response is missing NextUpdate, we check the day after.
+		// Technically, if NextUpdate is missing, we can try whenever.
+		// https://tools.ietf.org/html/rfc6960#section-4.2.2.1
+		return defaultOCSPCheckInterval
+	}
+	dur := nextUpdate.Sub(now) / 2
+
+	// If negative, then wait a couple of minutes before getting another staple.
+	if dur < 0 {
+		return minOCSPCheckInterval
+	}
+
+	return dur
+}
+
+func (oc *OCSPMonitor) getStatus() ([]byte, *ocsp.Response, error) {
+	raw, resp := oc.getCacheStatus()
+	if len(raw) > 0 && resp != nil {
+		// Check if the OCSP is still valid.
+		if err := validOCSPResponse(resp); err == nil {
+			return raw, resp, nil
+		}
+	}
+	var err error
+	raw, resp, err = oc.getLocalStatus()
+	if err == nil {
+		return raw, resp, nil
+	}
+
+	return oc.getRemoteStatus()
+}
+
+func (oc *OCSPMonitor) getCacheStatus() ([]byte, *ocsp.Response) {
+	oc.mu.Lock()
+	defer oc.mu.Unlock()
+	return oc.raw, oc.resp
+}
+
+func (oc *OCSPMonitor) getLocalStatus() ([]byte, *ocsp.Response, error) {
+	opts := oc.srv.getOpts()
+	storeDir := opts.StoreDir
+	if storeDir == _EMPTY_ {
+		return nil, nil, fmt.Errorf("store_dir not set")
+	}
+
+	// This key must be based upon the current full certificate, not the public key,
+	// so MUST be on the full raw certificate and not an SPKI or other reduced form.
+	key := fmt.Sprintf("%x", sha256.Sum256(oc.Leaf.Raw))
+
+	oc.mu.Lock()
+	raw, err := os.ReadFile(filepath.Join(storeDir, defaultOCSPStoreDir, key))
+	oc.mu.Unlock()
+	if err != nil {
+		return nil, nil, err
+	}
+
+	resp, err := ocsp.ParseResponse(raw, oc.Issuer)
+	if err != nil {
+		return nil, nil, fmt.Errorf("failed to get local status: %w", err)
+	}
+	if err := validOCSPResponse(resp); err != nil {
+		return nil, nil, err
+	}
+
+	// Cache the response.
+	oc.mu.Lock()
+	oc.raw = raw
+	oc.resp = resp
+	oc.mu.Unlock()
+
+	return raw, resp, nil
+}
+
+func (oc *OCSPMonitor) getRemoteStatus() ([]byte, *ocsp.Response, error) {
+	opts := oc.srv.getOpts()
+	var overrideURLs []string
+	if config := opts.OCSPConfig; config != nil {
+		overrideURLs = config.OverrideURLs
+	}
+	getRequestBytes := func(u string, reqDER []byte, hc *http.Client) ([]byte, error) {
+		reqEnc := base64.StdEncoding.EncodeToString(reqDER)
+		u = fmt.Sprintf("%s/%s", u, reqEnc)
+		start := time.Now()
+		resp, err := hc.Get(u)
+		if err != nil {
+			return nil, err
+		}
+		defer resp.Body.Close()
+
+		oc.srv.Debugf("Received OCSP response (method=GET, status=%v, url=%s, duration=%.3fs)",
+			resp.StatusCode, u, time.Since(start).Seconds())
+		if resp.StatusCode > 299 {
+			return nil, fmt.Errorf("non-ok http status on GET request (reqlen=%d): %d", len(reqEnc), resp.StatusCode)
+		}
+		return io.ReadAll(resp.Body)
+	}
+	postRequestBytes := func(u string, body []byte, hc *http.Client) ([]byte, error) {
+		hreq, err := http.NewRequest("POST", u, bytes.NewReader(body))
+		if err != nil {
+			return nil, err
+		}
+		hreq.Header.Add("Content-Type", "application/ocsp-request")
+		hreq.Header.Add("Accept", "application/ocsp-response")
+
+		start := time.Now()
+		resp, err := hc.Do(hreq)
+		if err != nil {
+			return nil, err
+		}
+		defer resp.Body.Close()
+
+		oc.srv.Debugf("Received OCSP response (method=POST, status=%v, url=%s, duration=%.3fs)",
+			resp.StatusCode, u, time.Since(start).Seconds())
+		if resp.StatusCode > 299 {
+			return nil, fmt.Errorf("non-ok http status on POST request (reqlen=%d): %d", len(body), resp.StatusCode)
+		}
+		return io.ReadAll(resp.Body)
+	}
+
+	// Request documentation:
+	// https://tools.ietf.org/html/rfc6960#appendix-A.1
+
+	reqDER, err := ocsp.CreateRequest(oc.Leaf, oc.Issuer, nil)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	responders := oc.Leaf.OCSPServer
+	if len(overrideURLs) > 0 {
+		responders = overrideURLs
+	}
+	if len(responders) == 0 {
+		return nil, nil, fmt.Errorf("no available ocsp servers")
+	}
+
+	oc.mu.Lock()
+	hc := oc.hc
+	oc.mu.Unlock()
+
+	var raw []byte
+	for _, u := range responders {
+		var postErr, getErr error
+		u = strings.TrimSuffix(u, "/")
+		// Prefer to make POST requests first.
+		raw, postErr = postRequestBytes(u, reqDER, hc)
+		if postErr == nil {
+			err = nil
+			break
+		} else {
+			// Fallback to use a GET request.
+			raw, getErr = getRequestBytes(u, reqDER, hc)
+			if getErr == nil {
+				err = nil
+				break
+			} else {
+				err = errors.Join(postErr, getErr)
+			}
+		}
+	}
+	if err != nil {
+		return nil, nil, fmt.Errorf("exhausted ocsp servers: %w", err)
+	}
+	resp, err := ocsp.ParseResponse(raw, oc.Issuer)
+	if err != nil {
+		return nil, nil, fmt.Errorf("failed to get remote status: %w", err)
+	}
+	if err := validOCSPResponse(resp); err != nil {
+		return nil, nil, err
+	}
+
+	if storeDir := opts.StoreDir; storeDir != _EMPTY_ {
+		key := fmt.Sprintf("%x", sha256.Sum256(oc.Leaf.Raw))
+		if err := oc.writeOCSPStatus(storeDir, key, raw); err != nil {
+			return nil, nil, fmt.Errorf("failed to write ocsp status: %w", err)
+		}
+	}
+
+	oc.mu.Lock()
+	oc.raw = raw
+	oc.resp = resp
+	oc.mu.Unlock()
+
+	return raw, resp, nil
+}
+
+func (oc *OCSPMonitor) run() {
+	s := oc.srv
+	s.mu.Lock()
+	quitCh := s.quitCh
+	s.mu.Unlock()
+
+	var doShutdown bool
+	defer func() {
+		// Need to decrement before shuting down, otherwise shutdown
+		// would be stuck waiting on grWG to go down to 0.
+		s.grWG.Done()
+		if doShutdown {
+			s.Shutdown()
+		}
+	}()
+
+	oc.mu.Lock()
+	shutdownOnRevoke := oc.shutdownOnRevoke
+	certFile := oc.certFile
+	stopCh := oc.stopCh
+	kind := oc.kind
+	oc.mu.Unlock()
+
+	var nextRun time.Duration
+	_, resp, err := oc.getStatus()
+	if err == nil && resp.Status == ocsp.Good {
+		nextRun = oc.getNextRun()
+		t := resp.NextUpdate.Format(time.RFC3339Nano)
+		s.Noticef(
+			"Found OCSP status for %s certificate at '%s': good, next update %s, checking again in %s",
+			kind, certFile, t, nextRun,
+		)
+	} else if err == nil && shutdownOnRevoke {
+		// If resp.Status is ocsp.Revoked, ocsp.Unknown, or any other value.
+		s.Errorf("Found OCSP status for %s certificate at '%s': %s", kind, certFile, ocspStatusString(resp.Status))
+		doShutdown = true
+		return
+	}
+
+	nextRunTimer := time.NewTimer(nextRun)
+	defer stopAndClearTimer(&nextRunTimer)
+
+	for {
+		// On reload, if the certificate changes then need to stop this monitor.
+		select {
+		case <-nextRunTimer.C:
+		case <-stopCh:
+			// In case of reload and have to restart the OCSP stapling monitoring.
+			return
+		case <-quitCh:
+			// Server quit channel.
+			return
+		}
+		_, resp, err := oc.getRemoteStatus()
+		if err != nil {
+			nextRun = oc.getNextRun()
+			s.Errorf("Bad OCSP status update for certificate '%s': %s, trying again in %v", certFile, err, nextRun)
+			nextRunTimer.Reset(nextRun)
+			continue
+		}
+
+		switch n := resp.Status; n {
+		case ocsp.Good:
+			nextRun = oc.getNextRun()
+			t := resp.NextUpdate.Format(time.RFC3339Nano)
+			s.Noticef(
+				"Received OCSP status for %s certificate '%s': good, next update %s, checking again in %s",
+				kind, certFile, t, nextRun,
+			)
+			nextRunTimer.Reset(nextRun)
+			continue
+		default:
+			s.Errorf("Received OCSP status for %s certificate '%s': %s", kind, certFile, ocspStatusString(n))
+			if shutdownOnRevoke {
+				doShutdown = true
+			}
+			return
+		}
+	}
+}
+
+func (oc *OCSPMonitor) stop() {
+	oc.mu.Lock()
+	stopCh := oc.stopCh
+	oc.mu.Unlock()
+	stopCh <- struct{}{}
+}
+
+// NewOCSPMonitor takes a TLS configuration then wraps it with the callbacks set for OCSP verification
+// along with a monitor that will periodically fetch OCSP staples.
+func (srv *Server) NewOCSPMonitor(config *tlsConfigKind) (*tls.Config, *OCSPMonitor, error) {
+	kind := config.kind
+	tc := config.tlsConfig
+	tcOpts := config.tlsOpts
+	opts := srv.getOpts()
+	oc := opts.OCSPConfig
+
+	// We need to track the CA certificate in case the CA is not present
+	// in the chain to be able to verify the signature of the OCSP staple.
+	var (
+		certFile string
+		caFile   string
+	)
+	if kind == kindStringMap[CLIENT] {
+		tcOpts = opts.tlsConfigOpts
+		if opts.TLSCert != _EMPTY_ {
+			certFile = opts.TLSCert
+		}
+		if opts.TLSCaCert != _EMPTY_ {
+			caFile = opts.TLSCaCert
+		}
+	}
+	if tcOpts != nil {
+		certFile = tcOpts.CertFile
+		caFile = tcOpts.CaFile
+	}
+
+	// NOTE: Currently OCSP Stapling is enabled only for the first certificate found.
+	var mon *OCSPMonitor
+	for _, currentCert := range tc.Certificates {
+		// Create local copy since this will be used in the GetCertificate callback.
+		cert := currentCert
+
+		// This is normally non-nil, but can still be nil here when in tests
+		// or in some embedded scenarios.
+		if cert.Leaf == nil {
+			if len(cert.Certificate) <= 0 {
+				return nil, nil, fmt.Errorf("no certificate found")
+			}
+			var err error
+			cert.Leaf, err = x509.ParseCertificate(cert.Certificate[0])
+			if err != nil {
+				return nil, nil, fmt.Errorf("error parsing certificate: %v", err)
+			}
+		}
+		var shutdownOnRevoke bool
+		mustStaple := hasOCSPStatusRequest(cert.Leaf)
+		if oc != nil {
+			switch {
+			case oc.Mode == OCSPModeNever:
+				if mustStaple {
+					srv.Warnf("Certificate at '%s' has MustStaple but OCSP is disabled", certFile)
+				}
+				return tc, nil, nil
+			case oc.Mode == OCSPModeAlways:
+				// Start the monitor for this cert even if it does not have
+				// the MustStaple flag and shutdown the server in case the
+				// staple ever gets revoked.
+				mustStaple = true
+				shutdownOnRevoke = true
+			case oc.Mode == OCSPModeMust && mustStaple:
+				shutdownOnRevoke = true
+			case oc.Mode == OCSPModeAuto && !mustStaple:
+				// "status_request" MustStaple flag not set in certificate. No need to do anything.
+				return tc, nil, nil
+			}
+		}
+		if !mustStaple {
+			// No explicit OCSP config and cert does not have MustStaple flag either.
+			return tc, nil, nil
+		}
+
+		if err := srv.setupOCSPStapleStoreDir(); err != nil {
+			return nil, nil, err
+		}
+
+		// TODO: Add OCSP 'responder_cert' option in case CA cert not available.
+		issuer, err := getOCSPIssuer(caFile, cert.Certificate)
+		if err != nil {
+			return nil, nil, err
+		}
+
+		mon = &OCSPMonitor{
+			kind:             kind,
+			srv:              srv,
+			hc:               &http.Client{Timeout: 30 * time.Second},
+			shutdownOnRevoke: shutdownOnRevoke,
+			certFile:         certFile,
+			stopCh:           make(chan struct{}, 1),
+			Leaf:             cert.Leaf,
+			Issuer:           issuer,
+		}
+
+		// Get the certificate status from the memory, then remote OCSP responder.
+		if _, resp, err := mon.getStatus(); err != nil {
+			return nil, nil, fmt.Errorf("bad OCSP status update for certificate at '%s': %s", certFile, err)
+		} else if resp != nil && resp.Status != ocsp.Good && shutdownOnRevoke {
+			return nil, nil, fmt.Errorf("found existing OCSP status for certificate at '%s': %s", certFile, ocspStatusString(resp.Status))
+		}
+
+		// Callbacks below will be in charge of returning the certificate instead,
+		// so this has to be nil.
+		tc.Certificates = nil
+
+		// GetCertificate returns a certificate that's presented to a client.
+		tc.GetCertificate = func(info *tls.ClientHelloInfo) (*tls.Certificate, error) {
+			ccert := cert
+			raw, _, err := mon.getStatus()
+			if err != nil {
+				return nil, err
+			}
+			return &tls.Certificate{
+				OCSPStaple:                   raw,
+				Certificate:                  ccert.Certificate,
+				PrivateKey:                   ccert.PrivateKey,
+				SupportedSignatureAlgorithms: ccert.SupportedSignatureAlgorithms,
+				SignedCertificateTimestamps:  ccert.SignedCertificateTimestamps,
+				Leaf:                         ccert.Leaf,
+			}, nil
+		}
+
+		// Check whether need to verify staples from a peer router or gateway connection.
+		switch kind {
+		case kindStringMap[ROUTER], kindStringMap[GATEWAY]:
+			tc.VerifyConnection = func(s tls.ConnectionState) error {
+				oresp := s.OCSPResponse
+				if oresp == nil {
+					return fmt.Errorf("%s peer missing OCSP Staple", kind)
+				}
+
+				// Peer connections will verify the response of the staple.
+				if len(s.VerifiedChains) == 0 {
+					return fmt.Errorf("%s peer missing TLS verified chains", kind)
+				}
+
+				chain := s.VerifiedChains[0]
+				peerLeaf := chain[0]
+				peerIssuer := certidp.GetLeafIssuerCert(chain, 0)
+				if peerIssuer == nil {
+					return fmt.Errorf("failed to get issuer certificate for %s peer", kind)
+				}
+
+				// Response signature of issuer or issuer delegate is checked in the library parse
+				resp, err := ocsp.ParseResponseForCert(oresp, peerLeaf, peerIssuer)
+				if err != nil {
+					return fmt.Errorf("failed to parse OCSP response from %s peer: %w", kind, err)
+				}
+
+				// If signer was issuer delegate double-check issuer delegate authorization
+				if resp.Certificate != nil {
+					ok := false
+					for _, eku := range resp.Certificate.ExtKeyUsage {
+						if eku == x509.ExtKeyUsageOCSPSigning {
+							ok = true
+							break
+						}
+					}
+					if !ok {
+						return fmt.Errorf("OCSP staple's signer missing authorization by CA to act as OCSP signer")
+					}
+				}
+
+				// Check that the OCSP response is effective, take defaults for clockskew and default validity
+				peerOpts := certidp.OCSPPeerConfig{ClockSkew: -1, TTLUnsetNextUpdate: -1}
+				sLog := certidp.Log{Debugf: srv.Debugf}
+				if !certidp.OCSPResponseCurrent(resp, &peerOpts, &sLog) {
+					return fmt.Errorf("OCSP staple from %s peer not current", kind)
+				}
+
+				if resp.Status != ocsp.Good {
+					return fmt.Errorf("bad status for OCSP Staple from %s peer: %s", kind, ocspStatusString(resp.Status))
+				}
+
+				return nil
+			}
+
+			// When server makes a peer connection, need to also present an OCSP Staple.
+			tc.GetClientCertificate = func(info *tls.CertificateRequestInfo) (*tls.Certificate, error) {
+				ccert := cert
+				raw, _, err := mon.getStatus()
+				if err != nil {
+					return nil, err
+				}
+				// NOTE: crypto/tls.sendClientCertificate internally also calls getClientCertificate
+				// so if for some reason these callbacks are triggered concurrently during a reconnect
+				// there can be a race. To avoid that, the OCSP monitor lock is used to serialize access
+				// to the staple which could also change inflight during an update.
+				mon.mu.Lock()
+				ccert.OCSPStaple = raw
+				mon.mu.Unlock()
+
+				return &ccert, nil
+			}
+		default:
+			// GetClientCertificate returns a certificate that's presented to a server.
+			tc.GetClientCertificate = func(info *tls.CertificateRequestInfo) (*tls.Certificate, error) {
+				return &cert, nil
+			}
+		}
+	}
+	return tc, mon, nil
+}
+
+func (s *Server) setupOCSPStapleStoreDir() error {
+	opts := s.getOpts()
+	storeDir := opts.StoreDir
+	if storeDir == _EMPTY_ {
+		return nil
+	}
+	storeDir = filepath.Join(storeDir, defaultOCSPStoreDir)
+	if stat, err := os.Stat(storeDir); os.IsNotExist(err) {
+		if err := os.MkdirAll(storeDir, defaultDirPerms); err != nil {
+			return fmt.Errorf("could not create OCSP storage directory - %v", err)
+		}
+	} else if stat == nil || !stat.IsDir() {
+		return fmt.Errorf("OCSP storage directory is not a directory")
+	}
+	return nil
+}
+
+type tlsConfigKind struct {
+	tlsConfig   *tls.Config
+	tlsOpts     *TLSConfigOpts
+	kind        string
+	isLeafSpoke bool
+	apply       func(*tls.Config)
+}
+
+func (s *Server) configureOCSP() []*tlsConfigKind {
+	sopts := s.getOpts()
+
+	configs := make([]*tlsConfigKind, 0)
+
+	if config := sopts.TLSConfig; config != nil {
+		opts := sopts.tlsConfigOpts
+		o := &tlsConfigKind{
+			kind:      kindStringMap[CLIENT],
+			tlsConfig: config,
+			tlsOpts:   opts,
+			apply:     func(tc *tls.Config) { sopts.TLSConfig = tc },
+		}
+		configs = append(configs, o)
+	}
+	if config := sopts.Websocket.TLSConfig; config != nil {
+		opts := sopts.Websocket.tlsConfigOpts
+		o := &tlsConfigKind{
+			kind:      kindStringMap[CLIENT],
+			tlsConfig: config,
+			tlsOpts:   opts,
+			apply:     func(tc *tls.Config) { sopts.Websocket.TLSConfig = tc },
+		}
+		configs = append(configs, o)
+	}
+	if config := sopts.MQTT.TLSConfig; config != nil {
+		opts := sopts.tlsConfigOpts
+		o := &tlsConfigKind{
+			kind:      kindStringMap[CLIENT],
+			tlsConfig: config,
+			tlsOpts:   opts,
+			apply:     func(tc *tls.Config) { sopts.MQTT.TLSConfig = tc },
+		}
+		configs = append(configs, o)
+	}
+	if config := sopts.Cluster.TLSConfig; config != nil {
+		opts := sopts.Cluster.tlsConfigOpts
+		o := &tlsConfigKind{
+			kind:      kindStringMap[ROUTER],
+			tlsConfig: config,
+			tlsOpts:   opts,
+			apply:     func(tc *tls.Config) { sopts.Cluster.TLSConfig = tc },
+		}
+		configs = append(configs, o)
+	}
+	if config := sopts.LeafNode.TLSConfig; config != nil {
+		opts := sopts.LeafNode.tlsConfigOpts
+		o := &tlsConfigKind{
+			kind:      kindStringMap[LEAF],
+			tlsConfig: config,
+			tlsOpts:   opts,
+			apply:     func(tc *tls.Config) { sopts.LeafNode.TLSConfig = tc },
+		}
+		configs = append(configs, o)
+	}
+	for _, remote := range sopts.LeafNode.Remotes {
+		if config := remote.TLSConfig; config != nil {
+			// Use a copy of the remote here since will be used
+			// in the apply func callback below.
+			r, opts := remote, remote.tlsConfigOpts
+			o := &tlsConfigKind{
+				kind:        kindStringMap[LEAF],
+				tlsConfig:   config,
+				tlsOpts:     opts,
+				isLeafSpoke: true,
+				apply:       func(tc *tls.Config) { r.TLSConfig = tc },
+			}
+			configs = append(configs, o)
+		}
+	}
+	if config := sopts.Gateway.TLSConfig; config != nil {
+		opts := sopts.Gateway.tlsConfigOpts
+		o := &tlsConfigKind{
+			kind:      kindStringMap[GATEWAY],
+			tlsConfig: config,
+			tlsOpts:   opts,
+			apply:     func(tc *tls.Config) { sopts.Gateway.TLSConfig = tc },
+		}
+		configs = append(configs, o)
+	}
+	for _, remote := range sopts.Gateway.Gateways {
+		if config := remote.TLSConfig; config != nil {
+			gw, opts := remote, remote.tlsConfigOpts
+			o := &tlsConfigKind{
+				kind:      kindStringMap[GATEWAY],
+				tlsConfig: config,
+				tlsOpts:   opts,
+				apply:     func(tc *tls.Config) { gw.TLSConfig = tc },
+			}
+			configs = append(configs, o)
+		}
+	}
+	return configs
+}
+
+func (s *Server) enableOCSP() error {
+	configs := s.configureOCSP()
+
+	for _, config := range configs {
+
+		// We do not staple Leaf Hub and Leaf Spokes, use ocsp_peer
+		if config.kind != kindStringMap[LEAF] {
+			// OCSP Stapling feature, will also enable tls server peer check for gateway and route peers
+			tc, mon, err := s.NewOCSPMonitor(config)
+			if err != nil {
+				return err
+			}
+			// Check if an OCSP stapling monitor is required for this certificate.
+			if mon != nil {
+				s.ocsps = append(s.ocsps, mon)
+
+				// Override the TLS config with one that follows OCSP stapling
+				config.apply(tc)
+			}
+		}
+
+		// OCSP peer check (client mTLS, leaf mTLS, leaf remote TLS)
+		if config.kind == kindStringMap[CLIENT] || config.kind == kindStringMap[LEAF] {
+			tc, plugged, err := s.plugTLSOCSPPeer(config)
+			if err != nil {
+				return err
+			}
+			if plugged && tc != nil {
+				s.ocspPeerVerify = true
+				config.apply(tc)
+			}
+		}
+	}
+
+	return nil
+}
+
+func (s *Server) startOCSPMonitoring() {
+	s.mu.Lock()
+	ocsps := s.ocsps
+	s.mu.Unlock()
+	if ocsps == nil {
+		return
+	}
+	for _, mon := range ocsps {
+		m := mon
+		m.mu.Lock()
+		kind := m.kind
+		m.mu.Unlock()
+		s.Noticef("OCSP Stapling enabled for %s connections", kind)
+		s.startGoRoutine(func() { m.run() })
+	}
+}
+
+func (s *Server) reloadOCSP() error {
+	if err := s.setupOCSPStapleStoreDir(); err != nil {
+		return err
+	}
+
+	s.mu.Lock()
+	ocsps := s.ocsps
+	s.mu.Unlock()
+
+	// Stop all OCSP Stapling monitors in case there were any running.
+	for _, oc := range ocsps {
+		oc.stop()
+	}
+
+	configs := s.configureOCSP()
+
+	// Restart the monitors under the new configuration.
+	ocspm := make([]*OCSPMonitor, 0)
+
+	// Reset server's ocspPeerVerify flag to re-detect at least one plugged OCSP peer
+	s.mu.Lock()
+	s.ocspPeerVerify = false
+	s.mu.Unlock()
+	s.stopOCSPResponseCache()
+
+	for _, config := range configs {
+		// We do not staple Leaf Hub and Leaf Spokes, use ocsp_peer
+		if config.kind != kindStringMap[LEAF] {
+			tc, mon, err := s.NewOCSPMonitor(config)
+			if err != nil {
+				return err
+			}
+			// Check if an OCSP stapling monitor is required for this certificate.
+			if mon != nil {
+				ocspm = append(ocspm, mon)
+
+				// Apply latest TLS configuration after OCSP monitors have started.
+				defer config.apply(tc)
+			}
+		}
+
+		// OCSP peer check (client mTLS, leaf mTLS, leaf remote TLS)
+		if config.kind == kindStringMap[CLIENT] || config.kind == kindStringMap[LEAF] {
+			tc, plugged, err := s.plugTLSOCSPPeer(config)
+			if err != nil {
+				return err
+			}
+			if plugged && tc != nil {
+				s.ocspPeerVerify = true
+				defer config.apply(tc)
+			}
+		}
+	}
+
+	// Replace stopped monitors with the new ones.
+	s.mu.Lock()
+	s.ocsps = ocspm
+	s.mu.Unlock()
+
+	// Dispatch all goroutines once again.
+	s.startOCSPMonitoring()
+
+	// Init and restart OCSP responder cache
+	s.stopOCSPResponseCache()
+	s.initOCSPResponseCache()
+	s.startOCSPResponseCache()
+
+	return nil
+}
+
+func hasOCSPStatusRequest(cert *x509.Certificate) bool {
+	// OID for id-pe-tlsfeature defined in RFC here:
+	// https://datatracker.ietf.org/doc/html/rfc7633
+	tlsFeatures := asn1.ObjectIdentifier{1, 3, 6, 1, 5, 5, 7, 1, 24}
+	const statusRequestExt = 5
+
+	// Example values:
+	// * [48 3 2 1 5] - seen when creating own certs locally
+	// * [30 3 2 1 5] - seen in the wild
+	// Documentation:
+	// https://tools.ietf.org/html/rfc6066
+
+	for _, ext := range cert.Extensions {
+		if !ext.Id.Equal(tlsFeatures) {
+			continue
+		}
+
+		var val []int
+		rest, err := asn1.Unmarshal(ext.Value, &val)
+		if err != nil || len(rest) > 0 {
+			return false
+		}
+
+		for _, n := range val {
+			if n == statusRequestExt {
+				return true
+			}
+		}
+		break
+	}
+
+	return false
+}
+
+// writeOCSPStatus writes an OCSP status to a temporary file then moves it to a
+// new path, in an attempt to avoid corrupting existing data.
+func (oc *OCSPMonitor) writeOCSPStatus(storeDir, file string, data []byte) error {
+	storeDir = filepath.Join(storeDir, defaultOCSPStoreDir)
+	tmp, err := os.CreateTemp(storeDir, "tmp-cert-status")
+	if err != nil {
+		return err
+	}
+
+	if _, err := tmp.Write(data); err != nil {
+		tmp.Close()
+		os.Remove(tmp.Name())
+		return err
+	}
+	if err := tmp.Close(); err != nil {
+		return err
+	}
+
+	oc.mu.Lock()
+	err = os.Rename(tmp.Name(), filepath.Join(storeDir, file))
+	oc.mu.Unlock()
+	if err != nil {
+		os.Remove(tmp.Name())
+		return err
+	}
+
+	return nil
+}
+
+func parseCertPEM(name string) ([]*x509.Certificate, error) {
+	data, err := os.ReadFile(name)
+	if err != nil {
+		return nil, err
+	}
+
+	var pemBytes []byte
+
+	var block *pem.Block
+	for len(data) != 0 {
+		block, data = pem.Decode(data)
+		if block == nil {
+			break
+		}
+		if block.Type != "CERTIFICATE" {
+			return nil, fmt.Errorf("unexpected PEM certificate type: %s", block.Type)
+		}
+
+		pemBytes = append(pemBytes, block.Bytes...)
+	}
+
+	return x509.ParseCertificates(pemBytes)
+}
+
+// getOCSPIssuerLocally determines a leaf's issuer from locally configured certificates
+func getOCSPIssuerLocally(trustedCAs []*x509.Certificate, certBundle []*x509.Certificate) (*x509.Certificate, error) {
+	var vOpts x509.VerifyOptions
+	var leaf *x509.Certificate
+	trustedCAPool := x509.NewCertPool()
+
+	// Require Leaf as first cert in bundle
+	if len(certBundle) > 0 {
+		leaf = certBundle[0]
+	} else {
+		return nil, fmt.Errorf("invalid ocsp ca configuration")
+	}
+
+	// Allow Issuer to be configured as second cert in bundle
+	if len(certBundle) > 1 {
+		// The operator may have misconfigured the cert bundle
+		issuerCandidate := certBundle[1]
+		err := issuerCandidate.CheckSignature(leaf.SignatureAlgorithm, leaf.RawTBSCertificate, leaf.Signature)
+		if err != nil {
+			return nil, fmt.Errorf("invalid issuer configuration: %w", err)
+		} else {
+			return issuerCandidate, nil
+		}
+	}
+
+	// Operator did not provide the Leaf Issuer in cert bundle second position
+	// so we will attempt to create at least one ordered verified chain from the
+	// trusted CA pool.
+
+	// Specify CA trust store to validator; if unset, system trust store used
+	if len(trustedCAs) > 0 {
+		for _, ca := range trustedCAs {
+			trustedCAPool.AddCert(ca)
+		}
+		vOpts.Roots = trustedCAPool
+	}
+
+	return certstore.GetLeafIssuer(leaf, vOpts), nil
+}
+
+// getOCSPIssuer determines an issuer certificate from the cert (bundle) or the file-based CA trust store
+func getOCSPIssuer(caFile string, chain [][]byte) (*x509.Certificate, error) {
+	var issuer *x509.Certificate
+	var trustedCAs []*x509.Certificate
+	var certBundle []*x509.Certificate
+	var err error
+
+	// FIXME(tgb): extend if pluggable CA store provider added to NATS (i.e. other than PEM file)
+
+	// Non-system default CA trust store passed
+	if caFile != _EMPTY_ {
+		trustedCAs, err = parseCertPEM(caFile)
+		if err != nil {
+			return nil, fmt.Errorf("failed to parse ca_file: %v", err)
+		}
+	}
+
+	// Specify bundled intermediate CA store
+	for _, certBytes := range chain {
+		cert, err := x509.ParseCertificate(certBytes)
+		if err != nil {
+			return nil, fmt.Errorf("failed to parse cert: %v", err)
+		}
+		certBundle = append(certBundle, cert)
+	}
+
+	issuer, err = getOCSPIssuerLocally(trustedCAs, certBundle)
+	if err != nil || issuer == nil {
+		return nil, fmt.Errorf("no issuers found")
+	}
+
+	if !issuer.IsCA {
+		return nil, fmt.Errorf("%s invalid ca basic constraints: is not ca", issuer.Subject)
+	}
+	return issuer, nil
+}
+
+func ocspStatusString(n int) string {
+	switch n {
+	case ocsp.Good:
+		return "good"
+	case ocsp.Revoked:
+		return "revoked"
+	default:
+		return "unknown"
+	}
+}
+
+func validOCSPResponse(r *ocsp.Response) error {
+	// Time validation not handled by ParseResponse.
+	// https://tools.ietf.org/html/rfc6960#section-4.2.2.1
+	if !r.NextUpdate.IsZero() && r.NextUpdate.Before(time.Now()) {
+		t := r.NextUpdate.Format(time.RFC3339Nano)
+		return fmt.Errorf("invalid ocsp NextUpdate, is past time: %s", t)
+	}
+	if r.ThisUpdate.After(time.Now()) {
+		t := r.ThisUpdate.Format(time.RFC3339Nano)
+		return fmt.Errorf("invalid ocsp ThisUpdate, is future time: %s", t)
+	}
+
+	return nil
+}