upstream/upstream-jitsi-videobridge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

snapshot of https://github.com/jitsi/jitsi-videobridge.git @ fcbf741c646bc237d2ed85a1e86910532532b831 (2026-09-15)
Some checks failed
Java CI with Maven / Java 17 (push) Failing after 7m2s
Details
Java CI with Maven / Java 21 (push) Failing after 5m0s
Details

Browse Source
This commit is contained in:
upstream-mirror
2026-09-15 16:12:38 +05:00
commit e1257b7a05
647 changed files with 110946 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
.editorconfig Normal file
View File

@@ -0,0 +1,12 @@
[*.{kt,kts}]
# Keep constructor parameters wrapped one per line, and kotest specs unindented.
ktlint_standard_class-signature = disabled
max_line_length=120
ktlint_code_style = intellij_idea
# I find trailing commas annoying
ktlint_standard_trailing-comma-on-call-site = disabled
ktlint_standard_trailing-comma-on-declaration-site = disabled
# This forbids underscores in package names, which we use
ktlint_standard_package-name = disabled

1
.gitattributes vendored Normal file
View File

@@ -0,0 +1 @@
*.java text eol=lf diff=java

27
.github/ISSUE_TEMPLATE/bug-report.md vendored Normal file
View File

@@ -0,0 +1,27 @@
---
name: Bug Report
about: Before posting, please make sure you check https://community.jitsi.org
---
*This Issue tracker is only for reporting bugs and tracking code related issues.*
Before posting, please make sure you check community.jitsi.org to see if the same or similar bugs have already been discussed. General questions, installation help, and feature requests can also be posted to community.jitsi.org.
## Description
---
## Current behavior
---
## Expected Behavior
---
## Possible Solution
---
## Steps to reproduce
---
# Environment details
---

11
.github/dependabot.yml vendored Normal file
View File

@@ -0,0 +1,11 @@
version: 2
updates:
- package-ecosystem: "maven"
directories:
- "/"
- "/rtp"
- "/jitsi-media-transform"
- "/jvb"
schedule:
interval: "weekly"
open-pull-requests-limit: 10

44
.github/workflows/maven.yml vendored Normal file
View File

@@ -0,0 +1,44 @@
# This workflow will build a Java project with Maven
# For more information see: https://help.github.com/actions/language-and-framework-guides/building-and-testing-java-with-maven
name: Java CI with Maven
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
env:
# Java version to use for the release
RELEASE_JAVA_VERSION: 17
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
java: [ 17, 21 ]
name: Java ${{ matrix.java }}
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: Set up Java ${{ matrix.java }}
uses: actions/setup-java@c1e323688fd81a25caa38c78aa6df2d33d3e20d9 # v4
with:
distribution: temurin
java-version: ${{ matrix.java }}
cache: maven
- name: Build with Maven
run: ./resources/build.sh -Pcoverage -B
- name: Upload coverage report
if: matrix.java == env.RELEASE_JAVA_VERSION
uses: codecov/codecov-action@b9fd7d16f6d7d1b5d2bec1a2887e65ceed900238 # v4
with:
token: ${{ secrets.CODECOV_TOKEN }}

16
.gitignore vendored Normal file
View File

@@ -0,0 +1,16 @@
/classes/
/dist/
/junit-reports/
/.settings/
target/
/openfire/target/
*.class
/.classpath
*.jar
/.project
*.swp
**/.idea/
*.iml
.mvn/
.kotlintest
**/.DS_Store

154
CLAUDE.md Normal file
View File

@@ -0,0 +1,154 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
Jitsi Videobridge (JVB) is a WebRTC-compatible Selective Forwarding Unit (SFU) that routes media streams between participants in video conferences. It is a backend component of the Jitsi Meet stack.
## Build System & Commands
This project uses Maven with Java 11+ and Kotlin. Requires Maven 3.6.3+.
### Build Commands
```bash
# Build the entire project
mvn install
# Build without running tests
mvn install -DskipTests
# Run all tests
mvn test
# Run tests in a specific module
cd jvb && mvn test
# Run a specific test class
mvn test -Dtest=ClassName
# Run a specific test method
mvn test -Dtest=ClassName#methodName
# Build package (creates archive in jvb/target/)
mvn install
# Output: jvb/target/jitsi-videobridge-2.3-SNAPSHOT-archive.zip
```
### Code Quality Commands
```bash
# Run checkstyle (Java formatting)
mvn checkstyle:check
# Run ktlint (Kotlin formatting, must be run inside a submodule directory, e.g. ./jvb/; sibling modules must be
# resolvable, so run "mvn install -DskipTests" from the root first if they are not)
mvn exec:exec@ktlint-check
# Auto-format Kotlin code (must be run inside a submodule directory, e.g. ./jvb/)
mvn exec:exec@ktlint-format
```
## Project Structure
This is a multi-module Maven project with three main modules:
### 1. `rtp/` - RTP Protocol Layer
Low-level RTP/RTCP packet handling and parsing.
### 2. `jitsi-media-transform/` - Media Processing
Core media transformation logic including:
- Codec-specific processing (VP8, VP9, AV1, H.264)
- Simulcast and SVC (Scalable Video Coding) handling
- Bandwidth allocation and quality adaptation
- SRTP encryption/decryption
### 3. `jvb/` - Main Videobridge Application
The main application that ties everything together:
**Key Java Classes:**
- `Videobridge.java` - Main service managing conferences
- `Conference.java` - Represents a single video conference, manages endpoints and media routing
- `Endpoint` (Kotlin) - Represents a participant in a conference
- `EncodingsManager.java` - Manages simulcast encodings for endpoints
**Key Kotlin Packages:**
- `cc/allocation/` - Bandwidth allocation and bitrate control
- `cc/vp8/`, `cc/vp9/`, `cc/av1/` - Codec-specific frame processing
- `load_management/` - CPU and packet rate based load management
- `relay/` - Multi-region support (Octo relay)
- `websocket/` - WebSocket transport for signaling
- `xmpp/` - XMPP/Colibri signaling
**Main Entry Point:** `org.jitsi.videobridge.Main.kt`
## Architecture Principles
### Media Forwarding Pipeline
1. **Ingress**: Media arrives at endpoints via ICE/DTLS/SRTP
2. **Processing**: Packets flow through codec-specific handlers that understand simulcast/SVC structure
3. **Bandwidth Allocation**: `BitrateController` determines which quality layers to forward to each receiver
4. **Rewriting**: RTP headers (sequence numbers, timestamps, picture IDs) are rewritten to make stream switches transparent
5. **Egress**: Forwarded packets are sent to receiving endpoints
### Simulcast & SVC Handling
The bridge supports temporal and spatial scalability in VP8, VP9, and AV1. It performs transparent stream switching by rewriting RTP sequence numbers, timestamps, and codec-specific picture IDs. See `doc/svc.md` for detailed rewriting principles.
**Key Constraint**: Switches must be completely transparent to receivers - no gaps in sequence numbers, continuous timestamps, and continuous picture IDs.
### Configuration System
Uses Typesafe Config (HOCON format):
- `reference.conf` contains all default values with documentation
- User config in `/etc/jitsi/videobridge/jvb.conf` or `~/.jvb/jvb.conf` overrides defaults
- Config is accessed via `jitsi-metaconfig` library with type-safe property delegates
## Code Style & Conventions
### Java Code
- **Brace Style**: Opening braces on new line for methods, classes, control structures
- **Line Length**: Maximum 120 characters
- **Indentation**: 4 spaces, NO tabs
- Validated by Checkstyle (see `jvb/checkstyle.xml`)
### Kotlin Code
- Follows ktlint conventions (standard Kotlin style)
- **Line Length**: Maximum 120 characters
- ktlint runs from Maven via exec-maven-plugin (version pinned by `ktlint.version` in the root pom); `mvn verify` runs the check
### Testing
- Uses JUnit 5 for new tests (JUnit 4 vintage engine for legacy tests)
- Kotlin tests use Kotest framework
- Mocking with MockK (Kotlin) or Mockito (Java)
## Important Implementation Notes
### Thread Safety
- `Conference.endpointsById` is a `ConcurrentHashMap` but writes must be synchronized on the map itself to keep it in sync with `endpointsCache`
- Most core components use concurrent data structures but may require explicit synchronization for compound operations
### Performance Considerations
- `endpointsCache` is a read-only cache updated on writes - used in hot path (per-packet processing)
- Buffer pools (`ByteBufferPool`) are used throughout to reduce GC pressure
- Packet queues are bounded to prevent memory exhaustion under load
### Logging
- Uses `org.jitsi.utils.logging2.Logger` (custom logging framework)
- Can optionally redact remote IP addresses (controlled by `videobridge.redact-remote-addresses`)
### Dependencies
- **ice4j**: ICE/STUN/TURN implementation
- **jicoco**: Common Jitsi infrastructure (config, metrics, REST)
- **Jetty**: HTTP server for REST APIs and WebSockets
- **Smack**: XMPP client library
## Testing & CI
Tests are organized by module. When implementing features, consider whether tests are needed and discuss with the user if implementation is non-trivial.
Run module-specific tests:
```bash
cd jvb && mvn test
cd jitsi-media-transform && mvn test
cd rtp && mvn test
```

63
CONFIG.md Normal file
View File

@@ -0,0 +1,63 @@
# Configuration
##### 10/14/19
We've begun the migration to a new configuration framework. The 'old' method (the `sip-communicator.properties` file and program arguments) will be supported throughout the transition and the bridge will be fully backwards-compatible with the old config. No changes are necessary until we pull the plug on the old config completely, and we'll give ample warning before that time. It's also worth noting that any new properties will only be supported in the new config files.
Moving forward, we'll be using lightbend's [config library](https://github.com/lightbend/config). Their GitHub page gives lots of details, but below are the changes most relevant to configuring the JVB.
### Configuration sources
During the transition, we'll continue to read from the original locations (`sip-communicator.properties` and program arguments) for all configuration properties, falling back to the new `reference.conf` file for defaults. The `reference.conf` file holds defaults for all configuration properties, and shouldn't be modified in a deployment--though it may be interesting for developers who want to see the defaults.
To override values, users can either put a file named `application.conf` in the classpath or pass a path to a file as a system property via `-Dconfig.file=/path/to/config.conf`. Values in the file given will override ones in `reference.conf`. The lightbend docs do a good job of describing [how configuration is loaded](https://github.com/lightbend/config#standard-behavior), but here's an example of overriding a configuration value for JVB:
### Overriding default values
Say we want to override some of the default values in `reference.conf`, namely: the XMPP API server values, the health check interval, and whether or not onstage video suspension is enabled. We'll create an `application.conf` files with the values we want to change:
```
videobridge {
health {
// Override the health check interval
interval=60 seconds
}
cc {
// Override the onstage video suspension setting
onstage-video-suspension-enabled=true
}
}
```
You can override just the values you want to change, all others will fallback to the defaults set in `reference.conf`
### Migrating from old config
Below are the mappings from old property values to new ones. Note: for new property names, the 'flat paths' are given, i.e. a value of `videobridge.health.interval` corresponds to the setting:
```
videobridge {
health {
interval
}
}
```
This list will be updated as properties are migrated:
| Old property name | New property name | Notes |
| -------- | ------- | ------- |
| org.jitsi.videobridge.health.INTERVAL | videobridge.health.interval | The new config models this as a duration, rather than an amount of milliseconds |
| org.jitsi.videobridge.health.TIMEOUT | videobridge.health.timeout | The new config models this as a duration, rather than an amount of milliseconds |
| org.jitsi.videobridge.health.STICKY_FAILURES | videobridge.health.sticky-failures | |
| org.jitsi.videobridge.EndpointConnectionStatus.FIRST_TRANSFER_TIMEOUT | videobridge.ep-connection-status.first-transfer-timeout | The new config models this as a duration, rather than an amount of milliseconds |
| org.jitsi.videobridge.EndpointConnectionStatus.MAX_INACTIVITY_LIMIT | videobridge.ep-connection-status.max-inactivity-limit | The new config models this as a duration, rather than an amount of milliseconds |
| org.jitsi.videobridge.BWE_CHANGE_THRESHOLD_PCT | videobridge.cc.bwe-change-threshold-pct | |
| org.jitsi.videobridge.THUMBNAIL_MAX_HEIGHT | videobridge.cc.thumbnail-max-height-px | |
| org.jitsi.videobridge.ONSTAGE_PREFERRED_HEIGHT | videobridge.cc.onstage-preferred-height-px | |
| org.jitsi.videobridge.ONSTAGE_PREFERRED_FRAME_RATE | videobridge.cc.onstage-preferred-framerate | |
| org.jitsi.videobridge.ENABLE_ONSTAGE_VIDEO_SUSPEND | videobridge.cc.enable-onstage-video-suspend | |
| org.jitsi.videobridge.TRUST_BWE | videobridge.cc.trust-bwe | |
| org.jitsi.videobridge.PADDING_PERIOD_MS | videobridge.cc.padding-period | The new config models this as a duration, rather than an amount of milliseconds |
| org.jitsi.videobridge.DISABLE_RTX_PROBING | n/a | This property has been deprecated |
| org.jitsi.videobridge.rest.COLIBRI_WS_DISABLE | videobridge.websockets.enabled | The semantics of this property have been inverted (disable -> enable) |
| org.jitsi.videobridge.rest.COLIBRI_WS_DOMAIN | videobridge.websockets.domain | |
| org.jitsi.videobridge.rest.COLIBRI_WS_TLS | videobridge.websockets.tls | |
| org.jitsi.videobridge.rest.COLIBRI_WS_SERVER_ID | videobridge.websockets.server-id | |
| org.jitsi.videobridge.SINGLE_PORT_HARVESTER_PORT | videobridge.ice.udp.port | |
| org.jitsi.videobridge.ICE_UFRAG_PREFIX | videobridge.ice.ufrag-prefix | |
| org.jitsi.videobridge.KEEP_ALIVE_STRATEGY | videobridge.ice.keep-alive-strategy | |

202
LICENSE Normal file
View File

@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

53
README.md Normal file
View File

@@ -0,0 +1,53 @@
# Intro
Jitsi Videobridge is a WebRTC-compatible Selective Forwarding Unit (SFU), i.e. a
multimedia router. It is one of the backend components in the [Jitsi Meet](https://github.com/jitsi/jitsi-meet) stack.
You can find more documentation in the
[doc/ directory in the source tree](https://github.com/jitsi/jitsi-videobridge/tree/master/doc) and in the
[Jitsi Meet Handbook](https://jitsi.github.io/handbook/).
If you have questions about the project, please post on the [Jitsi Community Forum](https://community.jitsi.org/).
GitHub issues are only used to track actionable items.
# Packages
## Debian/Ubuntu
You can download binary packages for Debian/Ubuntu:
* [stable](https://download.jitsi.org/stable/) ([instructions](https://jitsi.org/downloads/ubuntu-debian-installations-instructions/))
* [testing](https://download.jitsi.org/testing/) ([instructions](https://jitsi.org/downloads/ubuntu-debian-installations-instructions-for-testing/))
* [nightly](https://download.jitsi.org/unstable/) ([instructions](https://jitsi.org/downloads/ubuntu-debian-installations-instructions-nightly/))
## Building your own package
You can build a custom package with just `mvn install` in the root directory of the repo. Look for the package in
`jvb/target/jitsi-videobridge-2.1-SNAPSHOT-archive.zip`.
# Running locally
You can run jitsi-videobridge locally with maven (or in your IDE). First create a `~/.jvb/jvb.conf` to configure the
environment to connect to and other options (see
[reference.conf](https://github.com/jitsi/jitsi-videobridge/blob/master/jvb/src/main/resources/reference.conf) for the
available options).
```sh
JVB_HOME="/path/to/the/cloned/repo"
JVB_CONFIG_DIR_LOCATION="~/"
JVB_CONFIG_DIR_NAME=".jvb"
JVB_CONFIG_FILE="$JVB_CONFIG_DIR_LOCATION/$JVB_JVB_CONFIG_DIR_NAME/jvb.conf"
mvn compile exec:exec -Dexec.executable=java -Dexec.args="-cp %classpath org.jitsi.videobridge.MainKt -Djava.library.path=$JVB_HOME/lib/native/linux-64 -Djava.util.logging.config.file=$JVB_HOME/lib/logging.properties -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=$JVB_CONFIG_DIR_LOCATION -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=$JVB_CONFIG_DIR_NAME -Dconfig.file=$JVB_CONFIG_FILE"
```
# Configuration
Application level configuration comes from a config file, usually installed in `/etc/jitsi/videobridge/jvb.conf`. The
values in that file override the defaults defined in
[reference.conf](https://github.com/jitsi/ice4j/blob/master/src/main/resources/reference.conf).
## Debian
On debian systems the `/etc/jitsi/videobridge/config` file can be used to set configuration for the Java virtual machine.
Notable examples:
```commandline
# Increase the java heap to 8GB
VIDEOBRIDGE_MAX_MEMORY=8192m
# Change the garbage collector (defaults to G1GC)
VIDEOBRIDGE_GC_TYPE=G1GC
```

9
SECURITY.md Normal file
View File

@@ -0,0 +1,9 @@
# Security
## Reporting security issues
We take security very seriously and develop all Jitsi projects to be secure and safe.
If you find (or simply suspect) a security issue in any of the Jitsi projects, please report it to us via [HackerOne](https://hackerone.com/8x8-bounty) or send us an email to security@jitsi.org.
**We encourage responsible disclosure for the sake of our users, so please reach out before posting in a public space.**

3
config/20-jvb-udp-buffers.conf Normal file
View File

@@ -0,0 +1,3 @@
# this sets the max, so that we can bump the JVB UDP single port buffer size.
net.core.rmem_max=10485760
net.core.netdev_max_backlog=100000

12
config/logrotate Normal file
View File

@@ -0,0 +1,12 @@
/var/log/jitsi/jvb.log {
hourly
size 100M
missingok
rotate 24
compress
delaycompress
notifempty
copytruncate
su jvb jitsi
}

2
debian/README vendored Normal file
View File

@@ -0,0 +1,2 @@
Jitsi Videobridge - WebRTC compatible Selective Forwarding Unit
(SFU) for multiuser video communication

8
debian/README.Debian vendored Normal file
View File

@@ -0,0 +1,8 @@
jitsi-videobridge for Debian
---------------------
If you want a WebRTC videoconferencing solution you have to install
jitsi-meet package which uses Jitsi Videobridge as a SFU (Selective
Forwarding Unit)
-- Yasen Pramatarov <yasen@bluejimp.com> Tue, 22 Apr 2014 21:40:37 +0300

6
debian/changelog vendored Normal file
View File

@@ -0,0 +1,6 @@
jitsi-videobridge2 (2.1-0-g676fb3d-1) unstable; urgency=low
* Initial release.
-- Damian Minkov <damencho@jitsi.org> Fri, 17 Jan 2020 15:51:09 +0000

1
debian/compat vendored Normal file
View File

@@ -0,0 +1 @@
12

8
debian/config vendored Normal file
View File

@@ -0,0 +1,8 @@
#!/bin/sh -e
# Source debconf library.
. /usr/share/debconf/confmodule
# server hostname
db_input critical jitsi-videobridge/jvb-hostname || true
db_go

20
debian/control vendored Normal file
View File

@@ -0,0 +1,20 @@
Source: jitsi-videobridge2
Section: net
Priority: extra
Maintainer: Jitsi Team <dev@jitsi.org>
Uploaders: Emil Ivov <emcho@jitsi.org>, Damian Minkov <damencho@jitsi.org>
Build-Depends: debhelper (>= 9), dh-systemd, openjdk-17-jdk | openjdk-21-jdk, maven
Standards-Version: 3.9.3
Homepage: https://jitsi.org/videobridge
Package: jitsi-videobridge2
Replaces: jitsi-videobridge
Conflicts: jitsi-videobridge (<= 1400-1)
Architecture: all
Pre-Depends: openjdk-17-jre-headless | openjdk-17-jre | openjdk-21-jre-headless | openjdk-21-jre, libssl3 | libssl1.1
Depends: ${misc:Depends}, procps, uuid-runtime, ruby-hocon
Recommends: libpcap0.8
Description: WebRTC compatible Selective Forwarding Unit (SFU)
Jitsi Videobridge is a WebRTC compatible Selective Forwarding Unit
(SFU) for multiuser video communication. It is an essential part
of Jitsi Meet

13
debian/copyright vendored Normal file
View File

@@ -0,0 +1,13 @@
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: Jitsi Videobridge
Upstream-Contact: Emil Ivov <emcho@jitsi.org>
Source: https://github.com/jitsi/jitsi-videobridge
Files: *
Copyright: 2015-present Atlassian Pty Ltd
License: Apache-2.0
License: Apache-2.0
On Debian systems, the full text of the Apache
License version 2 can be found in the file
'/usr/share/common-licenses/Apache-2.0'.

0
debian/docs vendored Normal file
View File

118
debian/init.d vendored Normal file
View File

@@ -0,0 +1,118 @@
#! /bin/sh
#
# INIT script for Jitsi Videobridge
# Version: 1.0 01-May-2014 yasen@bluejimp.com
#
### BEGIN INIT INFO
# Provides: jitsi-videobridge
# Required-Start: $local_fs $remote_fs
# Required-Stop: $local_fs $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Jitsi Videobridge
# Description: WebRTC compatible Selective Forwarding Unit (SFU)
### END INIT INFO
. /lib/lsb/init-functions
# Include videobridge defaults if available
if [ -f /etc/jitsi/videobridge/config ]; then
. /etc/jitsi/videobridge/config
fi
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/share/jitsi-videobridge/jvb.sh
NAME=jvb
USER=jvb
# A tmpfs backed directory just for the JVB process. This is introduced
# to hold packet arrival times, but it may be otherwise useful in the future.
TMPPATH=/var/run/jitsi-videobridge
PIDFILE=/var/run/jitsi-videobridge.pid
LOGFILE=/var/log/jitsi/jvb.log
DESC=jitsi-videobridge
if [ ! -d "$TMPPATH" ]; then
mkdir "$TMPPATH"
chown $USER:adm "$TMPPATH"
fi
if [ ! $JVB_HOST ]; then
JVB_HOST=localhost
fi
# TODO: remove this when support for --apis is removed
DAEMON_OPTS="$JVB_OPTS"
if [ ! -x $DAEMON ] ;then
echo "Daemon not executable: $DAEMON"
exit 1
fi
set -e
stop() {
if [ -f $PIDFILE ]; then
PID=$(cat $PIDFILE)
fi
echo -n "Stopping $DESC: "
if [ $PID ]; then
kill $PID || true
rm $PIDFILE || true
echo "$NAME stopped."
else
echo "$NAME doesn't seem to be running."
fi
}
start() {
if [ -f $PIDFILE ]; then
echo "$DESC seems to be already running, we found pidfile $PIDFILE."
exit 1
fi
echo -n "Starting $DESC: "
DAEMON_START_CMD="JAVA_SYS_PROPS=\"$JAVA_SYS_PROPS\" exec $DAEMON $DAEMON_OPTS < /dev/null >> $LOGFILE 2>&1"
AUTHBIND_CMD=""
if [ "$AUTHBIND" = "yes" ]; then
AUTHBIND_CMD="/usr/bin/authbind --deep /bin/bash -c "
DAEMON_START_CMD="'$DAEMON_START_CMD'"
fi
start-stop-daemon --start --quiet --background --chuid $USER --make-pidfile --pidfile $PIDFILE \
--exec /bin/bash -- -c "$AUTHBIND_CMD $DAEMON_START_CMD"
echo "$NAME started."
}
reload() {
echo 'Not yet implemented.'
}
status() {
status_of_proc -p $PIDFILE java "$NAME" && exit 0 || exit $?
}
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
stop
start
;;
reload)
reload
;;
force-reload)
reload
;;
status)
status
;;
*)
N=/etc/init.d/$NAME
echo "Usage: $N {start|stop|restart|reload|status}" >&2
exit 1
;;
esac
exit 0

10
debian/install vendored Normal file
View File

@@ -0,0 +1,10 @@
jvb/target/dependency/* usr/share/jitsi-videobridge/lib
jvb/lib/videobridge.rc usr/share/jitsi-videobridge/lib
jvb/lib/logging.properties etc/jitsi/videobridge
jitsi-videobridge.jar usr/share/jitsi-videobridge
jvb/resources/jvb.sh usr/share/jitsi-videobridge
resources/graceful_shutdown.sh usr/share/jitsi-videobridge
resources/collect-dump-logs.sh usr/share/jitsi-videobridge
jitsi-videobridge etc/logrotate.d
config/20-jvb-udp-buffers.conf etc/sysctl.d

29
debian/jitsi-videobridge2.service vendored Normal file
View File

@@ -0,0 +1,29 @@
[Unit]
Description=Jitsi Videobridge
After=network-online.target
Wants=network-online.target
[Service]
SuccessExitStatus=143
# configuration error prevents restart loops
RestartPreventExitStatus=78
# allow bind to 80 and 443
AmbientCapabilities=CAP_NET_BIND_SERVICE
EnvironmentFile=/etc/jitsi/videobridge/config
Environment=LOGFILE=/var/log/jitsi/jvb.log
User=jvb
RuntimeDirectory=jitsi-videobridge
RuntimeDirectoryMode=0750
PIDFile=/var/run/jitsi-videobridge/jitsi-videobridge.pid
# more threads for this process
TasksMax=65000
# allow more open files for this process
LimitNPROC=65000
LimitNOFILE=65000
ExecStart=/bin/bash -c "exec /usr/share/jitsi-videobridge/jvb.sh ${JVB_OPTS} < /dev/null >> ${LOGFILE} 2>&1"
ExecStartPost=/bin/bash -c "echo $MAINPID > /var/run/jitsi-videobridge/jitsi-videobridge.pid"
Restart=on-failure
RestartSec=2
[Install]
WantedBy=multi-user.target

40
debian/manpage.1 vendored Normal file
View File

@@ -0,0 +1,40 @@
.\" Hey, EMACS: -*- nroff -*-
.\" First parameter, NAME, should be all caps
.\" Second parameter, SECTION, should be 1-8, maybe w/ subsection
.\" other parameters are allowed: see man(7), man(1)
.TH JITSI-VIDEOBRIDGE SECTION "April 09, 2014"
.\" Please adjust this date whenever revising the manpage.
.\"
.\" Some roff macros, for reference:
.\" .nh disable hyphenation
.\" .hy enable hyphenation
.\" .ad l left justify
.\" .ad b justify to both left and right margins
.\" .nf disable filling
.\" .fi enable filling
.\" .br insert line break
.\" .sp <n> insert n+1 empty lines
.\" for manpage-specific macros, see man(7)
.SH NAME
jitsi-videobridge \- WebRTC compatible Selective Forwarding Unit
(SFU) for multiuser video communication
.SH SYNOPSIS
.B /usr/share/jitsi-videobridge/jvb.sh
.RI [ options ] ...
.br
.B /etc/init.d/jitsi-videobridge (start|stop|restart)
.SH DESCRIPTION
This manual page documents briefly the
.B jitsi-videobridge
service.
.PP
.\" TeX users may be more comfortable with the \fB<whatever>\fP and
.\" \fI<whatever>\fP escape sequences to invoke bold face and italics,
.\" respectively.
\fBjitsi-videobridge\fP is a Selective Forwarding Unit (SFU) for
building applications for WebRTC compatible video conferences.
.SH AUTHOR
jitsi-videobridge was written by Jitsi.org and Bluejimp.com.
.PP
This manual page was written by Yasen Pramatarov <yasen@bluejimp.com>,
for the Debian project (and may be used by others).

1
debian/po/POTFILES.in vendored Normal file
View File

@@ -0,0 +1 @@
[type: gettext/rfc822deb] templates

30
debian/po/templates.pot vendored Normal file
View File

@@ -0,0 +1,30 @@
# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER
# This file is distributed under the same license as the PACKAGE package.
# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: jitsi-videobridge\n"
"Report-Msgid-Bugs-To: jitsi-videobridge@packages.debian.org\n"
"POT-Creation-Date: 2014-08-11 10:41+0200\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"Language: \n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=CHARSET\n"
"Content-Transfer-Encoding: 8bit\n"
#. Type: string
#. Description
#: ../templates:1001
msgid "Hostname:"
msgstr ""
#. Type: string
#. Description
#: ../templates:1001
msgid "The jitsi-videobridge package needs the DNS hostname of your instance."
msgstr ""

184
debian/postinst vendored Normal file
View File

@@ -0,0 +1,184 @@
#!/bin/bash
# postinst script for jitsi-videobridge
set -e
case "$1" in
configure)
CONFIG="/etc/jitsi/videobridge/config"
# we don't want to regenerate config on upgrade
OLDCONFIG="false"
# migrate any old config found
if [ -f "/etc/default/jitsi-videobridge" ]; then
mv /etc/default/jitsi-videobridge $CONFIG
fi
if [ -f $CONFIG ]; then
. $CONFIG
if [ -n "$JVB_HOSTNAME" ] && [ -n "$JVB_SECRET" ]; then
OLDCONFIG="true"
fi
fi
# debconf hostname question
. /usr/share/debconf/confmodule
# the default vars
db_get jitsi-videobridge/jvb-hostname
JVB_HOSTNAME_IN=$(echo "$RET" | xargs echo -n)
# generate config on new install or when we are reconfiguring
# and all install is different than current one
if [ "$OLDCONFIG" = "false" ] || [ "$JVB_HOSTNAME" != "$JVB_HOSTNAME_IN" ]; then
JVB_HOSTNAME="$JVB_HOSTNAME_IN"
if [ "$OLDCONFIG" = "false" ]; then
# 8-chars random secret, alternative to pwgen 8
db_get jitsi-videobridge/jvbsecret
if [ -n "$RET" ]; then
JVB_SECRET="$RET"
else
JVB_SECRET=`head -c 8 /dev/urandom | tr '\0-\377' 'a-zA-Z0-9a-zA-Z0-9a-zA-Z0-9a-zA-Z0-9@@@@####'`
fi
fi
fi
# Store it, so we can use it if xmpp server is on same machine
# we are currently doing this on every upgrade in order to be able to successfully upgrade
# from old packages, otherwise we can do it only when secret is first time generated
db_set jitsi-videobridge/jvbsecret $JVB_SECRET
# and we're done with debconf
db_stop
HOCON_CONFIG="/etc/jitsi/videobridge/jvb.conf"
# let's check whether there is a setting in the $CONFIG
# for home folder and logging props file, if missing add it
if ! grep -q "JAVA_SYS_PROPS" "$CONFIG"; then
echo >> $CONFIG
echo '# adds java system props that are passed to jvb (default are for home and logging config file)' >> $CONFIG
echo "JAVA_SYS_PROPS=\"-Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi\
-Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge\
-Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi\
-Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties\"" >> $CONFIG
fi
# Updates config so new and old installs will start using the new config file
if ! grep -q "\-Dconfig.file" "$CONFIG"; then
sed -i 's|JAVA_SYS_PROPS="|JAVA_SYS_PROPS="-Dconfig.file='"$HOCON_CONFIG"' |g' $CONFIG
fi
# unused old parameter, systemd unit files does not resolve bash variables,
# and this breaks startup script
sed -i 's/\$JVB_EXTRA_JVM_PARAMS//g' $CONFIG
if [ ! -f $HOCON_CONFIG ]; then
echo "Generating a default hocon config"
echo "videobridge {
apis.xmpp-client.configs {
shard {
HOSTNAME=localhost
DOMAIN=\"auth.$JVB_HOSTNAME\"
USERNAME=jvb
PASSWORD=\"$JVB_SECRET\"
MUC_JIDS=\"jvbbrewery@internal.auth.$JVB_HOSTNAME\"
MUC_NICKNAME=$(uuidgen)
}
}
}
ice4j {
harvest {
mapping {
aws {
enabled = false
}
stun {
addresses = [\"meet-jit-si-turnrelay.jitsi.net:443\"]
}
}
}
}" >> $HOCON_CONFIG
fi
# The REST API is disabled by default. It it was previously enabled via --apis, enable it in jvb.conf
if grep -v '^\s*#' /etc/jitsi/videobridge/config | grep -- '--apis=' | grep -i rest >/dev/null 2>&1 ;then
if ! hocon -f $HOCON_CONFIG get "videobridge.apis.rest.enabled" > /dev/null 2>&1 ;then
hocon -f $HOCON_CONFIG set "videobridge.apis.rest.enabled" "true"
fi
# Remove it from /etc/jitsi/videobridge/config to prevent it perpetually updating jvb.conf
sed -i 's/.*--apis.*//' $CONFIG
fi
# we don't want to start the daemon as root
if ! getent group jitsi > /dev/null ; then
groupadd jitsi
fi
if ! getent passwd jvb > /dev/null ; then
useradd -r -g jitsi --shell /bin/bash --create-home -d /usr/share/jitsi-videobridge jvb
fi
if ! groups jvb | grep '\bjitsi\b' > /dev/null ; then
usermod -g jitsi jvb
fi
mkdir -p /usr/share/jitsi-videobridge
# we claim the home folder of jvb in case it is owned by someone else
OWNER=$(stat -c '%U' /usr/share/jitsi-videobridge)
GROUP=$(stat -c '%G' /usr/share/jitsi-videobridge)
if ! dpkg-statoverride --list /usr/share/jitsi-videobridge/* >/dev/null && [ "$OWNER:$GROUP" != "jvb:jitsi" ]; then
chown -R jvb:jitsi /usr/share/jitsi-videobridge
OWNER=jvb
GROUP=jitsi
fi
CONFIG_DIR=$(dirname $CONFIG)
if ! dpkg-statoverride --list "$CONFIG_DIR" >/dev/null; then
chown -R jvb:jitsi "$CONFIG_DIR"
chmod 750 "$CONFIG_DIR"
fi
# die logz
if [ ! -d /var/log/jitsi ]; then
mkdir -p /var/log/jitsi
fi
chown $OWNER:$GROUP /var/log/jitsi
chmod 770 /var/log/jitsi
ls /var/log/jitsi/jvb* 1>/dev/null 2>&1 && chown -f -R $OWNER:$GROUP /var/log/jitsi/jvb*
ls /var/log/jitsi/jvb* 1>/dev/null 2>&1 && chmod -f -R 640 /var/log/jitsi/jvb*
# ensure videobridge is not running - it will be started at the end
if [ -d /run/systemd/system ]; then
systemctl stop jitsi-videobridge2.service >/dev/null || true
fi
# clean up old jvb group
if getent group jvb > /dev/null; then
groupdel jvb
fi
# load the UDP buffer sizes required by the JVB. As reported in
# https://github.com/jitsi/jitsi-videobridge/issues/461
# OpenVZ containers do not typically allow permission to modify the
# kernel with sysctl config values.
sysctl --system || true
;;
abort-upgrade|abort-remove|abort-deconfigure)
;;
*)
echo "postinst called with unknown argument \`$1'" >&2
exit 1
;;
esac
# dh_installdeb will replace this with shell code automatically
# generated by other debhelper scripts.
#DEBHELPER#
exit 0

53
debian/postrm vendored Normal file
View File

@@ -0,0 +1,53 @@
#!/bin/bash
# postrm script for jitsi-videobridge
set -e
# Load debconf
. /usr/share/debconf/confmodule
case "$1" in
purge)
# Clear the debconf variable
db_purge
# and then remove the user
if getent passwd jvb > /dev/null ; then
deluser jvb || true
fi
# clear logs
if [ -d "/var/log/jitsi" ]; then
rm -rf "/var/log/jitsi"
fi
rm -f /etc/jitsi/videobridge/config
rm -f /etc/jitsi/videobridge/jvb.conf
rm -f /etc/jitsi/videobridge/sip-communicator.properties
rm -f /etc/jitsi/videobridge/logging.properties
rm -rf /usr/share/jitsi-videobridge/.java
rm -rf /usr/share/jitsi-videobridge/.sip-communicator
# make sure we remove keys we used ('jitsi-videobridge/jvb-hostname')
db_unregister jitsi-videobridge/jvb-hostname || true
;;
remove|upgrade|failed-upgrade|abort-install|abort-upgrade|disappear)
;;
*)
echo "postrm called with unknown argument \`$1'" >&2
exit 1
;;
esac
# dh_installdeb will replace this with shell code automatically
# generated by other debhelper scripts.
#DEBHELPER#
db_stop
exit 0

34
debian/rules vendored Executable file
View File

@@ -0,0 +1,34 @@
#!/usr/bin/make -f
# Uncomment this to turn on verbose mode.
#export DH_VERBOSE=1
#export DEB_VERBOSE_ALL=true
%:
dh $@
override_dh_auto_clean:
mvn -Dmaven.repo.local=${HOME}/.m2/repository clean
override_dh_auto_configure:
# do nothing
override_dh_auto_test:
# do nothing
override_dh_auto_build:
mvn -Dmaven.repo.local=${HOME}/.m2/repository -DskipTests -Dassembly.skipAssembly=true install
mvn -Dmaven.repo.local=${HOME}/.m2/repository dependency:copy-dependencies -DincludeScope=runtime
override_dh_install:
mkdir debian/tmp
cp config/logrotate debian/tmp/jitsi-videobridge
cp jvb/target/jitsi-videobridge*.jar debian/tmp/jitsi-videobridge.jar
dh_install
override_dh_installinit:
dh_installinit --noscripts
override_dh_auto_install:
dh_auto_install
dh_installsystemd --restart-after-upgrade

1
debian/source/format vendored Normal file
View File

@@ -0,0 +1 @@
3.0 (quilt)

10
debian/templates vendored Normal file
View File

@@ -0,0 +1,10 @@
Template: jitsi-videobridge/jvb-hostname
Type: string
Default: localhost
_Description: Hostname:
The jitsi-videobridge package needs the DNS hostname of your instance.
Template: jitsi-videobridge/jvbsecret
Type: password
_Description: The secret used to connect to xmpp server as component
Component secret:

305
doc/allocation.md Normal file
View File

@@ -0,0 +1,305 @@
# Bandwidth Allocation Algorithm
Bandwidth allocation is the process of selecting the set of layers to forward to a specific endpoint (the "receiver"),
or "allocating" the available bandwidth among the available layers. When conditions change, the algorithm is re-run,
and the set of forwarded layers is updated according to the new result.
The overall goal is to provide the most relevant and suitable set of streams to the receiver, given a limited bandwidth.
## Input
### Available bandwidth
The available bandwidth (or the Bandwidth Estimation, BWE) is the estimated bandwidth between the bridge and the
receiver. It is calculated elsewhere, and is only used as input for bandwidth allocation.
### Available sources
This is the list of streams being sent from the other endpoints in the conference. Streams have multiple layers, and
the algorithm selects one layer (or no layers) for each source.
For example, a simulcast sender can encode its video in 3 different encodings, with 3 different frame rates each. This
gives the allocator 9 layers to choose from.
The list of available sources changes when endpoints join or leave the conference, or when they signal a change in their
streams (such as when an endpoint switches from a video camera to screensharing or vice-versa).
### Receiver-specified settings
The following settings are controlled by the receiver, with messages over the "bridge channel".
#### LastN
LastN is the maximum number of video streams that the receiver wants to receive. To effectively stop receiving video
(for example to conserve bandwidth), the receiver can set LastN=0.
#### Selected sources
This is a list of sources to be prioritized first, overriding the natural speech activity order of the endpoints which
own the sources.
For example, if the receiver wants to always receive a screensharing source, regardless of who is speaking
in the conference, it can "select" this source.
#### On-stage sources
This is a list of sources for which allocation should be prioritized up to a higher resolution (since they are
displayed "on stage"). On-stage sources are prioritized higher than selected sources, and in addition:
1. Allocation for them is greedy up to the preferred resolution
([360p by default](https://github.com/jitsi/jitsi-videobridge/blob/master/jvb/src/main/resources/reference.conf#L40))
2. Above the preferred resolution, only frame rates of
[at least 30 fps](https://github.com/jitsi/jitsi-videobridge/blob/master/jvb/src/main/resources/reference.conf#L41)
are considered.
#### Video Constraints
Video constraints are resolution (`maxHeight`) and frame rate (`maxFrameRate`) constraints for each source. These are
"soft" constraints in the sense that the bridge may exceed them in some circumstances (see below).
When set to a negative number, they indicate no constraints.
When set to 0, they signal that no video should be forwarded for the associated source, and this is never exceeded.
When set to a positive number, the algorithm will attempt to select a layer which satisfies the constraints. If no
layers satisfy the constraints, and there is sufficient bandwidth, the algorithm will exceed the constraints and
select the lowest layer. In practice this is relevant only in the case where a sender does not use simulcast (or SVC),
and encodes a single high-resolution stream. Given enough bandwidth, the stream will be forwarded even when the receiver
signaled low constraints.
## Implementation
The bandwidth allocation algorithm is implemented in [BandwidthAllocator](https://github.com/jitsi/jitsi-videobridge/blob/master/jvb/src/main/kotlin/org/jitsi/videobridge/cc/allocation/BandwidthAllocator.kt)
It consists of 3 phases:
### 1. Prioritize
This phase orders the available sources in the desired way. It starts with sources coming from the endpoints ordered by
speech activity (dominant speaker, followed by the previous dominant speaker, etc). Then, it moves the sources from
the endpoints which are NOT sending video to the bottom of the list (this is actually implemented in
[ConferenceSpeechActivity](https://github.com/jitsi/jitsi-videobridge/blob/master/jvb/src/main/java/org/jitsi/videobridge/ConferenceSpeechActivity.java).
Finally, the selected sources are moved to the TOP of the list.
TODO: Update the algorithm, to only move selected endpoint when they are sending video.
### 2. Apply LastN
This phase disables the video sources in the list that are not among the first `LastN`. Note that the effective
`LastN` value comes from the number signaled by the client, potentially also limited by [static](https://github.com/jitsi/jitsi-videobridge/blob/master/jvb/src/main/kotlin/org/jitsi/videobridge/JvbLastN.kt)
and [dynamic](https://github.com/jitsi/jitsi-videobridge/blob/master/jvb/src/main/kotlin/org/jitsi/videobridge/load_management/LastNReducer.kt)
configuration of the bridge. This is implemented by setting the `maxHeight` constraint to 0.
The resulting constraints are the "effective" constraints used by the rest of the algorithm. Once calculated, they are
announced via an event, so that the sender-side constraints can be applied. Doing this step here, early in the process,
allows us to do "aggressive layer suspension" (i.e. set sender-side constraints based on LastN).
### 3. Allocation
The final phase is the actual allocation.
#### 3.1 Initialize potential layers
The first step is to initialize a list of layers to consider for each source. It starts with the list of all layers
for the source, and prunes ones which should not be considered:
A) The ones with resolution and frame rate higher than the constraints
B) The ones which are inactive (the sending endpoint is currently not transmitting them)
C) Layers with high resolution but insufficient frame rate, that is at least the [preferred resolution](https://github.com/jitsi/jitsi-videobridge/blob/master/jvb/src/main/resources/reference.conf#L41),
and frame rate less than the [preferred frame rate](https://github.com/jitsi/jitsi-videobridge/blob/master/jvb/src/main/resources/reference.conf#L42).
For example, with the defaults of preferred resolution 360p and preferred frame rate 30 fps, the following layers will
not be considered: 360p/7.5fps, 360p/15fps, 720p/7.5fps, 720p/15fps.
#### 3.2 Allocation loop
It starts with no layers selected for any source, and remaining bandwidth equal to the total available bandwidth.
Until there is remaining bandwidth, it loops over the sources in the order obtained in [phase 1](#1.-Prioritize),
and tries to `improve()` the layer of each.
The normal `improve()` step selects the next higher layer if there is sufficient bandwidth. For on-stage sources
the `improve()` step works eagerly up to the "preferred" resolution.
The preferred resolution [can be configured](https://github.com/jitsi/jitsi-videobridge/blob/master/jvb/src/main/resources/reference.conf#L40).
# Signaling
This section describes the signaling between the client and the bridge that affects bandwidth allocation.
## Message format
The receiver's video constraint message is used to signal the preference of the client in regard to which media streams
it wants to receive. Usually only a portion of all available videos is displayed on the client. Each video is identified
by a source name and each endpoint can send multiple videos. The default format used in jitsi-meet follows a pattern
where the first part is the endpoint ID followed by '-v' and the zero-based index of the video source (see the example
below).
```json
{
"colibriClass": "ReceiverVideoConstraints",
"lastN": 2,
"selectedSources": ["A-v0", "B-v0"],
"onStageSources": ["A-v1","C-v0", "D-v0"],
"defaultConstraints": { "maxHeight": 180 },
"constraints": {
"A-v1": { "maxHeight": 720 },
"B-v0": { "maxHeight": 360 }
}
}
```
All fields are optional. The ones which are included will be updated, and the ones which are not included are not
changed.
The `defaultConstraints` are used for sources not explicitly included in `constraints` (including new sources).
The initial values are `lastN: -1` (unlimited), `defaultConstraints: {maxHeight: 180}`
([configurable](https://github.com/jitsi/jitsi-videobridge/blob/master/jvb/src/main/resources/reference.conf#L38)),
and the rest empty.
### Examples
#### Stage view (1)
Stage view with source `A-v0` in high definition and all other sources in 180p:
```json
{
"colibriClass": "ReceiverVideoConstraints",
"onStageSources": ["A-v0"],
"defaultConstraints": { "maxHeight": 180 },
"constraints": {
"A-v0": { "maxHeight": 720 }
}
}
```
#### Stage view (2)
Stage view with source `A-v0` in high definition, `B-v0`, `C-v0`, `D-v0` in 180p and all others disabled:
```json
{
"colibriClass": "ReceiverVideoConstraints",
"onStageSources": ["A-v0"],
"defaultConstraints": { "maxHeight": 0 },
"constraints": {
"A-v0": { "maxHeight": 720 },
"B-v0": { "maxHeight": 180 },
"C-v0": { "maxHeight": 180 },
"D-v0": { "maxHeight": 180 }
}
}
```
#### Stage view (3)
Stage view with source `A-v0` in high definition, `B-v0`, `C-v0`, `D-v0` disabled and all others in 180p:
```json
{
"colibriClass": "ReceiverVideoConstraints",
"onStageSources": ["A-v0"],
"defaultConstraints": { "maxHeight": 180 },
"constraints": {
"A-v0": { "maxHeight": 720 },
"B-v0": { "maxHeight": 0 },
"C-v0": { "maxHeight": 0 },
"D-v0": { "maxHeight": 0 }
}
}
```
#### Stage view (4)
Stage view with source `A-v0` in high definition and all other sources in 180p, with "D-v0" prioritized higher than
the dominant speaker's video source:
```json
{
"colibriClass": "ReceiverVideoConstraints",
"onStageEndpoints": ["A-v0"],
"selectedEndpoints": ["D-v0"],
"defaultConstraints": { "maxHeight": 180 },
"constraints": {
"A-v0": { "maxHeight": 720 }
}
}
```
#### Tile view (1)
Tile view with all sources in 180p/15fps:
```json
{
"colibriClass": "ReceiverVideoConstraints",
"defaultConstraints": { "maxHeight": 180, "maxFrameRate": 15 }
}
```
#### Tile view (2)
Tile view with all sources in 360p:
```json
{
"colibriClass": "ReceiverVideoConstraints",
"defaultConstraints": { "maxHeight": 360 }
}
```
#### Tile view (3)
Tile view with 180p, sources `A-v0` and `B-v0` prioritized, and sources `C-v0` and `D-v0` disabled:
```json
{
"colibriClass": "ReceiverVideoConstraints",
"selectedSources": ["A-v0", "B-v0"],
"defaultConstraints": { "maxHeight": 180 },
"constraints": {
"C-v0": { "maxHeight": 0 },
"D-v0": { "maxHeight": 0 }
}
}
```
#### Tile view (4)
Tile view with all sources disabled except `A-v0`, `B-v0`, `C-v0`:
```json
{
"colibriClass": "ReceiverVideoConstraints",
"defaultConstraints": { "maxHeight": 0 },
"constraints": {
"A-v0": { "maxHeight": 180 },
"B-v0": { "maxHeight": 180 },
"C-v0": { "maxHeight": 180 }
}
}
```
#### Multi-stage view (1)
With two on-stage sources, and up-to 4 other sources at 180p:
```json
{
"colibriClass": "ReceiverVideoConstraints",
"onStageSources": ["A-v0", "B-v0"],
"lastN": 6,
"defaultConstraints": { "maxHeight": 180 },
"constraints": {
"A-v0": { "maxHeight": 720 },
"B-v0": { "maxHeight": 720 }
}
}
```
## Old message format
The old format works with endpoint IDs rather than source names. It also uses `selectedEndpoints` and `onStageEndpoints`
instead of `selectedSources` and `onStageSources`. All use cases described in the examples above are valid here as
well, but the assumption is that every endpoint has only one video source.
```json
{
"colibriClass": "ReceiverVideoConstraints",
"lastN": 2,
"selectedEndpoints": ["A", "B"],
"onStageEndpoints": ["C", "D"],
"defaultConstraints": { "maxHeight": 180 },
"constraints": {
"A": { "maxHeight": 720 },
"B": { "maxHeight": 360 }
}
}
```
The support for source names vs legacy endpoint ID based format is determined on the ColibriV2 level. The endpoint
create request must indicate "source-names" capability on the endpoint that will be using the new format. This is done
automatically by Jicofo when multi stream support is enabled in jitsi-meet, but if you're using JVB without Jicofo then
remember to do that yourself.
```xml
<iq xmlns='jabber:client' to='jvbbrewery@internal-muc.meet.jitsi/jvb1' id='VFLJ9-10' type='get'>
<conference-modify xmlns='jitsi:colibri2' meeting-id='62a5bc4c-c79c-4eab-a071-6740eb549296'>
<endpoint xmlns='jitsi:colibri2' id='fefbee3e' create='true' >
<capability name='source-names'/> <!--- SOURCE NAME CAPABILITY -->
<media type='audio'>
...
</media>
<media type='video'>
...
</media>
<transport ice-controlling='true'/>
</endpoint>
</conference-modify>
</iq>
```

18
doc/constraints.md Normal file
View File

@@ -0,0 +1,18 @@
## Video Constraints
###### Sender video constraints
The bridge sends the following message to a sender to notify it that resolutions
higher than the specified need not be transmitted for a specific video source:
```
{
"colibriClass": "SenderSourceConstraints",
"sourceName": "endpoint1-v0",
"maxHeight": 180
}
```
Note that in order to handle portrait-mode video correctly,
despite the name `maxHeight`, this parameter is actually used to specify the smaller
of the dimensions of the video (which for portrait mode will be its width).

36
doc/datachannel-communication.md Normal file
View File

@@ -0,0 +1,36 @@
# Messages by COLIBRI class:
## EndpointMessage:
This type of messages could be used by the clients for sending JSON objects to the other clients in the conference. A client can send a message to specific endpoint or a broadcast message.
* The messages processed by Jitsi Videobridge has the following format:
```
{
colibriClass: "EndpointMessage",
to: `<ENDPOINT-ID>`,
msgPayload: `<PAYLOAD>`
}
```
`<PAYLOAD>` is valid JSON string.
`<ENDPOINT-ID>` is endpoint id of existing participant or `""` for broadcast message.
* The messages sent by Jitsi Videobridge has the following format:
```
{
colibriClass: "EndpointMessage",
to: `<ENDPOINT-ID>`,
from: `<ENDPOINT-ID>`,
msgPayload: <PAYLOAD>
}
```
The only modification made by Jitsi Videobridge before the message is forwarded is adding `from` property to it. The value of this property is the endpoint id of the sender of the message.

201
doc/debugging.md Normal file
View File

@@ -0,0 +1,201 @@
# Introduction
This document describes some HTTP endpoints that can be useful when debugging jitsi-videobridge. With a few exceptions,
these are NOT intended to be used against production systems. Most endpoints return valid JSON which can be fed to e.g. `jq`:
```
curl http://localhost:8080/debug | jq .
```
# Basic queries
### Health
Returns `200 OK` if the instance is currently healthy, and `4XX/5XX` otherwise.
```
GET /about/health
```
### Version
Returns the currently running version.
```
GET /about/version
```
### Get Dominant speaker
Returns a dominant speaker identification
```
GET /colibri/v2/conferences/{meetingId}/dominant-speaker-identification
```
### Overall statistics
Returns overall bridge statistics. These include number of current and total conferences and endpoints, number of packets/bytes received/sent, current packet rate and bit rate. They are generally suitable for storing as timeseries in order to monitor a service.
```
GET /colibri/stats
```
# Detailed debug state
The following can be used to get very detailed state of running conferences and endpoints.
### List conferences
This includes a list of all conferences and their endpoints, but not the full state.
```
GET /debug
```
### List all conferences with full state
This includes the full state of all conferences and their endpoints. The output can be substantial (~25KB per endpoint).
```
GET /debug?full=true
```
## List a specific conference with full state
This includes the full state of a specific conference and its endpoints. The output can be substantial (~25KB per endpoint).
The full output can be suspended (leaving just the list of endpoints) by adding `?full=false`.
```
GET /debug/CONFERENCE_ID
```
## Get full state for an endpoint
This includes the full state of a specific endpoint in a specific conference.
```
GET /debug/CONFERENCE_ID/ENDPOINT_ID
```
# Global debug options and statistic features
These are features that affect the entire bridge (as opposed to a single endpoint or conference).
### Payload verification
This mode enables verification of the RTP payload between each step of the packet pipeline.
This is very CPU intensive and disabled by default (it is only useful for testing new code).
Enable:
```
POST /debug/stats/jvb/payload-verification/true
```
Disable:
```
POST /debug/stats/jvb/payload-verification/false
```
Query:
```
GET /debug/stats/jvb/payload-verification
```
### Pipeline statistics
This keeps track of processing delay, number of discarded packets,
number of packets by various types, etc. from each Node in the packet
processing pipeline. The data is organized by Node type. This has low overhead
and is enabled by default. Note that the data does not update in real-time, but
only when endpoints expire.
Enable:
```
POST /debug/stats/jvb/node-stats/true
```
Disable:
```
POST /debug/stats/jvb/node-stats/false
```
Query:
```
GET /debug/stats/jvb/node-stats
```
### Memory pool statistics
These include current size, number of requests, allocation rates, etc. for the application memory pools. This is disabled by default.
Enable:
```
POST /debug/stats/jvb/pool-stats/true
```
Disable:
```
POST /debug/stats/jvb/pool-stats/false
```
Query:
```
GET /debug/stats/jvb/pool-stats
```
### Packet queue statistics
This keeps track of the number of dropped packets and exception caught in the
various packet queues. It is enabled by default.
Enable:
```
POST /debug/stats/jvb/queue-stats/true
```
Disable:
```
POST /debug/stats/jvb/queue-stats/false
```
Query:
```
GET /debug/stats/jvb/queue-stats
```
### Packet transit time statistics
This keeps track of the overall transit time for RTP/RTCP packets (average,
max, and a distribution), as well as the jitter introduced in processing. It
is always enabled.
Query:
```
GET /debug/stats/jvb/transit-stats
```
### Task pool stats
This keeps track of statistics for the various task pools (CPU, IO, scheduled).
It is always enabled.
Query:
```
GET /debug/stats/jvb/task-pool-stats
```
### XMPP Delay stats
This keeps track of the response time for requests received over XMPP.
Query:
```
GET /debug/stats/jvb/xmpp-delay-stats
```
### Node tracing
This adds an entry to the stack trace from each Node in the packet processing pipeline.
Enable:
```
POST /debug/stats/jvb/node-tracing/true
```
Disable:
```
POST /debug/stats/jvb/node-tracing/false
```
# Endpoint-specific debug options and features
These are features that can be enabled for a specific endpoint in a specific conference.
### Saving RTP/RTCP in PCAP
This enables saving the RTP and RTCP traffic sent/received from a specific endpoint to a PCAP file in `/tmp/`.
The use of this feature is disabled by default, and the bridge needs to be explicitly configured to
allow it by setting `jmt.debug.pcap.enabled=true` in `/etc/jitsi/videobridge/jvb.conf`.
Enable:
```
POST /debug/features/endpoint/CONFERENCE_ID/ENDPOINT_ID/pcap-dump/true
```
Disable:
```
POST /debug/features/endpoint/CONFERENCE_ID/ENDPOINT_ID/pcap-dump/false
```

19
doc/health-checks.md Normal file
View File

@@ -0,0 +1,19 @@
The REST API allows querying Videobridge whether it deems itself in a healthy state (i.e. the application is operational and the functionality it provides should perform as expected) at the time of the query or not. Videobridge performs periodic internal tests, and returns the latest result in response to requests to the `/about/health` endpoint.
<table>
<tr>
<th>HTTP Method</th>
<th>Resource</th>
<th>Response</th>
</tr>
<tr>
<td><code>GET</code></td>
<td><code>/about/health</code></td>
<td>
<ul>
<li><code>200 OK</code> The service is healthy. The response is with <code>Content-Type: application/json</code> but no JSON value is provided i.e. <code>Content-Length: 0</code>.</li>
<li><code>500 Internal Server Error</code> or any other <code>5xx</code> status code if Videobridge determined that it is in an unhealthy state.</li>
</ul>
</td>
</tr>
</table>

66
doc/muc.md Normal file
View File

@@ -0,0 +1,66 @@
# Intro
When the XMPP API is enabled jitsi-videobridge uses an XMPP client connection and advertises presence in a Multi-User
Chat (MUC) room.
With this mode a jitsi-videobridge instance can connect to a set of XMPP servers, and new servers
can be added [at runtime](https://github.com/jitsi/jitsi-videobridge/blob/master/doc/rest-muc-client.md).
# XMPP server configuration
The XMPP server can be provisioned with a single user account to be shared between all jitsi-videobridge instances that
connect to it. For prosody this can be done using:
```
prosodyctl register jvb $DOMAIN $PASSWORD
```
We recommend using a separate XMPP domain, not accessible by anonymous users.
See the [jitsi-meet-prosody postinst
script](https://github.com/jitsi/jitsi-meet/blob/master/debian/jitsi-meet-prosody.postinst)
for a full example. It is also possible to use a completely separate XMPP server accessible only to jitsi-videobridge
and jicofo (not publicly accessible at all).
# Jicofo configuration
To make use of the MUC mode in the bridge, jicofo needs to be configured to
join the same MUC. To do this set `jicofo.bridge.brewery-jid` to the MUC address in `jicofo.conf`.
# Videobridge configuration
To enable the MUC mode add the following to the `videobridge` section of the config file:
```
stats {
# Enable broadcasting stats/presence in a MUC
enabled = true
transports = [
{ type = "muc" }
]
}
apis {
xmpp-client {
configs {
# Connect to the first XMPP server
xmpp-server-1 {
hostname="example.net"
domain = "auth.example.net"
username = "jvb"
password = "$PASSWORD"
muc_jids = "JvbBrewery@internal.auth.example.net"
# The muc_nickname must be unique across all jitsi-videobridge instances
muc_nickname = "unique-instance-id"
# disable_certificate_verification = true
}
# Connect to a second XMPP server
xmpp-server-2 {
hostname="another.example.net"
domain = "auth.example.net"
username = "jvb"
password = "$PASSWORD"
muc_jids = "JvbBrewery@internal.auth.example.net"
# The muc_nickname must be unique across all jitsi-videobridge instances
muc_nickname = "unique-instance-id2"
# disable_certificate_verification = true
}
}
}
}
```

41
doc/network-configuration.md Normal file
View File

@@ -0,0 +1,41 @@
# Network Configuration
## IPv6 Configuration
JVB supports IPv6 in dual-stack (IPv4 + IPv6) environments.
To enable it, add the following to your `jvb.conf`:
```hocon
ice4j {
harvest {
use-ipv6 = true
use-link-local-addresses = false
mapping {
stun {
addresses = [ "your-stun-server:3478" ]
}
static-mappings = [
{
local-address = "YOUR_PRIVATE_IPV4"
public-address = "YOUR_PUBLIC_IPV4"
}
]
}
}
}
```
### Notes
- Jitsi Web and the Videobridge can have different IPv6 addresses
- When IPv6 is enabled, a UDP6 listener will open on port 10000
### Verifying ICE Candidates
To confirm correct candidates are advertised, enable verbose logging in `/etc/jitsi/videobridge/logging.properties`:
```
org.jitsi.videobridge.xmpp.XmppConnection.level=ALL
```
Then grep the logs for `SENT` to inspect the ICE candidates being sent.
### Related Issues
- [#1722](https://github.com/jitsi/jitsi-videobridge/issues/1722)

86
doc/relay.md Normal file
View File

@@ -0,0 +1,86 @@
# Setting up relays (cascaded bridges)
### Relays
Relays (aka secure octo) use ICE and DTLS/SRTP between each pair of bridges, so a secure
network is not required. It uses and requires either SCTP or colibri websockets for the
bridge-bridge connections.
## Jitsi Videobridge configuration
### Relay configuration
Relays can be configured with the following properties in `/etc/jitsi/videobridge/jvb.conf` (also see
[reference.conf](https://github.com/jitsi/jitsi-videobridge/blob/master/jvb/src/main/resources/reference.conf#L132)).
```
videobridge {
relay {
enabled=true
region="region1"
relay-id="unique-id-of-the-jitsi-videobridge-instance"
}
}
```
Legacy configuration is detected and automatically adapted, so no configuration changes are necessary when upgrading.
Relays require colibri websockets for the bridge-to-bridge connections, which can be enabled as described in
[this document](https://github.com/jitsi/jitsi-videobridge/blob/master/doc/web-sockets.md). Note that the `colibri-relay-ws`
endpoints also need to be proxied.
## Jicofo configuration
Octo needs to be enabled, and a suitable bridge selection strategy needs to be configured in `/etc/jitsi/jicofo/jicofo.conf`:
```
jicofo {
bridge {
selection-strategy = RegionBasedBridgeSelectionStrategy
}
octo {
enabled = true
}
}
```
The `RegionBasedBridgeSelectionStrategy` matches the region of the clients to
the region of the Jitsi Videobridge instances. That is, it always attempts to select a bridge
in the region of the client.
The `SplitBridgeSelectionStrategy` can be used for testing. It tries to select a new bridge
for each client, regardless of the regions. This is useful while testing, because you can
verify that Octo works before setting up the region configuration for the clients.
*Important note*: Jicofo will not mix bridges with different versions in the same conference. Even with
`SplitBridgeSelectionStrategy` if the bridges have different versions only bridges with the same version will be used.
To prevent this behavior (for testing purposes only) set `jicofo.ocfo.allow-mixed-versions=true` in `jicofo.conf`.
## Debugging
Jicofo's debug interface can be used for troubleshooting.
To list the bridges available to jicofo (with their versions, regions, etc):
```commandline
curl "http://localhost:8888/debug" | jq .bridge_selector
```
To list currently running conferences:
```commandline
curl "http://localhost:8888/debug" | jq .focus_manager
```
To list the full state of a conference (get actual conference ID with the command above, escaping the '@' sign):
```commandline
curl "http://localhost:8888/debug/conference/test\@conference.example.com" | jq .
```
Look under `colibri_session_manager` for the different bridge sessions.
## Configuring client regions
Clients learn about the region that they are in via the value of
`config.deploymentInfo.userRegion`. Thus, in order for relays to actually work
for geolocation, `config.js` must be served with the correct values. The best
way to implement this is outside the scope of this document, and it depends on
the environment in which Jitsi Meet is installed.
One easy solution can be to set the variable based on the domain name, and then
access the conference through the desired domain name, e.g. use
`us-east.jitsi.example.com`, `us-west.jitsi.example.com`,
`eu.jitsi.example.com` and have them serve custom versions of `config.js`.

315
doc/rest-colibri2.md Normal file
View File

@@ -0,0 +1,315 @@
# Introduction
This document describes the Colibri2 REST APIs
See [this document](rest.md) for how to configure the HTTP(S) interfaces of jitsi-videobridge.
We envision using the HTTP/JSON version of the colibri2 protocol you can see some examples [here](https://github.com/jitsi/jitsi-xmpp-extensions/blob/master/src/test/kotlin/org/jitsi/xmpp/extensions/colibri2/json/Colibri2JSONSerializerTest.kt).
# Implementation
<table>
<tr>
<th>HTTP Method</th>
<th>Resource</th>
<th>Response</th>
</tr>
<tr>
<td>GET</td>
<td>/colibri/v2/conferences/{meetingId}/dominant-speaker-identification</td>
<td>
200 OK with JSON objects which represents <code>endpointsBySpeechActivity</code>,<code>dominantEndpoint</code>, <code>endpointsInLastNOrder</code>,<code> dominantSpeakerIdentification</code> <br />
For example:
<pre>
{
"endpointsBySpeechActivity": [
"79f0273d"
],
"dominantSpeakerIdentification": null,
"endpointsInLastNOrder": [
"79f0273d"
],
"dominantEndpoint": "null"
}</pre>
</td>
</tr>
<tr>
<td>POST</td>
<td>/colibri/v2/conferences/</td>
<td>
200 OK with a JSON object which represents the created conference if the request was with <code>Content-Type: application/json</code> and was a JSON object which
consist field like <code> meeting-id</code>, <code>endpoints</code>, media types, media payloads<br />
See the test example
<a href="https://github.com/jitsi/jitsi-xmpp-extensions/blob/master/src/test/kotlin/org/jitsi/xmpp/extensions/colibri2/json/Colibri2JSONSerializerTest.kt">
Colibri2JSONSerializerTest
</a> <br />
For example, a request could look like:
<pre>
{
"meeting-id":"beccf2ed-5441-4bfe-96d6-f0f3a6796378",
"name":"hardencounterssinkright@muc.meet.jitsi",
"create":true,
"endpoints":[
{
"create":true,
"id":"79f0273d",
"stats-id":"Garett-w1o",
"muc-role":"moderator",
"medias":[
{
"type":"audio",
"payload-types":[
{
"name": "red", "id": "112", "channels": "2", "clockrate": "48000",
"parameters": { "null": "111/111" }
},
{
"name": "opus", "id": "111", "channels": "2", "clockrate": "48000",
"parameters": {"useinbandfec": "1", "minptime": "10" },
"rtcp-fbs": [{"type": "transport-cc"}]
}
],
"rtp-hdrexts":[
{ "uri":"urn:ietf:params:rtp-hdrext:ssrc-audio-level", "id":1 },
{ "uri":"http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01", "id":5 }
],
"extmap-allow-mixed":true
},
{
"type": "video",
"payload-types":[
{
"name": "VP8", "id": "100", "clockrate": "90000",
"parameters": {"x-google-start-bitrate": "800"},
"rtcp-fbs":[
{ "type": "ccm", "subtype": "fir" },
{ "type": "nack" },
{ "type": "nack", "subtype": "pli" },
{ "type": "transport-cc" }
]
},
{
"name": "VP9", "id": "101", "clockrate": "90000",
"parameters": {"x-google-start-bitrate": "800"},
"rtcp-fbs":[
{ "type": "ccm", "subtype": "fir" },
{ "type": "nack" },
{ "type": "nack", "subtype": "pli" },
{ "type": "transport-cc" }
]
}
],
"rtp-hdrexts":[
{ "uri":"http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time", "id":3 },
{ "uri":"http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01", "id":5 }
],
"extmap-allow-mixed":true
}
],
"transport": { "ice-controlling": true },
"capabilities": [ "source-names" ]
}
],
"connects": [
{ "url": "wss://example.com/audio", "protocol": "mediajson", "type": "transcriber", "audio": true },
{ "url": "wss://example.com/video", "protocol": "mediajson", "type": "recorder", "video": true }
]
}</pre>
The respective response could look like:
<pre>
{
"endpoints": [
{
"id": "79f0273d",
"transport": {
"transport": {
"candidates": [
{
"generation": "0",
"component": "1",
"protocol": "udp",
"port": "10000",
"ip": "172.18.0.4",
"foundation": "1",
"id": "5bbb5d5e101f870f0ffffffffdf286239",
"priority": "2130706431",
"type": "host",
"network": "0"
},
{
"generation": "0",
"rel-port": "10000",
"component": "1",
"protocol": "udp",
"port": "10000",
"ip": "192.168.1.1",
"foundation": "2",
"id": "56f8f898101f870f0fffffffff3be6336",
"rel-addr": "172.18.0.4",
"priority": "1694498815",
"type": "srflx",
"network": "0"
},
{
"generation": "0",
"rel-port": "10000",
"component": "1",
"protocol": "udp",
"port": "10000",
"ip": "152.58.154.133",
"foundation": "2",
"id": "c6ac5101f870f0ffffffffcb50fcba",
"rel-addr": "172.18.0.4",
"priority": "1694498815",
"type": "srflx",
"network": "0"
}
],
"xmlns": "urn:xmpp:jingle:transports:ice-udp:1",
"ufrag": "26ant1ioqu7qnr",
"rtcp-mux": true,
"pwd": "9q8uj7tg6j4rc4i6ulc1pmsq9",
"web-sockets": [
"wss://localhost:8443/colibri-ws/172.18.0.4/7e441eda3701676e/79f0273d?pwd=9q8uj7tg6j4rc4i6ulc1pmsq9"
],
"fingerprints": [
{
"fingerprint": "F1:05:72:F8:F3:F4:B4:65:5B:9A:FC:81:24:11:23:78:D5:42:54:B1:F2:66:2F:82:CC:2B:01:1C:DB:98:BE:C0",
"setup": "actpass",
"hash": "sha-256"
}
]
}
}
}
],
"sources": [
{
"sources": [
{
"ssrc": 3191927963,
"name": "jvb-a0"
}
],
"id": "jvb-a0",
"type": "audio"
},
{
"sources": [
{
"ssrc": 1863439907,
"name": "jvb-v0"
}
],
"id": "jvb-v0",
"type": "video"
}
]
}</pre>
</td>
</tr>
<tr>
<td>PATCH</td>
<td>/colibri/v2/conferences/{mettingId}</td>
<td>
Expire conference <br />
200 OK which represents success. <br />
For example:
<pre>
The Request body to expire conference:
{
"expire":true
}
</pre>
</td>
</tr>
<tr>
<td>PATCH</td>
<td>/colibri/v2/conferences/{mettingId}</td>
<td>
200 OK which represents success. <br />
Here, I am doing it for
<a href="https://github.com/jitsi/jitsi-xmpp-extensions/blob/master/src/test/kotlin/org/jitsi/xmpp/extensions/colibri2/json/Colibri2JSONSerializerTest.kt#L326">
Conference-modified with transport for an endpoint and feedback sources
</a> <br />
In the same way you can do it for
<a href="https://github.com/jitsi/jitsi-xmpp-extensions/blob/master/src/test/kotlin/org/jitsi/xmpp/extensions/colibri2/json/Colibri2JSONSerializerTest.kt#L388C17-L388C59">
Update endpoint with transport and sources
</a> <br />
For example, a request could look like:
<pre>
{
"endpoints": [
{
"id":"79f0273e",
"transport": {
"transport": {
"candidates": [
{
"generation": "0",
"rel-port": "9",
"component": "1",
"protocol": "udp",
"port": "10000",
"ip": "129.80.210.199",
"foundation": "2",
"id": "653aa1ba295b62480ffffffffdc52c0d9",
"rel-addr": "0.0.0.0",
"priority": "1694498815",
"type": "srflx",
"network": "0"
}
],
"xmlns": "urn:xmpp:jingle:transports:ice-udp:1",
"ufrag": "2ivqh1fvtf0l3h",
"rtcp-mux": true,
"pwd": "1a5ejbent91k6io6a3fauikg22",
"web-sockets": [
"wss://beta-us-ashburn-1-global-2808-jvb-83-102-26.jitsi.net:443/colibri-ws/default-id/3d937bbdf97a23e0/79f0273e?pwd=1a5ejbent91k6io6a3fauikg22"
],
"fingerprints": [
{
"fingerprint": "2E:CC:85:71:32:5B:B5:60:64:C8:F6:7B:6D:45:D4:34:2B:51:A0:06:B5:EA:2F:84:BC:7B:64:1F:A3:0A:69:23",
"setup": "actpass",
"hash": "sha-256",
"cryptex": true
}
]
}
}
}
],
"sources": [
{
"type": "audio",
"id": "jvb-a0",
"sources": [
{ "ssrc":411312308, "name": "jvb-a0", "parameters": { "msid": "mixedmslabel mixedlabelaudio0" } }
]
},
{
"type": "video",
"id": "jvb-v0",
"sources": [
{ "ssrc":3929652146, "name": "jvb-v0", "parameters": { "msid": "mixedmslabel mixedlabelvideo0" } }
]
}
]
}</pre>
The respective response could look like:
<pre>
{
"endpoints": [
{
"id": "79f0273e"
}
]
}</pre>
</td>
</tr>
</table>

34
doc/rest-muc-client.md Normal file
View File

@@ -0,0 +1,34 @@
The configuration for the XMPP client connections that jitsi-videobridge uses can be modified at run time using REST calls to `/colibri/muc-client/`.
# Adding an XMPP client connection
A new XMPP client connection (i.e. a MucClient) can be added by posting a JSON which contains its configuration to `/colibri/muc-client/add`:
```
{
"id": "new-client-connection",
"domain":"xmpp.example.com",
"hostname":"10.0.0.1",
"username":"xmpp-username",
"password":"xmpp-password",
"muc_jids":"JvbBrewery@conference.xmpp.example.com",
"muc_nickname":"unique-resource",
"disable_certificate_verification":"false"
}
```
If a configuration with the specified ID already exists the behavior depends on whether the `hostname`, `port`,
`domain` and `username` fields match the existing client. If they match no changes are made, otherwise the existing
`MucClient` is removed and a new one is added.
# Removing an XMPP client connection
An XMPP client connection (i.e. a MucClient) can be removed by posting a JSON which contains its ID to `/colibri/muc-client/remove`:
```
{
"id": "new-client-connection"
}
```
The request will be successful (return 200) if an XMPP client connection was removed.
# Listing XMPP client connections
IDs of previously added XMPP client connections can be listed using a GET call to `/colibri/muc-client/list`.

61
doc/rest.md Normal file
View File

@@ -0,0 +1,61 @@
Introduction
==============
Jitsi-videobridge supports two HTTP(S) interfaces, a _public_ and a _private_ one.
The two interfaces use different ports. The _private_ interface exposes
HTTP endpoints which are not meant to be publicly accessible (but might be used
by other components of the infrastructure, e.g. a signaling server), such as:
* The [COLIBRI control interface](rest-colibri2.md) (```/colibri/```)
* The health-check interface (```/about/health```)
* The version interface (```/about/version```)
The _public_ interface is used for Colibri Web Sockets that clients connect to.
Configuration
==============
The following configuration properties can be added to the jitsi-videobridge configuration file (whichever file is
passed to `-Dconfig.file` when running the bridge, usually `/etc/jitsi/videobridge/jvb.conf`) to control the behavior
of the HTTP interfaces.
For the _private_ interface, use the following block and values:
```hocon
videobridge {
apis {
rest {
enabled = true
}
}
http-servers {
private {
// The port for the private HTTP interface (or -1 to disable it). Defaults to 8080.
port = <Number>
// The port for the private HTTP interface if TLS is to be used (or -1 to disable). Defaults to 8443.
tls-port = <Number>
// The address on which the server will listen
host = <String>
// The file path to the keystore to be used with HTTPS for the private interface. If this is not specified,
// HTTPS is disabled for the private interface
key-store-path = <String (path to key store)>
// The password to be used by the SslContextFactory when HTTPS is used
key-store-password = <String>
// Whether or not client certificate authentication is to be required when
// HTTPS is used
need-client-auth: <Boolean>
}
}
}
```
The _public_ interface uses the same values, but under a different scope:
```hocon
videobridge {
http-servers {
public {
// Same values as above
}
}
}
```

129
doc/statistics.md Normal file
View File

@@ -0,0 +1,129 @@
Introduction
============
Jitsi Videobridge exports statistics/metrics as key-value pairs in three ways: in a custom JSON format, as Prometheus
metrics and as periodic reports published in an XMPP MUC.
# REST
The statistics are available through the `/colibri/stats` endpoint on the [private REST interface](rest.md)
(if it has been enabled) in JSON format:
```json
{
"key": "value"
}
```
Note that the report itself is generated periodically and a cached version is returned. The period defaults to 5 seconds
and can be configured with the `videobridge.stats.interval` property in `jvb.conf`.
# Prometheus
Metrics are exposed in a Prometheus/openmetrics format at `/metrics` on the private HTTP interface. Not all metrics are
currently available in this format, and improving this is a work in progress.
```commandline
# Query prometheus metrics in openmetrics format
curl 0:8080/metrics -H 'Accept: application/openmetrics-text'
# Query prometheus metrics in 004 format
curl 0:8080/metrics -H 'Accept: text/plain; version=0.0.4;'
# Query prometheus metrics in a JSON key-value format
curl 0:8080/metrics -H 'Accept: application/json'
```
Note: Conference size distribution metrics (by total endpoints, by audio senders, and by video senders) are available as
Prometheus histograms (`conferences_by_size`, `conferences_by_audio_sender`, `conferences_by_video_sender`) but are not
exposed via the JSON `/colibri/stats` endpoint.
# XMPP MUC
The statistics can also be published periodically via XMPP (which allows jicofo to monitor a set of bridges and perform
load balancing, or allows an application to monitor the MUC and collect metrics from multiple bridges). In this case the
key-value pairs are represented in XML format with a `stats` element like this:
```xml
<stats xmlns=' http://jitsi.org/protocol/colibri'>
<stat value='value' name='key'/>
</stats>
```
By default, statistics are pushed every 5 seconds and this can be configured in `jvb.conf` with the
[`videobridge.apis.xmpp-client.presence-interval`](https://github.com/jitsi/jitsi-videobridge/blob/master/jvb/src/main/resources/reference.conf#L65) property.
# Supported metrics
Below is a non-exhaustive list of currently supported metrics. In the descriptions "current" means an instantaneous
value (when the report was generated), and "total" means a cumulative value since the application was started.
* `bit_rate_download` - the current incoming bitrate (RTP) in kilobits per second.
* `bit_rate_upload` - the current outgoing bitrate (RTP) in kilobits per second.
* `conferences` - The current number of conferences.
* `current_timestamp` - the UTC time at which the report was generated.
* `dtls_failed_endpoints` - the total number of endpoints which failed to establish a DTLS connection.
* `endpoints` - the current number of endpoints, including `octo` endpoints.
* `endpoints_sending_audio` - current number of endpoints sending (non-silence) audio.
* `endpoints_sending_video` - current number of endpoints sending video.
* `endpoints_with_spurious_remb` - total number of endpoints which have sent an RTCP REMB packet when REMB was not
signaled.
* `graceful_shutdown` - whether jitsi-videobridge is currently in graceful shutdown mode (hosting existing conferences,
but not accepting new ones).
* `inactive_conferences` - current number of conferences in which no endpoints are sending audio nor video. Note that
this includes conferences which are currently using a peer-to-peer transport.
* `inactive_endpoints` - current number of endpoints in inactive conferences (see `inactive_conferences`).
* `largest_conference` - the size of the current largest conference (counting all endpoints, including `octo`
endpoints which are connected to a different jitsi-videobridge instance)
* `local_active_endpoints` - the current number of local endpoints (not `octo`) which are in an active conference. This
includes endpoints which are not sending audio or video, but are in an active conference (i.e. they are receive-only).
* `local_endpoints` - the current number of local (non-`octo`) endpoints.
* `num_eps_oversending` - current number of endpoints to which we are oversending.
* `octo_conferences` - current number of conferences in which `octo` is enabled.
* `octo_endpoints` - current number of `octo` endpoints (connected to remove jitsi-videobridge instances).
* `octo_receive_bitrate` - current incoming bitrate on the `octo` channel (combined for all conferences) in bits per
second.
* `octo_receive_packet_rate` - current incoming packet rate on the `octo` channel (combined for all conferences) in
packets per second.
* `octo_send_bitrate` - current outgoing bitrate on the `octo` channel (combined for all conferences) in bits per
second.
* `octo_send_packet_rate` - current outgoing packet rate on the `octo` channel (combined for all conferences) in
packets per second.
* `p2p_conferences` - current number of peer-to-peer conferences. These are conferences of size 2 in which no endpoint
is sending audio not video. Presumably the endpoints are using a peer-to-peer transport at this time.
* `packet_rate_download` - current RTP incoming packet rate in packets per second.
* `packet_rate_upload` - current RTP outgoing packet rate in packets per second.
* `participants` - current number of endpoints, including `octo` endpoints. Deprecated.
* `preemptive_kfr_sent` - total number of preemptive keyframe requests sent.
* `receive_only_endpoints` - current number of endpoints which are not sending audio nor video.
* `region` - preconfigured region used for bridge selection in jicofo.
* `relay_id` - encodes the `octo` address of this bridge.
* `rtt_aggregate` - round-trip-time measured via RTCP averaged over all local endpoints with a valid RTT measurement in
milliseconds.
* `stress_level` - current stress level on the bridge, with 0 indicating no load and 1 indicating the load is at full
capacity (though values >1 are permitted).
* `threads` - current number of JVM threads.
* `total_bytes_received` - total number of bytes received in RTP.
* `total_bytes_received_octo` - total number of bytes received on the `octo` channel.
* `total_bytes_sent` - total number of bytes sent in RTP.
* `total_bytes_sent_octo` - total number of bytes sent on the `octo` channel.
* `total_colibri_web_socket_messages_received` - total number of messages received on a Colibri "bridge channel"
messages received on a WebSocket.
* `total_colibri_web_socket_messages_sent` - total number of messages sent over a Colibri "bridge channel" messages
sent over a WebSocket.
* `total_conference_seconds` - total number of conference-seconds served (only updates once a conference expires).
* `total_conferences_completed` - total number of conferences completed.
* `total_conferences_created` - total number of conferences created.
* `total_data_channel_messages_received` - total number of Colibri "bridge channel" messages received on SCTP data
channels.
* `total_data_channel_messages_sent` - total number of Colibri "bridge channel" messages sent over SCTP data
channels.
* `total_dominant_speaker_changes` - total number of times the dominant speaker in a conference changed.
* `total_failed_conferences` - total number of conferences in which no endpoints succeeded to establish an ICE
connection.
* `total_ice_failed` - total number of endpoints which failed to establish an ICE connection.
* `total_ice_succeeded` - total number of endpoints which successfully established an ICE connection.
* `total_ice_succeeded_relayed` - total number of endpoints which connected through a TURN relay (currently broken).
* `total_packets_dropped_octo` - total number of packets dropped on the `octo` channel.
* `total_packets_received` - total number of RTP packets received.
* `total_packets_received_octo` - total number packets received on the `octo` channel.
* `total_packets_sent` - total number of RTP packets sent.
* `total_packets_sent_octo` - total number packets sent over the `octo` channel.
* `total_partially_failed_conferences` - total number of conferences in which at least one endpoint failed to establish
an ICE connection.
* `total_participants` - total number of endpoints created.
* `version` - the version of jitsi-videobridge.
* `videochannels` - current number of endpoints with a video channel (i.e. which support receiving video). Deprecated.

124
doc/svc.md Normal file
View File

@@ -0,0 +1,124 @@
# SSRC rewriting principles for VP8 simulcast/SVC
WebRTC has no notion of [simulcast][simulcast] when receiving a video stream,
and it is therefore critical that any switch made by the SFU must be completely
transparent to the receiver [1].
1) In the case of sending an RTP stream without using picture ids the SFU has
to rewrite the RTP sequence numbers so that the stream looks continuous, and
rewrite the RTP timestamps so that there is no significant jump compared to the
last frame. The timestamp may not jump backward.
2) If picture ids are used, besides following the steps outlined in 1), the SFU
also has to rewrite the picture ids to make them continuous. Note that it is
not possible to suddenly drop picture ids from the stream, as the receiver only
keeps state relevant to picture ids, if they are used and therefore won't fall
back to RTP sequence numbers.
3) If picture ids, tl0 picture indexes and temporal indexes are used, besides
following the steps outlined in 2), the SFU also has to rewrite the tl0 picture
indexes. Note that it is possible to drop the tl0 picture indexes and the
temporal indexes from the stream, but if they are used again the tl0 picture
index has to continue from where it left off. Also note that the state of the
tl0 picture indexes is only updated by the receiver when both the tl0 picture
index and temporal index is received.
# Analysis
With simple simulcast, the SFU simply needs to apply a fixed delta (i.e. linear
translation) to the RTP sequence numbers/timestamps [4] and VP8
picture IDs/tl0picidx [3] that it forwards. The specific deltas are determined
when the stream switching occurs. This straightforward approach doesn't work
well if the SFU wants to leverage VP8 temporal scalability. We start our
analysis by exploring the implications of the first guideline and, as we will
see, 2 and 3 follow naturally.
With temporal scalability in VP8 (and other codecs)[2], where the SFU can drop frames to achieve
lower frame rates, applying a fixed delta would leave gaps in the sequence
numbers (because of the packets that the SFU drops). So the SFU needs to keep
track of what it's sent and manage the sequence number space of the
SFU-receiver leg (i.e. generate sequence numbers for every egress packet of a
frame). This mode of operation has several implications.
First, it means that if the SFU decides to skip a frame for whatever reason,
it cannot go back and change that decision because there would be no space left
in the sequence numbers. Furthermore, if packet re-ordering or loss has occurred,
a decision must be made immediately as to how much of a gap to leave in the
sequence numbers so delayed packets have a place to be transmitted.
When a frame arrives and it is to be forwarded, the SFU needs to determine how to
assign its projected sequence numbers. Decisions on how to assign sequence numbers
are done by comparing the _new_ frame with an _existing_ (previously-projected) frame.
(In most cases this existing frame will be the previous projected frame, in sequence number
ordering to the new frame, though in some corner cases involving very old frames
a different existing frame is chosen.)
If there is a gap in receive sequence numbers between the existing and the new frame,
the SFU can determine the worst-case gap size of projected sequence numbers to
leave such that any not-yet-received packets can be forwarded correctly.
However, the SFU can't (in general) know the temporal layer of frames it has
not yet received. If it turns out that the not-yet-received frames were in fact
not ones that the SFU wanted to forward, this will result in sequence number
gaps in the projected packet sequence, which the receiver may interpret as packet
loss. (Care must be taken in bandwidth estimation algorithms not to incorrectly
reduce bandwidth based on this "false" packet loss; in particular, this means
that senders operating on receiver-side REMB-based bandwidth estimation mustn't
adjust the estimates based on received loss.)
# Examples of projection
## Scenario A: Happy path (target TL0)
Sender sends: TL0 (1, 2, 3), TL2 (4, 5), TL1 (6, 7), TL2 (8), TL0 (9, 10, 11)
Bridge receives: what the sender sends
Bridge sends: 101, 102, 103, (9 -> 104), (10 -> 105), (11 -> 106)
When packet 9 arrives, the bridge can determine that sequence numbers 4 - 8
belong to non-routed packets and can be suppressed
## Scenario B: packet loss (target TL2)
Sender sends: TL0 (1, 2, 3), TL2 (4, 5), TL1 (6, 7), TL2 (8), TL0 (9, 10, 11)
Bridge receives: 1, 2, 3, 4, _(X)_, 6, 7, 8, 9, 10, 11, 5
Bridge sends: 101, 102, 103, 104, 106, 107, 108, 109, 110, 111, 105
When packet 6 arrives, the bridge computes the gap from the previously projected
frame. Because it wants to route both the TL2 and TL1 frames, it leaves
a gap in the outgoing sequence numbers, into which packet 5 can later be slotted.
## Scenario C: packet loss 1 (target TL0)
Sender sends: TL0 (1, 2, 3), TL2 (4, 5), TL1 (6, 7), TL2 (8), TL0 (9, 10, 11)
Bridge receives: 1, 2, 3, 4, _(X)_, 6, 7, 8, 9, 10, 11, 5
Bridge sends: 101, 102, 103, (9 -> 104), (10 -> 105), (11 -> 106)
Similarly to Scenario A, when packet 9 arrives, the bridge can determine that
sequence numbers 4-8 belong to non-routed packets. The status of the
unreceived packet 5 can be determined because packet 4 will not have its
"end of frame" flag set.
## Scenario D: packet loss 2 (target TL1)
Sender sends: TL0 (1, 2, 3), TL2 (4, 5), TL1 (6, 7), TL2 (8), TL0 (9, 10, 11)
Bridge receives: 1, 2, 3, 4, 5, 6, 7, _(X)_, 9, 10, 11, 8
Bridge sends: 101, 102, 103, (6 -> 104), (7 -> 105), (9 -> 107), (10 -> 108), (11 -> 109)
In this case, when packet 9 arrives, the bridge cannot determine what type of
frame packet 8 was -- it could have been a TL1. Therefore it must leave a gap in
the outgoing sequence numbers.
[simulcast]: https://ieeexplore.ieee.org/abstract/document/7992929
[1]: https://groups.google.com/d/topic/discuss-webrtc/gik2VH4hUjk/discussion
[2]: https://webrtchacks.com/sfu-simulcast/
[3]: https://tools.ietf.org/html/rfc7741
[4]: https://tools.ietf.org/html/rfc3550

130
doc/web-sockets.md Normal file
View File

@@ -0,0 +1,130 @@
# Introduction
WebSockets can be used instead of WebRTC Data Channels for transport of Colibri
client-to-bridge messages. This needs support from the bridge as well as the
client.
When this is enabled, a bridge will advertise a Colibri WebSocket URL together with
its ICE candidates. The URL will be specific to an endpoint (in fact the ICE username
fragment is reused, encoded as a URL parameter, for authentication), and connections
to it will be routed to the Endpoint representation in the bridge.
The URL has the following format:
```
wss://example.com/colibri-ws/server-id/conf-id/endpoint-id?pwd=123
```
For secure-octo relays the URL has the following format:
```
wss://example.com/colibri-relay-ws/server-id/conf-id/relay-id?pwd=123
```
# Bridge configuration
To enable WebSockets on the bridge, both the publicly-accessible HTTP server and WebSockets
themselves must be enabled.
The following block in jvb.conf configures the publicly-accessible HTTP server:
To enable the publicly-accessible HTTP server without TLS:
```
videobridge {
http-servers {
public {
port = 9090
}
}
}
```
To enable the publicly-accessible HTTP server with TLS:
```
videobridge {
http-servers {
public {
tls-port = 443
key-store-path=/etc/jitsi/videobridge/ssl.store
key-store-password=KEY_STORE_PASSWORD
}
}
}
```
To enable WebSockets:
```
videobridge {
websockets {
enabled = true
tls = true # or false, depending on your HTTP server config
# The port here is the 'advertise port' for websockets, which means the publicly-accessible
# port clients will use. This may match the public http server port, but could also be different
# if a proxy is being used.
domain = "example.com:443"
# A server ID can optionally be provided. This is useful when a set of jitsi-videobridge instances
# are fronted by an HTTP proxy and they advertise the same domain.
server-id = jvb2
}
}
```
# Proxy configuration
If you are using an HTTP proxy, it needs to support WebSocket. The following is
an example `nginx` configuration which fronts two bridges. The two bridges use
ports `9090` and `9091` without TLS, and are configured with
`COLIBRI_WS_SERVER_ID=jvb1` and `jvb2` (see above) respectively.
```
# colibri (JVB) websockets for jvb1
location ~ ^/colibri-ws/jvb1/(.*) {
proxy_pass http://127.0.0.1:9090/colibri-ws/jvb1/$1$is_args$args;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
tcp_nodelay on;
}
location ~ ^/colibri-ws/jvb2/(.*) {
proxy_pass http://127.0.0.1:9091/colibri-ws/jvb2/$1$is_args$args;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host alpha.jitsi.net;
tcp_nodelay on;
}
```
This configuration allows two jitsi-videobridge instances to run on the same
machine, which is useful while testing Octo.
For secure-octo the `/colibri-relay-ws` endpoints also need to be proxied:
```
# colibri secure-octo relay websockets for jvb1
location ~ ^/colibri-relay-ws/jvb1/(.*) {
proxy_pass http://127.0.0.1:9090/colibri-relay-ws/jvb1/$1$is_args$args;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
tcp_nodelay on;
}
location ~ ^/colibri-relay-ws/jvb2/(.*) {
proxy_pass http://127.0.0.1:9091/colibri-relay-ws/jvb2/$1$is_args$args;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host alpha.jitsi.net;
tcp_nodelay on;
}
```
# Troubleshooting
To verify that WebSockets are configured and used, first check that the Colibri
WebSocket URL is advertised to the clients. Open a conference and look for
"session-initiate" in the javascript console logs. Extend the XML and look for
`description -> content -> transport`. You should see a `web-socket` element
like this (you can verify this on meet.jit.si):
```xml
<web-socket xmlns="http://jitsi.org/protocol/colibri" url="wss://meet-jit-si-eu-west-2b-s5-jvb-51.jitsi.net:443/colibri-ws/default-id/4f9cb343985d4779/c814b6a6?pwd=23btmrjol5i83thk1t9s78bnkk"/>
```
Make sure that the URL is correct, and that your infrastructure routes it to
the correct jitsi-videobridge instance. Finally, check the `Network` tab in the
Chrome dev console and look for requests to this URL. You should see a websocket
opened to this URL, and messages exchanged every few seconds.

21
jitsi-media-transform/README.md Normal file
View File

@@ -0,0 +1,21 @@
# Jitsi Media Transform
Jitsi Media Transform contains classes for processing and transforming RTP and RTCP packets
# Code style
We use [ktlint](https://pinterest.github.io/ktlint/) for linting and autoformatting. It runs as part of
`mvn verify` (the build fails on style violations), and the version used is pinned by `ktlint.version` in the
root `pom.xml`, so no separate installation is needed.
To run only the check, or to autoformat, run these in this module's directory:
```
mvn exec:exec@ktlint-check
mvn exec:exec@ktlint-format
```
If you also want the `ktlint` command-line tool (for editor integration or a git hook), install a version matching
`ktlint.version`, e.g. on macOS with Homebrew:
```
brew install ktlint
```
It can then be run directly with `ktlint` (or `ktlint -F` to autoformat), install a pre-commit hook with
`ktlint installGitPreCommitHook`, and configure IntelliJ IDEA with `ktlint applyToIDEAProject`.

37
jitsi-media-transform/checkstyle.xml Normal file
View File

@@ -0,0 +1,37 @@
<?xml version="1.0"?>
<!DOCTYPE module PUBLIC
"-//Checkstyle//DTD Checkstyle Configuration 1.3//EN"
"https://checkstyle.org/dtds/configuration_1_3.dtd">
<module name="Checker">
<module name="LineLength">
<property name="fileExtensions" value="java"/>
<property name="max" value="120"/>
<property name="ignorePattern" value="http://|https://"/>
</module>
<module name="TreeWalker">
<module name="LeftCurly">
<property name="option" value="nl"/>
<property name="tokens" value="LITERAL_CATCH"/>
<property name="tokens" value="LITERAL_ELSE"/>
<property name="tokens" value="LITERAL_FINALLY"/>
<property name="tokens" value="LITERAL_IF"/>
<property name="tokens" value="LITERAL_TRY"/>
</module>
<!-- Skips empty blocks {}, to not skip them: value="alone" -->
<module name="RightCurly">
<property name="option" value="alone_or_singleline"/>
</module>
<module name="WhitespaceAfter">
<!-- Drops TYPECAST from default tokens -->
<property name="tokens" value="COMMA , SEMI , LITERAL_IF , LITERAL_ELSE , LITERAL_WHILE , LITERAL_DO , LITERAL_FOR , DO_WHILE"/>
</module>
</module>
<module name="FileTabCharacter">
<property name="eachLine" value="true"/>
</module>
</module>

408
jitsi-media-transform/pom.xml Normal file
View File

@@ -0,0 +1,408 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.jitsi</groupId>
<artifactId>jvb-parent</artifactId>
<version>2.3-SNAPSHOT</version>
</parent>
<groupId>org.jitsi</groupId>
<artifactId>jitsi-media-transform</artifactId>
<dependencies>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib</artifactId>
<version>${kotlin.version}</version>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>jitsi-srtp</artifactId>
<version>1.1-23-gaf3cd06</version>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>rtp</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>jitsi-utils</artifactId>
<version>${jitsi.utils.version}</version>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>jicoco-config</artifactId>
<version>${jicoco.version}</version>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>jitsi-metaconfig</artifactId>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-reflect</artifactId>
<version>${kotlin.version}</version>
</dependency>
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bctls-jdk18on</artifactId>
<version>${bouncycastle.version}</version>
</dependency>
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcprov-jdk18on</artifactId>
<version>${bouncycastle.version}</version>
</dependency>
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcpkix-jdk18on</artifactId>
<version>${bouncycastle.version}</version>
</dependency>
<dependency>
<groupId>org.pcap4j</groupId>
<artifactId>pcap4j-core</artifactId>
<version>1.8.2</version>
</dependency>
<dependency>
<groupId>org.pcap4j</groupId>
<artifactId>pcap4j-packetfactory-static</artifactId>
<version>1.8.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.github.spotbugs/spotbugs-annotations -->
<dependency>
<groupId>com.github.spotbugs</groupId>
<artifactId>spotbugs-annotations</artifactId>
<version>${spotbugs.version}</version>
</dependency>
<!-- test -->
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>jicoco-test-kotlin</artifactId>
<version>${jicoco.version}</version>
</dependency>
<dependency>
<groupId>io.pkts</groupId>
<artifactId>pkts-core</artifactId>
<version>3.0.10</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-launcher</artifactId>
<version>${junit.platform.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.kotest</groupId>
<artifactId>kotest-runner-junit5-jvm</artifactId>
<version>${kotest.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.kotest</groupId>
<artifactId>kotest-assertions-core-jvm</artifactId>
<version>${kotest.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.kotest</groupId>
<artifactId>kotest-property-jvm</artifactId>
<version>${kotest.version}</version>
<scope>test</scope>
</dependency>
<!-- Specify this dependency here because kotest needs it, and otherwise an older
version will be pulled in -->
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.mockk</groupId>
<artifactId>mockk-jvm</artifactId>
<version>${mockk.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<version>2.3.1</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<!--<sourceDirectory>${project.basedir}/src/main/kotlin</sourceDirectory>-->
<plugins>
<plugin>
<artifactId>kotlin-maven-plugin</artifactId>
<groupId>org.jetbrains.kotlin</groupId>
<version>${kotlin.version}</version>
<executions>
<execution>
<id>compile</id>
<goals> <goal>compile</goal> </goals>
<configuration>
<sourceDirs>
<sourceDir>${project.basedir}/src/main/kotlin</sourceDir>
<sourceDir>${project.basedir}/src/main/java</sourceDir>
</sourceDirs>
</configuration>
</execution>
<execution>
<id>test-compile</id>
<goals> <goal>test-compile</goal> </goals>
<configuration>
<sourceDirs>
<sourceDir>${project.basedir}/src/test/kotlin</sourceDir>
<sourceDir>${project.basedir}/src/test/java</sourceDir>
</sourceDirs>
</configuration>
</execution>
</executions>
<configuration>
<jvmTarget>17</jvmTarget>
</configuration>
</plugin>
<plugin>
<!-- To try and get both java and kotlin sources in the sources jar -->
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>${build-helper-maven-plugin.version}</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>${project.basedir}/src/main/kotlin</source>
<source>${project.basedir}/src/main/java</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>${maven-source-plugin.version}</version>
<executions>
<execution>
<id>attach-sources</id>
<phase>verify</phase>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
<configuration>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>${maven-compiler-plugin.version}</version>
<executions>
<!-- Replacing default-compile as it is treated specially by maven -->
<execution>
<id>default-compile</id>
<phase>none</phase>
</execution>
<!-- Replacing default-testCompile as it is treated specially by maven -->
<execution>
<id>default-testCompile</id>
<phase>none</phase>
</execution>
<execution>
<id>java-compile</id>
<phase>compile</phase>
<goals> <goal>compile</goal> </goals>
</execution>
<execution>
<id>java-test-compile</id>
<phase>test-compile</phase>
<goals> <goal>testCompile</goal> </goals>
</execution>
</executions>
<configuration>
<source>17</source>
<target>17</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<systemPropertyVariables>
<kotest.framework.classpath.scanning.autoscan.disable>true</kotest.framework.classpath.scanning.autoscan.disable>
</systemPropertyVariables>
<excludes>
<!-- init() currently throws an exception -->
<exclude>DtlsStackTest.*</exclude>
</excludes>
</configuration>
</plugin>
<!-- ktlint: check runs in the verify phase; auto-format with "mvn exec:exec@ktlint-format" -->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>3.6.3</version>
<dependencies>
<dependency>
<groupId>com.pinterest.ktlint</groupId>
<artifactId>ktlint-cli</artifactId>
<version>${ktlint.version}</version>
<classifier>all</classifier>
</dependency>
</dependencies>
<executions>
<execution>
<id>ktlint-check</id>
<phase>verify</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>java</executable>
<includePluginDependencies>true</includePluginDependencies>
<arguments>
<argument>-classpath</argument>
<classpath>
<dependency>com.pinterest.ktlint:ktlint-cli</dependency>
</classpath>
<argument>com.pinterest.ktlint.Main</argument>
<argument>--relative</argument>
<argument>src/main/kotlin/**/*.kt</argument>
<argument>src/test/kotlin/**/*.kt</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>ktlint-format</id>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>java</executable>
<includePluginDependencies>true</includePluginDependencies>
<arguments>
<argument>-classpath</argument>
<classpath>
<dependency>com.pinterest.ktlint:ktlint-cli</dependency>
</classpath>
<argument>com.pinterest.ktlint.Main</argument>
<argument>--format</argument>
<argument>--relative</argument>
<argument>src/main/kotlin/**/*.kt</argument>
<argument>src/test/kotlin/**/*.kt</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<version>3.6.0</version>
<configuration>
<configLocation>checkstyle.xml</configLocation>
</configuration>
<executions>
<execution>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>com.github.spotbugs</groupId>
<artifactId>spotbugs-maven-plugin</artifactId>
<version>${spotbugs-maven-plugin.version}</version>
<configuration>
<excludeFilterFile>${project.basedir}/spotbugs-exclude.xml</excludeFilterFile>
<includeTests>true</includeTests>
</configuration>
<executions>
<execution>
<phase>verify</phase>
<goals><goal>check</goal></goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>com.github.spotbugs</groupId>
<artifactId>spotbugs</artifactId>
<version>${spotbugs.version}</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>jitsi-maven-repository-releases</id>
<layout>default</layout>
<name>Jitsi Maven Repository (Releases)</name>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
<url>https://github.com/jitsi/jitsi-maven-repository/raw/master/releases/</url>
</repository>
<repository>
<id>jitsi-maven-repository-snapshots</id>
<layout>default</layout>
<name>Jitsi Maven Repository (Snapshots)</name>
<releases>
<enabled>false</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
<url>https://github.com/jitsi/jitsi-maven-repository/raw/master/snapshots/</url>
</repository>
</repositories>
<profiles>
<profile>
<id>coverage</id>
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco.version}</version>
<executions>
<execution>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<!-- attached to Maven test phase -->
<execution>
<id>report</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>

84
jitsi-media-transform/spotbugs-exclude.xml Normal file
View File

@@ -0,0 +1,84 @@
<?xml version="1.0" encoding="UTF-8"?>
<FindBugsFilter>
<Match>
<Or>
<!-- Our Packet hierarchy intentionally uses clone() without calling super.clone(). -->
<Bug pattern="CN_IDIOM_NO_SUPER_CALL"/>
<!-- Serialization -->
<Bug pattern="SE_BAD_FIELD"/>
<Bug pattern="SE_BAD_FIELD_STORE"/>
<!-- We expose arrays in multiple places on purpose. -->
<Bug pattern="EI_EXPOSE_REP"/>
<Bug pattern="EI_EXPOSE_REP2"/>
<!-- We throw from constructors on illegal arguments, i.e. programming errors. -->
<Bug pattern="CT_CONSTRUCTOR_THROW"/>
<!-- We don't expect to run on 32-bit platforms. -->
<Bug pattern="AT_NONATOMIC_64BIT_PRIMITIVE"/>
</Or>
</Match>
<!-- Kotlin-specific -->
<Match>
<Or>
<!-- False positives with lateinit -->
<Bug pattern="NP_NONNULL_FIELD_NOT_INITIALIZED_IN_CONSTRUCTOR"/>
<!-- False positives with lateinit -->
<Bug pattern="RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE"/>
<!-- False positives with lateinit -->
<Bug pattern="NP_NULL_PARAM_DEREF"/>
<!-- False positives with lateinit -->
<Bug pattern="NP_NONNULL_RETURN_VIOLATION"/>
<!-- False positives with method references -->
<Bug pattern="NP_PARAMETER_MUST_BE_NONNULL_BUT_MARKED_AS_NULLABLE"/>
<!-- False positives with kotlin's "when" -->
<Bug pattern="SF_SWITCH_NO_DEFAULT"/>
<Bug pattern="SF_SWITCH_FALLTHROUGH"/>
<!-- False positives with lazy vals -->
<Bug pattern="NP_NULL_ON_SOME_PATH"/>
<!-- False positives with lazy vals -->
<Bug pattern="NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE"/>
<!-- False positives with lazy vals -->
<Bug pattern="BC_IMPOSSIBLE_INSTANCEOF"/>
<!-- False positives with Kotlin inlines -->
<Bug pattern="RV_RETURN_VALUE_IGNORED_NO_SIDE_EFFECT"/>
<!-- False positives for the synthetic bridge methods Kotlin generates on anonymous
subclasses of generic collections (e.g. an anonymous LRUCache overriding removeEldestEntry). -->
<Bug pattern="UMAC_UNCALLABLE_METHOD_OF_ANONYMOUS_CLASS"/>
<!-- False positives for an unknown reason -->
<Bug pattern="BC_BAD_CAST_TO_ABSTRACT_COLLECTION"/>
<Bug pattern="RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE"/>
<Bug pattern="UR_UNINIT_READ"/>
<Bug pattern="UC_USELESS_OBJECT"/>
<Bug pattern="AT_STALE_THREAD_WRITE_OF_PRIMITIVE"/>
<Bug pattern="MS_EXPOSE_REP"/>
</Or>
<Source name="~.*\.kt" />
</Match>
<!-- Tests -->
<Match>
<Or>
<!-- Tests have a valid reason to ignore return values (expecting exceptions, etc). -->
<Bug pattern="RV_RETURN_VALUE_IGNORED_NO_SIDE_EFFECT"/>
<!-- Same for exceptions. -->
<Bug pattern="DE_MIGHT_IGNORE"/>
<!-- In tests it is convenient to use Random() randomly. -->
<Bug pattern="DMI_RANDOM_USED_ONLY_ONCE"/>
<!-- Looks like a false positive. -->
<Bug pattern="UMAC_UNCALLABLE_METHOD_OF_ANONYMOUS_CLASS"/>
<!-- Confused by kotest beforeSpec patterns. -->
<Bug pattern="NP_NONNULL_PARAM_VIOLATION" />
<!-- False positives for an unknown reason. -->
<Bug pattern="SA_LOCAL_SELF_ASSIGNMENT"/>
</Or>
<Or>
<Source name="~.*\.*Test.kt" />
<Source name="~.*\.*Test.java" />
<Class name="~.*\.module_tests\..*" />
</Or>
</Match>
</FindBugsFilter>

114
jitsi-media-transform/src/main/java/org/jitsi_modified/impl/neomedia/codec/video/h264/DePacketizer.java Normal file
View File

@@ -0,0 +1,114 @@
/*
* Copyright @ 2015 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi_modified.impl.neomedia.codec.video.h264;
import org.jitsi.utils.logging2.*;
import static org.jitsi_modified.impl.neomedia.codec.video.h264.H264.*;
/**
* Implements <tt>Codec</tt> to represent a depacketizer of H.264 RTP packets
* into NAL units.
*
* @author Lyubomir Marinov
* @author Damian Minkov
*/
public class DePacketizer
{
/**
* The <tt>Logger</tt> used by the <tt>DePacketizer</tt> class and its
* instances for logging output.
*/
private static final Logger logger = new LoggerImpl(DePacketizer.class.toString());
/**
* Returns true if the buffer contains a H264 key frame at offset
* <tt>offset</tt>.
*
* @param buf the byte buffer to check
* @param off the offset in the byte buffer where the actuall data starts
* @param len the length of the data in the byte buffer
* @return true if the buffer contains a H264 key frame at offset
* <tt>offset</tt>.
*/
public static boolean isKeyFrame(byte[] buf, int off, int len)
{
if (buf == null || buf.length < off + Math.max(len, 1))
{
return false;
}
int nalType = buf[off] & kTypeMask;
// Single NAL Unit Packet
if (nalType == kFuA)
{
// Fragmented NAL units (FU-A).
if (parseFuaNaluForKeyFrame(buf, off, len))
{
return true;
}
}
else
{
if (parseSingleNaluForKeyFrame(buf, off, len))
{
return true;
}
}
return false;
}
/**
* Checks if a fragment of a NAL unit from a specific FU-A RTP packet
* payload is keyframe or not.
*/
private static boolean parseFuaNaluForKeyFrame(byte[] buf, int off, int len) {
if (len < kFuAHeaderSize)
{
return false;
}
return ((buf[off + 1] & kTypeMask) == kIdr);
}
/**
* Checks if a fragment of a NAL unit from a specific FU-A RTP packet
* payload is keyframe or not.
*/
private static boolean parseSingleNaluForKeyFrame(byte[] buf, int off, int len)
{
int naluStart = off + kNalHeaderSize;
int naluLength = len - kNalHeaderSize;
int nalType = buf[off] & kTypeMask;
if (nalType == kStapA)
{
// Skip the StapA header (StapA nal type + length).
if (len <= kStapAHeaderSize)
{
logger.error("StapA header truncated.");
return false;
}
if (!verifyStapANaluLengths(buf, naluStart, naluLength))
{
logger.error("StapA packet with incorrect NALU packet lengths.");
return false;
}
nalType = buf[off + kStapAHeaderSize] & kTypeMask;
}
return (nalType == kIdr || nalType == kSps ||
nalType == kPps || nalType == kSei);
}
}

83
jitsi-media-transform/src/main/java/org/jitsi_modified/impl/neomedia/codec/video/h264/H264.java Normal file
View File

@@ -0,0 +1,83 @@
/*
* Copyright @ 2015 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi_modified.impl.neomedia.codec.video.h264;
/**
* Common utilities and constants for H264.
*/
public class H264
{
/**
* The bytes to prefix any NAL unit to be output by this
* <tt>DePacketizer</tt> and given to a H.264 decoder. Includes
* start_code_prefix_one_3bytes. According to "B.1 Byte stream NAL unit
* syntax and semantics" of "ITU-T Rec. H.264 Advanced video coding for
* generic audiovisual services", zero_byte "shall be present" when "the
* nal_unit_type within the nal_unit() is equal to 7 (sequence parameter
* set) or 8 (picture parameter set)" or "the byte stream NAL unit syntax
* structure contains the first NAL unit of an access unit in decoding
* order".
*/
static final byte[] NAL_PREFIX = {0, 0, 0, 1};
/**
* Constants used to detect H264 keyframes in rtp packet
*/
static final byte kTypeMask = 0x1F; // Nalu
static final byte kIdr = 5;
static final byte kSei = 6;
static final byte kSps = 7;
static final byte kPps = 8;
static final byte kStapA = 24;
static final byte kFuA = 28; // Header sizes
static final int kNalHeaderSize = 1;
static final int kFuAHeaderSize = 2;
static final int kLengthFieldSize = 2;
static final int kStapAHeaderSize = kNalHeaderSize + kLengthFieldSize;
static final int kNalUSize = 2;
/**
* Check if Single-Time Aggregation Packet (STAP-A) NAL unit is correctly formed.
*
* @param data STAP-A payload
* @param offset Starting position of NAL unit
* @param lengthRemaining Bytes left in STAP-A
* @return True if STAP-A NAL Unit is correct
*/
static boolean verifyStapANaluLengths(byte[] data, int offset, int lengthRemaining)
{
if (data.length < offset + lengthRemaining)
{
return false;
}
while (lengthRemaining != 0)
{
if (lengthRemaining < kNalUSize)
{
return false;
}
int naluSize = kNalUSize + getUint16(data, offset);
offset += naluSize;
lengthRemaining -= naluSize;
}
return true;
}
static int getUint16(byte[] data, int offset)
{
return ((data[offset] & 0xff) << 8) | (data[offset + 1] & 0xff);
}
}

595
jitsi-media-transform/src/main/java/org/jitsi_modified/impl/neomedia/codec/video/vp8/DePacketizer.java Normal file
View File

@@ -0,0 +1,595 @@
/*
* Copyright @ 2015 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi_modified.impl.neomedia.codec.video.vp8;
import org.jitsi.utils.*;
import org.jitsi.rtp.extensions.ByteKt;
/**
* A depacketizer from VP8.
* See {@link "http://tools.ietf.org/html/rfc7741"}
*
* @author Boris Grozev
* @author George Politis
* @author Jonathan Lennox
*/
public class DePacketizer
{
/**
* Returns true if the buffer contains a VP8 key frame at offset
* <tt>offset</tt>.
*
* @param buf the byte buffer to check
* @param off the offset in the byte buffer where the actual data starts
* @param len the length of the data in the byte buffer
* @return true if the buffer contains a VP8 key frame at offset
* <tt>offset</tt>.
*/
public static boolean isKeyFrame(byte[] buf, int off, int len)
{
// Check if this is the start of a VP8 partition in the payload
// descriptor.
if (!DePacketizer.VP8PayloadDescriptor.isValid(buf, off, len))
{
return false;
}
if (!DePacketizer.VP8PayloadDescriptor.isStartOfFrame(buf, off))
{
return false;
}
int szVP8PayloadDescriptor = DePacketizer
.VP8PayloadDescriptor.getSize(buf, off, len);
return DePacketizer.VP8PayloadHeader.isKeyFrame(
buf, off + szVP8PayloadDescriptor);
}
/**
* A class that represents the VP8 Payload Descriptor structure defined
* in {@link "https://tools.ietf.org/html/rfc7741"}
*/
public static class VP8PayloadDescriptor
{
/**
* I bit from the X byte of the Payload Descriptor.
*/
private static final byte I_BIT = (byte) 0x80;
/**
* K bit from the X byte of the Payload Descriptor.
*/
private static final byte K_BIT = (byte) 0x10;
/**
* L bit from the X byte of the Payload Descriptor.
*/
private static final byte L_BIT = (byte) 0x40;
/**
* M bit from the I byte of the Payload Descriptor.
*/
private static final byte M_BIT = (byte) 0x80;
/**
* Maximum length of a VP8 Payload Descriptor.
*/
public static final int MAX_LENGTH = 6;
/**
* S bit from the first byte of the Payload Descriptor.
*/
private static final byte S_BIT = (byte) 0x10;
/**
* T bit from the X byte of the Payload Descriptor.
*/
private static final byte T_BIT = (byte) 0x20;
/**
* X bit from the first byte of the Payload Descriptor.
*/
private static final byte X_BIT = (byte) 0x80;
/**
* N bit from the first byte of the Payload Descriptor.
*/
private static final byte N_BIT = (byte) 0x20;
/**
* Gets whether a VP8 payload has a temporal layer index (TID).
*
* @param buf the byte buffer that holds the VP8 packet.
* @param off the offset in the byte buffer where the VP8 payload starts.
* @param len the length of the VP8 payload.
* @return true if the VP8 payload has a temporal layer index, false if not.
*/
public static boolean hasTemporalLayerIndex(byte[] buf, int off, int len)
{
return getTemporalLayerIndex(buf, off, len) != -1;
}
/**
* Gets the temporal layer index (TID), if that's set.
*
* @param buf the byte buffer that holds the VP8 packet.
* @param off the offset in the byte buffer where the VP8 packet starts.
* @param len the length of the VP8 packet.
* @return the temporal layer index (TID), if that's set, -1 otherwise.
*/
public static int getTemporalLayerIndex(byte[] buf, int off, int len)
{
if (buf == null || buf.length < off + len || len < 2)
{
return -1;
}
if ((buf[off] & X_BIT) == 0 || (buf[off + 1] & T_BIT) == 0)
{
return -1;
}
int sz = getSize(buf, off, len);
if (buf.length < off + sz || sz < 1)
{
return -1;
}
return (buf[off + sz - 1] & 0xc0) >> 6;
}
/**
* Sets the temporal layer index (TID), if there's room for it.
*
* @param buf the byte buffer that holds the VP8 packet.
* @param off the offset in the byte buffer where the VP8 packet starts.
* @param len the length of the VP8 packet.
* @param tid the temporal layer value to
* @return true if the layer was set successfully, false otherwise.
*/
public static boolean setTemporalLayerIndex(byte[] buf, int off, int len, int tid)
{
if (tid < 0 || tid > 3)
{
throw new IllegalArgumentException("Bad tid value " + tid);
}
if (buf == null || buf.length < off + len || len < 2)
{
return false;
}
if ((buf[off] & X_BIT) == 0 || (buf[off + 1] & T_BIT) == 0)
{
return false;
}
int sz = getSize(buf, off, len);
if (buf.length < off + sz || sz < 1)
{
return false;
}
buf[off + sz - 1] = (byte)((tid << 6) | buf[off + sz - 1] & 0x3f);
return true;
}
/**
* Returns a simple Payload Descriptor, with PartID = 0, the 'start
* of partition' bit set according to <tt>startOfPartition</tt>, and
* all other bits set to 0.
*
* @param startOfPartition whether to 'start of partition' bit should be
* set
* @return a simple Payload Descriptor, with PartID = 0, the 'start
* of partition' bit set according to <tt>startOfPartition</tt>, and
* all other bits set to 0.
*/
public static byte[] create(boolean startOfPartition)
{
byte[] pd = new byte[1];
pd[0] = startOfPartition ? (byte) 0x10 : 0;
return pd;
}
/**
* The size in bytes of the Payload Descriptor at offset
* <tt>offset</tt> in <tt>input</tt>. The size is between 1 and 6.
*
* @param baf the <tt>ByteArrayBuffer</tt> that holds the VP8 payload
* descriptor.
* @return The size in bytes of the Payload Descriptor at offset
* <tt>offset</tt> in <tt>input</tt>, or -1 if the input is not a valid
* VP8 Payload Descriptor. The size is between 1 and 6.
*/
public static int getSize(ByteArrayBuffer baf)
{
if (baf == null)
{
return -1;
}
return getSize(baf.getBuffer(), baf.getOffset(), baf.getLength());
}
/**
* The size in bytes of the Payload Descriptor at offset
* <tt>offset</tt> in <tt>input</tt>. The size is between 1 and 6.
*
* @param input input
* @param offset offset
* @param length length
* @return The size in bytes of the Payload Descriptor at offset
* <tt>offset</tt> in <tt>input</tt>, or -1 if the input is not a valid
* VP8 Payload Descriptor. The size is between 1 and 6.
*/
public static int getSize(byte[] input, int offset, int length)
{
if (!isValid(input, offset, length))
return -1;
if ((input[offset] & X_BIT) == 0)
return 1;
int size = 2;
if ((input[offset + 1] & I_BIT) != 0)
{
size++;
if ((input[offset + 2] & M_BIT) != 0)
size++;
}
if ((input[offset + 1] & L_BIT) != 0)
size++;
if ((input[offset + 1] & (T_BIT | K_BIT)) != 0)
size++;
return size;
}
/**
* Determines whether the VP8 payload specified in the buffer that is
* passed as an argument has a picture ID or not.
*
* @param buf the byte buffer that contains the VP8 payload.
* @param off the offset in the byte buffer where the VP8 payload
* starts.
* @param len the length of the VP8 payload in the byte buffer.
* @return true if the VP8 payload contains a picture ID, false
* otherwise.
*/
public static boolean hasPictureId(byte[] buf, int off, int len)
{
return isValid(buf, off, len)
&& (buf[off] & X_BIT) != 0 && (buf[off + 1] & I_BIT) != 0;
}
/**
* Determines whether the VP8 payload specified in the buffer that is
* passed as an argument has an extended picture ID or not.
*
* @param buf the byte buffer that contains the VP8 payload.
* @param off the offset in the byte buffer where the VP8 payload
* starts.
* @param len the length of the VP8 payload in the byte buffer.
* @return true if the VP8 payload contains an extended picture ID,
* false otherwise.
*/
public static boolean hasExtendedPictureId(byte[] buf, int off, int len)
{
return hasPictureId(buf, off, len) && (buf[off + 2] & M_BIT) != 0;
}
/**
* Determines whether the VP8 payload specified in the buffer that is
* passed as an argument has a TL0PICIDX or not.
*
* @param buf the byte buffer that contains the VP8 payload.
* @param off the offset in the byte buffer where the VP8 payload
* starts.
* @param len the length of the VP8 payload in the byte buffer.
* @return true if the VP8 payload contains a TL0PICIDX,
* false otherwise.
*/
public static boolean hasTL0PICIDX(byte[] buf, int off, int len)
{
return getLByteOffset(buf, off, len) >= 0;
}
/**
* Gets the value of the PictureID field of a VP8 Payload Descriptor.
*
* @param input
* @param offset
* @return the value of the PictureID field of a VP8 Payload Descriptor,
* or -1 if the fields is not present.
*/
public static int getPictureId(byte[] input, int offset)
{
if (input == null
|| !hasPictureId(input, offset, input.length - offset))
{
return -1;
}
boolean isLong = (input[offset + 2] & M_BIT) != 0;
if (isLong)
return (input[offset + 2] & 0x7f) << 8
| (input[offset + 3] & 0xff);
else
return input[offset + 2] & 0x7f;
}
/**
* Sets the extended picture ID for the VP8 payload specified in the
* buffer that is passed as an argument.
*
* @param buf the byte buffer that contains the VP8 payload.
* @param off the offset in the byte buffer where the VP8 payload
* starts.
* @param len the length of the VP8 payload in the byte buffer.
* @return true if the operation succeeded, false otherwise.
*/
public static boolean setExtendedPictureId(
byte[] buf, int off, int len, int val)
{
if (!hasExtendedPictureId(buf, off, len))
{
return false;
}
buf[off + 2] = (byte) (0x80 | (val >> 8) & 0x7F);
buf[off + 3] = (byte) (val & 0xFF);
return true;
}
/**
* Sets the TL0PICIDX field for the VP8 payload specified in the
* buffer that is passed as an argument.
*
* @param buf the byte buffer that contains the VP8 payload.
* @param off the offset in the byte buffer where the VP8 payload
* starts.
* @param len the length of the VP8 payload in the byte buffer.
* @return true if the operation succeeded, false otherwise.
*/
public static boolean setTL0PICIDX(byte[] buf, int off, int len,
int val)
{
int offL = getLByteOffset(buf, off, len);
if (offL < 0)
{
return false;
}
buf[off + offL] = (byte) val;
return true;
}
/**
* Checks whether the arguments specify a valid buffer.
*
* @param buf
* @param off
* @param len
* @return true if the arguments specify a valid buffer, false
* otherwise.
*/
public static boolean isValid(byte[] buf, int off, int len)
{
return buf != null && buf.length >= off + len && off > -1
&& len > 0;
}
/**
* Checks whether the '<tt>start of partition</tt>' bit is set in the
* VP8 Payload Descriptor at offset <tt>offset</tt> in <tt>input</tt>.
*
* @param input input
* @param offset offset
* @return <tt>true</tt> if the '<tt>start of partition</tt>' bit is set,
* <tt>false</tt> otherwise.
*/
public static boolean isStartOfPartition(byte[] input, int offset)
{
return (input[offset] & S_BIT) != 0;
}
/**
* Sets the '<tt>start of partition</tt>' bit in the
* VP8 Payload Descriptor at offset <tt>offset</tt> in <tt>input</tt>.
*
* @param input input
* @param offset offset
* @param start whether it is start or not.
*/
public static void setStartOfPartition(byte[] input, int offset, boolean start)
{
input[offset] = ByteKt.putBitWithMask(input[offset], S_BIT, start);
}
/**
* Returns <tt>true</tt> if both the '<tt>start of partition</tt>' bit
* is set and the <tt>PID</tt> fields has value 0 in the VP8 Payload
* Descriptor at offset <tt>offset</tt> in <tt>input</tt>.
*
* @param input
* @param offset
* @return <tt>true</tt> if both the '<tt>start of partition</tt>' bit
* is set and the <tt>PID</tt> fields has value 0 in the VP8 Payload
* Descriptor at offset <tt>offset</tt> in <tt>input</tt>.
*/
public static boolean isStartOfFrame(byte[] input, int offset)
{
return isStartOfPartition(input, offset)
&& getPartitionId(input, offset) == 0;
}
/**
* Returns the value of the <tt>PID</tt> (partition ID) field of the
* VP8 Payload Descriptor at offset <tt>offset</tt> in <tt>input</tt>.
*
* @param input
* @param offset
* @return the value of the <tt>PID</tt> (partition ID) field of the
* VP8 Payload Descriptor at offset <tt>offset</tt> in <tt>input</tt>.
*/
public static int getPartitionId(byte[] input, int offset)
{
return input[offset] & 0x07;
}
/**
* Gets a boolean that indicates whether or not the non-reference bit is
* set.
*
* @param buf the byte buffer that holds the VP8 payload descriptor.
* @param off the offset in the byte buffer where the payload descriptor
* starts.
* @param len the length of the payload descriptor in the byte buffer.
* @return true if the non-reference bit is NOT set, false otherwise.
*/
public static boolean isReference(byte[] buf, int off, int len)
{
return (buf[off] & N_BIT) == 0;
}
/**
* Gets the TL0PICIDX from the payload descriptor.
*
* @param buf the byte buffer that holds the VP8 payload descriptor.
* @param off the offset in the byte buffer where the payload descriptor
* starts.
* @param len the length of the payload descriptor in the byte buffer.
* @return the TL0PICIDX from the payload descriptor, or -1 if the packet
* does not have one.
*/
public static int getTL0PICIDX(byte[] buf, int off, int len)
{
int offL = getLByteOffset(buf, off, len);
if (offL < 0)
{
return -1;
}
return buf[off + offL] & 0xff;
}
/**
* Return the offset of the {@code L} byte relative to the start of
* the payload descriptor, or -1 if the PR has no {@code L} byte.
*
* @param buf the byte buffer that holds the VP8 payload descriptor.
* @param off the offset in the byte buffer where the payload descriptor
* starts.
* @param len the length of the payload descriptor in the byte buffer.
* @return
*/
private static int getLByteOffset(byte[] buf, int off, int len)
{
if (!isValid(buf, off, len)
|| (buf[off] & X_BIT) == 0 || (buf[off + 1] & L_BIT) == 0)
{
return -1;
}
// We could use hasPictureId(), but it would unnecessarily repeat
// the isValid() check.
int offL = 2;
if ((buf[off + 1] & I_BIT) != 0)
{
offL++;
if ((buf[off + 2] & M_BIT) != 0)
{
offL++;
}
}
return offL;
}
}
/**
* A class that represents the VP8 Payload Header structure described
* in {@link "http://tools.ietf.org/html/rfc7741"}
*/
public static class VP8PayloadHeader
{
/**
* S bit of the Payload Header.
*/
private static final byte S_BIT = (byte) 0x01;
/**
* Returns true if the <tt>P</tt> (inverse key frame flag) field of the
* VP8 Payload Header at offset <tt>offset</tt> in <tt>input</tt> is 0.
*
* @return true if the <tt>P</tt> (inverse key frame flag) field of the
* VP8 Payload Header at offset <tt>offset</tt> in <tt>input</tt> is 0,
* false otherwise.
*/
public static boolean isKeyFrame(byte[] input, int offset)
{
// When set to 0 the current frame is a key frame. When set to 1
// the current frame is an interframe. Defined in [RFC6386]
return (input[offset] & S_BIT) == 0;
}
/* Sets the <tt>P</tt> (inverse key frame flag) field of the
* VP8 Payload header at offset <tt>offset</tt> in <tt>input</tt>.
* Sets P to 0 if keyFrame is false, or to 1 if keyFrame is true.
*/
public static void setKeyFrame(byte[] input, int offset, boolean keyFrame)
{
input[offset] = ByteKt.putBitWithMask(input[offset], S_BIT, !keyFrame);
}
}
/**
* A class that represents a keyframe header structure (see RFC 6386,
* paragraph 9.1).
*
* @author George Politis
*/
public static class VP8KeyframeHeader
{
// From RFC 6386, the keyframe header has this format.
//
// Start code byte 0 0x9d
// Start code byte 1 0x01
// Start code byte 2 0x2a
//
// 16 bits : (2 bits Horizontal Scale << 14) | Width (14 bits)
// 16 bits : (2 bits Vertical Scale << 14) | Height (14 bits)
/**
* @return the height of this instance.
*/
public static int getHeight(byte[] buf, int off)
{
return (((buf[off + 6] & 0xff) << 8) | buf[off + 5] & 0xff) & 0x3fff;
}
}
}

810
jitsi-media-transform/src/main/java/org/jitsi_modified/impl/neomedia/codec/video/vp9/DePacketizer.java Normal file
View File

@@ -0,0 +1,810 @@
/*
* Copyright @ 2015 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi_modified.impl.neomedia.codec.video.vp9;
import org.jitsi.rtp.extensions.*;
/**
* A depacketizer from VP9.
* See {@link "https://tools.ietf.org/html/draft-ietf-payload-vp9-10"}
*
* @author George Politis
*/
public class DePacketizer
{
/*
* VP9 Payload header, flexible mode (F=1):
* 0 1 2 3 4 5 6 7
* +-+-+-+-+-+-+-+-+
* |I|P|L|F|B|E|V|Z| (REQUIRED)
* +-+-+-+-+-+-+-+-+
* I: |M| PICTURE ID | (REQUIRED)
* +-+-+-+-+-+-+-+-+
* M: | EXTENDED PID | (RECOMMENDED)
* +-+-+-+-+-+-+-+-+
* L: | TID |U| SID |D| (CONDITIONALLY RECOMMENDED)
* +-+-+-+-+-+-+-+-+ -\
* P,F: | P_DIFF |N| (CONDITIONALLY REQUIRED) - up to 3 times
* +-+-+-+-+-+-+-+-+ -/
* V: | SS |
* | .. |
* +-+-+-+-+-+-+-+-+
*
* Non-flexible mode (F=0):
* 0 1 2 3 4 5 6 7
* +-+-+-+-+-+-+-+-+
* |I|P|L|F|B|E|V|Z| (REQUIRED)
* +-+-+-+-+-+-+-+-+
* I: |M| PICTURE ID | (RECOMMENDED)
* +-+-+-+-+-+-+-+-+
* M: | EXTENDED PID | (RECOMMENDED)
* +-+-+-+-+-+-+-+-+
* L: | TID |U| SID |D| (CONDITIONALLY RECOMMENDED)
* +-+-+-+-+-+-+-+-+
* | TL0PICIDX | (CONDITIONALLY REQUIRED)
* +-+-+-+-+-+-+-+-+
* V: | SS |
* | .. |
* +-+-+-+-+-+-+-+-+
*/
/**
* I bit from the first byte of the Payload Descriptor:
* Picture ID present.
*/
private static final byte I_BIT = (byte) (1 << 7);
/**
* P bit from the first byte of the Payload Descriptor:
* Inter-picture predicted frame.
*/
private static final byte P_BIT = (byte) (1 << 6);
/**
* L bit from the first byte of the Payload Descriptor:
* Layer indices present.
*/
private static final byte L_BIT = (byte) (1 << 5);
/**
* F bit from the first byte of the Payload Descriptor:
* flexible mode.
*/
private static final byte F_BIT = (byte) (1 << 4);
/**
* B bit from the first byte of the Payload Descriptor:
* Start of frame.
*/
private static final byte B_BIT = (byte) (1 << 3);
/**
* E bit from the first byte of the Payload Descriptor:
* End of frame.
*/
private static final byte E_BIT = (byte) (1 << 2);
/**
* V bit from the first byte of the Payload Descriptor:
* Scalability structure present.
*/
private static final byte V_BIT = (byte) (1 << 1);
/**
* Z bit from first byte of the Payload Descriptor:
* Not an upper-level reference frame
*/
private static final byte Z_BIT = (byte)1;
/**
* M bit from the first picture ID byte of the Payload Descriptor:
* extended picture ID. Only present if I is true in the first byte.
*/
private static final byte M_BIT = (byte) (1 << 7);
/**
* U bit from the layer indices byte of the Payload Descriptor:
* switching up point. Only present if L is true in the first byte.
*/
private static final byte U_BIT = (byte) (1 << 4);
/**
* D bit from the layer indices byte of the Payload Descriptor:
* inter-layer dependency. Only present if L is true in the first byte.
*/
private static final byte D_BIT = (byte) (1);
/**
* N bit from the picture difference byte of the Payload Descriptor:
* more picture diffs present. Only present if P and F are true in the first byte.
*/
private static final byte N_BIT = (byte) (1);
/**
* A class that represents the VP9 Payload Descriptor structure defined
* in {@link "https://tools.ietf.org/html/draft-ietf-payload-vp9-10"}
*/
public static class VP9PayloadDescriptor
{
/**
* Returns <tt>true</tt> if the arguments specify a valid non-empty
* VP9 packet.
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
*
* @return <tt>true</tt> if the arguments specify a valid non-empty
* buffer.
*/
private static boolean isValid(byte[] buf, int off, int len)
{
return
(buf != null && buf.length >= off + len && off > -1 && len > 0);
}
/**
* Returns <tt>true</tt> if the B bit from the first byte of the payload
* descriptor has value 0. (Note this is start of frame, not start of picture!)
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
*
* @return <tt>true</tt> if the B bit from the first byte of the
* payload descriptor has value 0, false otherwise.
*
*/
public static boolean isStartOfFrame(byte[] buf, int off, int len)
{
// Check if this is the start of a VP9 layer frame in the payload
// descriptor.
return isValid(buf, off, len) && (buf[off] & B_BIT) != 0;
}
/**
* Sets the B bit (start of frame) in the first byte of the payload
* descriptor.
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
* @param start the new value of the start of frame bit.
*/
public static void setStartOfFrame(byte[] buf, int off, int len, boolean start)
{
// Check if this is the start of a VP9 layer frame in the payload
// descriptor.
if (!isValid(buf, off, len))
{
throw new IllegalStateException("Can't set startOfFrame for invalid VP9 packet");
}
buf[off] = ByteKt.putBitWithMask(buf[off], B_BIT, start);
}
/**
* Returns <tt>true</tt> if the E bit from the first byte of the payload
* descriptor has value 0. (Note this is end of frame, not end of picture!)
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
*
* @return <tt>true</tt> if the E bit from the first byte of the
* payload descriptor has value 0, false otherwise.
*/
public static boolean isEndOfFrame(byte[] buf, int off, int len)
{
// Check if this is the end of a VP9 layer frame in the payload
// descriptor.
return isValid(buf, off, len) && (buf[off] & E_BIT) != 0;
}
/**
* Sets the E bit (end of frame) in the first byte of the payload
* descriptor.
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
* @param end the new value of the end of frame bit.
*/
public static void setEndOfFrame(byte[] buf, int off, int len, boolean end)
{
if (!isValid(buf, off, len))
{
throw new IllegalStateException("Can't set endOfFrame for invalid VP9 packet");
}
buf[off] = ByteKt.putBitWithMask(buf[off], E_BIT, end);
}
/**
* Returns <tt>true</tt> if the packet is encoded in flexible mode
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
*
* @return <tt>true</tt> if the packet is encoded in flexible mode,
* false otherwise
*/
public static boolean isFlexibleMode(byte[] buf, int off, int len)
{
return isValid(buf, off, len) && (buf[off] & F_BIT) != 0;
}
/**
* Returns <tt>true</tt> if the packet contains a scalability structure
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
*
* @return <tt>true</tt> if the packet contains a scalability structure,
* false otherwise
*/
public static boolean hasScalabilityStructure(byte[] buf, int off, int len)
{
return isValid(buf, off, len) && (buf[off] & V_BIT) != 0;
}
/**
* Returns <tt>true</tt> if the packet might be an upper-level reference.
*
* (Note this is inverted from the sense of the Z bit in the payload header.)
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
*
* @return <tt>true</tt> if the packet contains a scalability structure,
* false otherwise
*/
public static boolean isUpperLevelReference(byte[] buf, int off, int len)
{
return isValid(buf, off, len) && (buf[off] & Z_BIT) == 0;
}
/**
* Sets the Z bit (upper layer reference) in the first byte of the payload
* descriptor.
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
* @param ref whether the frame might be an upper-layer reference.
* (Note this is inverted from the sense of the Z bit in the payload header.)
*/
public static void setUpperLevelReference(byte[] buf, int off, int len, boolean ref)
{
if (!isValid(buf, off, len))
{
throw new IllegalStateException("Can't set upperLevelReference for invalid VP9 packet");
}
buf[off] = ByteKt.putBitWithMask(buf[off], Z_BIT, !ref);
}
/**
* Returns <tt>true</tt> if the packet has a picture ID.
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
*
* @return <tt>true</tt> if the packet has a picture ID, <tt>false</tt>
* if not.
*/
public static boolean hasPictureId(byte[] buf, int off, int len)
{
return isValid(buf, off, len) && (buf[off] & I_BIT) != 0 && len > 1;
}
/**
* Returns <tt>true</tt> if the packet has an extended (15-bit) picture ID.
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
*
* @return <tt>true</tt> if the packet has a picture ID, <tt>false</tt>
* if not.
*/
public static boolean hasExtendedPictureId(byte[] buf, int off, int len)
{
return hasPictureId(buf, off, len) &&
(buf[off+1] & M_BIT) != 0 && len > 2;
}
/**
* Gets the value of the PictureID field of a VP9 Payload Descriptor.
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
*
* @return <tt>true</tt> if the packet has a picture ID, <tt>false</tt>
* if not.
*/
public static int getPictureId(byte[] buf, int off, int len)
{
if (!hasPictureId(buf, off, len))
{
return -1;
}
if (hasExtendedPictureId(buf, off, len))
{
return (buf[off + 1] & 0x7f) << 8
| (buf[off + 2] & 0xff);
}
else
{
return buf[off + 1] & 0x7f;
}
}
/**
* Sets the extended picture ID for the VP9 payload specified in the
* buffer that is passed as an argument.
*
* @param buf the byte buffer that contains the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload
* starts.
* @param len the length of the VP9 payload in the byte buffer.
* @return true if the operation succeeded, false otherwise.
*/
public static boolean setExtendedPictureId(
byte[] buf, int off, int len, int val)
{
if (!hasExtendedPictureId(buf, off, len))
{
return false;
}
buf[off + 1] = (byte) (0x80 | (val >> 8) & 0x7F);
buf[off + 2] = (byte) (val & 0xFF);
return true;
}
/**
* Returns <tt>true</tt> if the packet contains an inter-picture
* predicted frame.
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
*
* @return <tt>true</tt> if the packet contains an inter-picture
* predicted frame, <tt>false</tt> if not.
*/
public static boolean isInterPicturePredicted(byte[] buf, int off, int len)
{
return isValid(buf, off, len) && (buf[off] & P_BIT) != 0;
}
/**
* Sets the P bit (inter-picture predicted) in the first byte of the payload
* descriptor.
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
* @param pred the new value of the inter-picture predicted bit.
*/
public static void setInterPicturePredicted(byte[] buf, int off, int len, boolean pred)
{
if (!isValid(buf, off, len))
{
throw new IllegalStateException("Can't set interPicturePredicted for invalid VP9 packet");
}
buf[off] = ByteKt.putBitWithMask(buf[off], P_BIT, pred);
}
/**
* Returns <tt>true</tt> if the packet is part of a keyframe.
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
*
* @return <tt>true</tt> if the packet is part of a keyframe, false if not.
*/
public static boolean isKeyFrame(byte[] buf, int off, int len)
{
if (!isValid(buf, off, len))
{
return false;
}
if ((buf[off] & P_BIT) != 0)
{
return false;
}
if (!hasLayerIndices(buf, off, len))
{
/* P bit without layer indices is a key frame. */
return true;
}
return (getSpatialLayerIndex(buf, off, len) == 0 ||
!usesInterLayerDependency(buf, off, len));
}
/**
* Query whether the packet has layer indices values.
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
*
* @return the temporal layer index (TID), if that's set, -1 otherwise.
*/
public static boolean hasLayerIndices(byte[] buf, int off, int len)
{
return (isValid(buf, off, len) && (buf[off] & L_BIT) != 0);
}
private static int getLayerIndexOffset(byte[] buf, int off, int len)
{
if (!hasLayerIndices(buf, off, len))
{
return -1;
}
int loff = off + 1;
if ((buf[off] & I_BIT) != 0)
{
loff += 1;
if ((buf[off + 1] & M_BIT) != 0)
{
// extended pid.
loff += 1;
}
}
if (loff >= off+len)
{
return -1;
}
return loff;
}
/**
* Determines whether the VP9 payload specified in the buffer that is
* passed as an argument has a TL0PICIDX or not.
*
* @param buf the byte buffer that contains the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload
* starts.
* @param len the length of the VP9 payload in the byte buffer.
* @return true if the VP9 payload contains a TL0PICIDX,
* false otherwise.
*/
public static boolean hasTL0PICIDX(byte[] buf, int off, int len)
{
int loff = getLayerIndexOffset(buf, off, len);
if (loff < 0 || loff + 1 >= off+len)
{
return false;
}
return !isFlexibleMode(buf, off, len);
}
/**
* Gets the TL0PICIDX from the payload descriptor.
*
* @param buf the byte buffer that holds the VP9 payload descriptor.
* @param off the offset in the byte buffer where the payload descriptor
* starts.
* @param len the length of the payload descriptor in the byte buffer.
* @return the TL0PICIDX from the payload descriptor, or -1 if the packet
* does not have one.
*/
public static int getTL0PICIDX(byte[] buf, int off, int len)
{
int loff = getLayerIndexOffset(buf, off, len);
if (!hasTL0PICIDX(buf, off, len))
return -1;
return buf[loff+1] & 0xff;
}
/**
* Sets the TL0PICIDX field for the VP9 payload specified in the
* buffer that is passed as an argument.
*
* @param buf the byte buffer that contains the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload
* starts.
* @param len the length of the VP9 payload in the byte buffer.
* @return true if the operation succeeded, false otherwise.
*/
public static boolean setTL0PICIDX(byte[] buf, int off, int len,
int val)
{
if (!hasTL0PICIDX(buf, off, len))
{
return false;
}
int loff = getLayerIndexOffset(buf, off, len);
buf[loff+1] = (byte) val;
return true;
}
/**
* Gets the temporal layer index (TID), if that's set.
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
*
* @return the temporal layer index (TID), if that's set, -1 otherwise.
*/
public static int getTemporalLayerIndex(byte[] buf, int off, int len)
{
int loff = getLayerIndexOffset(buf, off, len);
if (loff < 0)
{
return -1;
}
return (buf[loff] & 0xE0) >> 5;
}
/**
* Gets the spatial layer index (SID), if that's set.
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
*
* @return the spatial layer index (SID), if that's set, -1 otherwise.
*/
public static int getSpatialLayerIndex(byte[] buf, int off, int len)
{
int loff = getLayerIndexOffset(buf, off, len);
if (loff < 0)
{
return -1;
}
return (buf[loff] & 0xE) >> 1;
}
/**
* Gets whether this frame is a temporal switching-up point, where it is
* valid to switch the stream to a higher temporal layer.
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
*
* @return <tt>true</tt> if the packet is a switching-up point, <tt>false</tt>
* if not.
*/
public static boolean isSwitchingUpPoint(byte[] buf, int off, int len)
{
int loff = getLayerIndexOffset(buf, off, len);
if (loff < 0)
{
return false;
}
return (buf[loff] & U_BIT) != 0;
}
/**
* Gets whether this frame uses inter-layer dependency, where the frame
* depends on lower spatial-layer frames of the same picture.
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
*
* @return <tt>true</tt> if the packet uses inter-layer dependency, <tt>false</tt>
* if not.
*/
public static boolean usesInterLayerDependency(byte[] buf, int off, int len)
{
int loff = getLayerIndexOffset(buf, off, len);
if (loff < 0)
{
return false;
}
return (buf[loff] & D_BIT) != 0;
}
/**
* Sets the packet's layer indices, if there's room for it in the packet.
*
* @param buf the byte buffer that holds the VP9 payload.
* @param off the offset in the byte buffer where the VP9 payload starts.
* @param len the length of the VP9 payload.
* @param sid the spatial layer of the packet
* @param tid the temporal layer of the packet
* @param isSwitchingUpPoint whether the packet is a switching-up point
* @param usesInterLayerDependency whether the packet uses inter-layer dependency
*
* @return Whether the layer indices were successfully set
*/
public static boolean setLayerIndices(byte[] buf, int off, int len,
int sid, int tid, boolean isSwitchingUpPoint, boolean usesInterLayerDependency)
{
int loff = getLayerIndexOffset(buf, off, len);
if (loff < 0)
{
return false;
}
if (sid < 0 || sid > 8)
{
throw new IllegalArgumentException("Invalid spatial ID " + sid);
}
if (tid < 0 || tid > 8)
{
throw new IllegalArgumentException("Invalid spatial ID " + sid);
}
buf[loff] = (byte)((tid << 5) |
(isSwitchingUpPoint ? U_BIT : 0) |
(sid << 1) |
(usesInterLayerDependency ? D_BIT : 0));
return true;
}
/**
* The size in bytes of the Payload Descriptor at off
* <tt>off</tt> in <tt>buf</tt>. The size is at least 1.
*
* @param buf buf
* @param off off
* @param len len
* @return The size in bytes of the Payload Descriptor at off
* <tt>off</tt> in <tt>buf</tt>, or -1 if the buffer is not a valid
* VP9 Payload Descriptor. The size is at least 1.
*/
public static int getSize(byte[] buf, int off, int len)
{
if (!isValid(buf, off, len))
return -1;
int pos = off + 1;
if ((buf[off] & I_BIT) != 0)
{
boolean extended = (buf[pos] & M_BIT) != 0;
pos++;
if (extended)
{
pos++;
}
}
if ((buf[off] & L_BIT) != 0)
{
pos++;
if ((buf[off] & F_BIT) == 0)
{
pos++;
}
}
if ((buf[off] & F_BIT) != 0 && (buf[off] & P_BIT) != 0)
{
for (int i = 0; i < 3; i++) {
int hasNext = buf[pos] & N_BIT;
pos++;
if (hasNext == 0)
{
break;
}
}
}
if ((buf[off] & V_BIT) != 0)
{
/* SS present */
int n_s = (buf[pos] & 0xE0) >> 5;
boolean resPresent = ((buf[pos] & (1 << 4)) != 0);
boolean pgPresent = ((buf[pos] & (1 << 3)) != 0);
pos++;
if (resPresent)
{
pos += 4 * (n_s + 1);
}
if (pgPresent)
{
int n_g = (buf[pos] & 0xff);
int i;
pos++;
for (i = 0; i < n_g; i++)
{
int r = (buf[pos] & 0x0C) >> 2;
pos += r + 1;
}
}
}
return pos - off;
}
/**
* The offset of the scalability structure within the buffer, if
* the buffer contains a VP9 packet with a scalability structure; otherwise -1.
*
* @param buf buf
* @param off off
* @param len len
* @return The size in bytes of the Payload Descriptor at off
* <tt>off</tt> in <tt>buf</tt>, or -1 if the buffer is not a valid
* VP9 Payload Descriptor. The size is at least 1.
*/
public static int getScalabilityStructureOffset(byte[] buf, int off, int len)
{
if (!hasScalabilityStructure(buf, off, len))
{
return -1;
}
int pos = off + 1;
if ((buf[off] & I_BIT) != 0)
{
boolean extended = (buf[pos] & M_BIT) != 0;
pos++;
if (extended)
{
pos++;
}
}
if ((buf[off] & L_BIT) != 0)
{
pos++;
if ((buf[off] & F_BIT) == 0)
{
pos++;
}
}
if ((buf[off] & F_BIT) != 0 && (buf[off] & P_BIT) != 0)
{
do {
pos++;
} while ((buf[pos] & N_BIT) != 0);
}
return pos;
}
}
}

532
jitsi-media-transform/src/main/java/org/jitsi_modified/impl/neomedia/rtp/remotebitrateestimator/AimdRateControl.java Normal file
View File

@@ -0,0 +1,532 @@
/*
* Copyright @ 2015 Atlassian Pty Ltd
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi_modified.impl.neomedia.rtp.remotebitrateestimator;
import org.jetbrains.annotations.*;
import org.jitsi.utils.logging.DiagnosticContext;
import org.jitsi.utils.logging.TimeSeriesLogger;
/**
* A rate control implementation based on additive increases of bitrate when no
* over-use is detected and multiplicative decreases when over-uses are
* detected. When we think the available bandwidth has changes or is unknown, we
* will switch to a "slow-start mode" where we increase multiplicatively.
*
* webrtc/modules/remote_bitrate_estimator/aimd_rate_control.cc
* webrtc/modules/remote_bitrate_estimator/aimd_rate_control.h
*
* @author Lyubomir Marinov
*/
class AimdRateControl
{
/**
* The <tt>TimeSeriesLogger</tt> used by the
* <tt>RemoteBitrateEstimatorAbsSendTime</tt> class and its instances for
* logging output.
*/
private static final TimeSeriesLogger logger
= TimeSeriesLogger.getTimeSeriesLogger(AimdRateControl.class);
private static final int kDefaultRttMs = 200;
private static final long kInitializationTimeMs = 5000;
private static final long kFirstIncomingEstimateExpirationMs = 2 * kInitializationTimeMs;
private static final long kLogIntervalMs = 1000;
private static final long kMaxFeedbackIntervalMs = 1000;
private static final long kMinFeedbackIntervalMs = 200;
private static final int kRtcpSize = 80;
private static final double kWithinIncomingBitrateHysteresis = 1.05;
private final DiagnosticContext diagnosticContext;
private float avgMaxBitrateKbps;
private float beta;
private boolean bitrateIsInitialized;
private long currentBitrateBps;
private final RateControlInput currentInput
= new RateControlInput(BandwidthUsage.kBwNormal, 0L, 0D);
private boolean inExperiment;
private long minConfiguredBitrateBps;
private RateControlRegion rateControlRegion;
private RateControlState rateControlState;
private long rtt;
private long timeFirstIncomingEstimate;
private long timeLastBitrateChange;
private long timeOfLastLog;
private boolean updated;
private float varMaxBitrateKbps;
public AimdRateControl(@NotNull DiagnosticContext diagnosticContext)
{
reset();
this.diagnosticContext = diagnosticContext;
}
private long additiveRateIncrease(
long nowMs,
long lastMs,
long responseTimeMs)
{
if (responseTimeMs <= 0)
throw new IllegalArgumentException("responseTimeMs");
double beta = 0.0;
if (lastMs > 0)
{
beta
= Math.min((nowMs - lastMs) / (double) responseTimeMs, 1.0);
if (inExperiment)
beta /= 2.0;
}
double bitsPerFrame = (double) currentBitrateBps / 30.0;
double packetsPerFrame = Math.ceil(bitsPerFrame / (8.0 * 1200.0));
double avgPacketSizeBits = bitsPerFrame / packetsPerFrame;
return (long) Math.max(1000.0, beta * avgPacketSizeBits);
}
private long changeBitrate(
long currentBitrateBps,
long incomingBitrateBps,
long nowMs)
{
if (!updated)
return this.currentBitrateBps;
// An over-use should always trigger us to reduce the bitrate, even though
// we have not yet established our first estimate. By acting on the over-use,
// we will end up with a valid estimate.
if (!bitrateIsInitialized
&& currentInput.bwState != BandwidthUsage.kBwOverusing)
{
return this.currentBitrateBps;
}
updated = false;
changeState(currentInput, nowMs);
// Calculated here because it's used in multiple places.
float incomingBitrateKbps = incomingBitrateBps / 1000.0F;
// Calculate the max bit rate std dev given the normalized variance and
// the current incoming bit rate.
float stdMaxBitRate
= (float) Math.sqrt(varMaxBitrateKbps * avgMaxBitrateKbps);
switch (rateControlState)
{
case kRcHold:
break;
case kRcIncrease:
{
if (avgMaxBitrateKbps >= 0F
&& incomingBitrateKbps
> avgMaxBitrateKbps + 3F * stdMaxBitRate)
{
changeRegion(RateControlRegion.kRcMaxUnknown, nowMs);
avgMaxBitrateKbps = -1F;
}
if (rateControlRegion == RateControlRegion.kRcNearMax)
{
// Approximate the over-use estimator delay to 100 ms.
long responseTime = rtt + 100;
long additiveIncreaseBps
= additiveRateIncrease(
nowMs,
timeLastBitrateChange,
responseTime);
currentBitrateBps += additiveIncreaseBps;
}
else // kRcMaxUnknown || kRcAboveMax
{
long multiplicativeIncreaseBps
= multiplicativeRateIncrease(
nowMs,
timeLastBitrateChange,
currentBitrateBps);
currentBitrateBps += multiplicativeIncreaseBps;
}
timeLastBitrateChange = nowMs;
break;
}
case kRcDecrease:
{
bitrateIsInitialized = true;
if (incomingBitrateBps < minConfiguredBitrateBps)
{
currentBitrateBps = minConfiguredBitrateBps;
}
else
{
// Set bit rate to something slightly lower than max to get rid
// of any self-induced delay.
currentBitrateBps = (long) (beta * incomingBitrateBps + 0.5);
if (currentBitrateBps > this.currentBitrateBps)
{
// Avoid increasing the rate when over-using.
if (rateControlRegion != RateControlRegion.kRcMaxUnknown)
{
currentBitrateBps
= (long) (beta * avgMaxBitrateKbps * 1000F + 0.5F);
}
currentBitrateBps
= Math.min(currentBitrateBps, this.currentBitrateBps);
}
changeRegion(RateControlRegion.kRcNearMax, nowMs);
if (incomingBitrateKbps
< avgMaxBitrateKbps - 3F * stdMaxBitRate)
{
avgMaxBitrateKbps = -1F;
}
updateMaxBitRateEstimate(incomingBitrateKbps);
}
// Stay on hold until the pipes are cleared.
changeState(RateControlState.kRcHold, nowMs);
timeLastBitrateChange = nowMs;
break;
}
default:
throw new IllegalStateException("rateControlState");
}
if ((incomingBitrateBps > 100000L || currentBitrateBps > 150000L)
&& currentBitrateBps > 1.5 * incomingBitrateBps)
{
// Allow changing the bit rate if we are operating at very low rates
// Don't change the bit rate if the send side is too far off
currentBitrateBps = this.currentBitrateBps;
timeLastBitrateChange = nowMs;
}
return currentBitrateBps;
}
private void changeRegion(RateControlRegion region, long nowMs)
{
if (rateControlRegion == region)
{
return;
}
rateControlRegion = region;
if (logger.isTraceEnabled())
{
logger.trace(diagnosticContext
.makeTimeSeriesPoint("aimd_region", nowMs)
.addField("aimd_id", hashCode())
.addField("region", region));
}
}
private void changeState(RateControlInput input, long nowMs)
{
switch (currentInput.bwState)
{
case kBwNormal:
if (rateControlState == RateControlState.kRcHold)
{
timeLastBitrateChange = nowMs;
changeState(RateControlState.kRcIncrease, nowMs);
}
break;
case kBwOverusing:
if (rateControlState != RateControlState.kRcDecrease)
{
changeState(RateControlState.kRcDecrease, nowMs);
}
break;
case kBwUnderusing:
changeState(RateControlState.kRcHold, nowMs);
break;
default:
throw new IllegalStateException("currentInput.bwState");
}
}
private void changeState(RateControlState newState, long nowMs)
{
if (rateControlState == newState)
{
return;
}
rateControlState = newState;
if (logger.isTraceEnabled())
{
logger.trace(diagnosticContext
.makeTimeSeriesPoint("aimd_state", nowMs)
.addField("aimd_id", hashCode())
.addField("state", rateControlState));
}
}
public long getFeedBackInterval()
{
// Estimate how often we can send RTCP if we allocate up to 5% of
// bandwidth to feedback.
long interval
= (long)
(kRtcpSize * 8.0 * 1000.0 / (0.05 * currentBitrateBps) + 0.5);
return
Math.min(
Math.max(interval, kMinFeedbackIntervalMs),
kMaxFeedbackIntervalMs);
}
public long getLatestEstimate()
{
return currentBitrateBps;
}
/**
* Returns <tt>true</tt> if the bitrate estimate hasn't been changed for
* more than an RTT, or if the <tt>incomingBitrate</tt> is more than 5%
* above the current estimate. Should be used to decide if we should reduce
* the rate further when over-using.
*
* @param timeNow
* @param incomingBitrateBps
* @return
*/
public boolean isTimeToReduceFurther(long timeNow, long incomingBitrateBps)
{
long bitrateReductionInterval = Math.max(Math.min(rtt, 200L), 10L);
if (timeNow - timeLastBitrateChange >= bitrateReductionInterval)
return true;
if (isValidEstimate())
{
long threshold
= (long) (kWithinIncomingBitrateHysteresis * incomingBitrateBps);
long bitrateDifference = getLatestEstimate() - incomingBitrateBps;
return bitrateDifference > threshold;
}
return false;
}
/**
* Returns <tt>true</tt> if there is a valid estimate of the incoming
* bitrate, <tt>false</tt> otherwise.
*
* @return
*/
public boolean isValidEstimate()
{
return bitrateIsInitialized;
}
private long multiplicativeRateIncrease(
long nowMs,
long lastMs,
long currentBitrateBps)
{
double alpha = 1.08;
if (lastMs > -1)
{
long timeSinceLastUpdateMs = Math.min(nowMs - lastMs, 1000);
alpha = Math.pow(alpha, timeSinceLastUpdateMs / 1000.0);
}
return (long) Math.max(currentBitrateBps * (alpha - 1.0), 1000.0);
}
public void reset()
{
reset(RemoteBitrateEstimatorAbsSendTime.kDefaultMinBitrateBps);
}
private void reset(long minBitrateBps)
{
minConfiguredBitrateBps = minBitrateBps;
currentBitrateBps = /* maxConfiguredBitrateBps */ 30000000L;
avgMaxBitrateKbps = -1F;
varMaxBitrateKbps = 0.4F;
rateControlState = RateControlState.kRcHold;
rateControlRegion = RateControlRegion.kRcMaxUnknown;
timeLastBitrateChange = -1L;
currentInput.bwState = BandwidthUsage.kBwNormal;
currentInput.incomingBitRate = 0L;
currentInput.noiseVar = 1D;
updated = false;
timeFirstIncomingEstimate = -1L;
bitrateIsInitialized = false;
beta = 0.85F;
rtt = kDefaultRttMs;
timeOfLastLog = -1L;
inExperiment = false;
}
public void setEstimate(long bitrateBps, long nowMs)
{
updated = true;
bitrateIsInitialized = true;
currentBitrateBps = changeBitrate(bitrateBps, bitrateBps, nowMs);
}
public void setMinBitrate(long minBitrateBps)
{
minConfiguredBitrateBps = minBitrateBps;
currentBitrateBps = Math.max(minBitrateBps, currentBitrateBps);
}
public void setRtt(long nowMs, long rtt)
{
if (logger.isTraceEnabled())
{
logger.trace(diagnosticContext
.makeTimeSeriesPoint("aimd_rtt", nowMs)
.addField("aimd_id", hashCode())
.addField("rtt", rtt));
}
this.rtt = rtt;
}
public void update(RateControlInput input, long nowMs)
{
if (input == null)
throw new NullPointerException("input");
// Set the initial bit rate value to what we're receiving the first half
// second.
if (!bitrateIsInitialized)
{
if (timeFirstIncomingEstimate < 0L)
{
if (input.incomingBitRate > 0L)
timeFirstIncomingEstimate = nowMs;
}
else
{
long timeSinceFirstIncomingEstimate = nowMs - timeFirstIncomingEstimate;
if (timeSinceFirstIncomingEstimate > kFirstIncomingEstimateExpirationMs)
{
if (input.incomingBitRate > 0L)
{
timeFirstIncomingEstimate = nowMs;
}
else
{
timeFirstIncomingEstimate = -1L;
}
}
else if (timeSinceFirstIncomingEstimate > kInitializationTimeMs
&& input.incomingBitRate > 0L)
{
currentBitrateBps = input.incomingBitRate;
bitrateIsInitialized = true;
}
}
}
if (updated && currentInput.bwState == BandwidthUsage.kBwOverusing)
{
// Only update delay factor and incoming bit rate. We always want to
// react on an over-use.
currentInput.noiseVar = input.noiseVar;
currentInput.incomingBitRate = input.incomingBitRate;
}
else
{
updated = true;
currentInput.copy(input);
}
}
public long updateBandwidthEstimate(long nowMs)
{
currentBitrateBps
= changeBitrate(
currentBitrateBps,
currentInput.incomingBitRate,
nowMs);
if (logger.isTraceEnabled() && isValidEstimate())
{
logger.trace(diagnosticContext
.makeTimeSeriesPoint("aimd_estimate", nowMs)
.addField("aimd_id", hashCode())
.addField("estimate_bps", currentBitrateBps)
.addField("incoming_bps", currentInput.incomingBitRate));
}
if (nowMs - timeOfLastLog > kLogIntervalMs)
timeOfLastLog = nowMs;
return currentBitrateBps;
}
private void updateMaxBitRateEstimate(float incomingBitrateKbps)
{
float alpha = 0.05F;
if (avgMaxBitrateKbps == -1F)
{
avgMaxBitrateKbps = incomingBitrateKbps;
}
else
{
avgMaxBitrateKbps
= (1 - alpha) * avgMaxBitrateKbps + alpha * incomingBitrateKbps;
}
// Estimate the max bit rate variance and normalize the variance with
// the average max bit rate.
float norm = Math.max(avgMaxBitrateKbps, 1F);
varMaxBitrateKbps
= (1 - alpha) * varMaxBitrateKbps
+ alpha
* (avgMaxBitrateKbps - incomingBitrateKbps)
* (avgMaxBitrateKbps - incomingBitrateKbps)
/ norm;
// 0.4 ~= 14 kbit/s at 500 kbit/s
if (varMaxBitrateKbps < 0.4F)
varMaxBitrateKbps = 0.4F;
// 2.5f ~= 35 kbit/s at 500 kbit/s
if (varMaxBitrateKbps > 2.5f)
varMaxBitrateKbps = 2.5f;
}
}

35
jitsi-media-transform/src/main/java/org/jitsi_modified/impl/neomedia/rtp/remotebitrateestimator/BandwidthUsage.java Normal file
View File

@@ -0,0 +1,35 @@
/*
* Copyright @ 2015 Atlassian Pty Ltd
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi_modified.impl.neomedia.rtp.remotebitrateestimator;
/**
* webrtc/modules/remote_bitrate_estimator/include/bwe_defines.h
*
* @author Lyubomir Marinov
*/
public enum BandwidthUsage
{
kBwNormal(0), kBwUnderusing(-1), kBwOverusing(1);
private int value;
BandwidthUsage(int value){
this.value = value;
}
public int getValue() {
return value;
}
}

374
jitsi-media-transform/src/main/java/org/jitsi_modified/impl/neomedia/rtp/remotebitrateestimator/InterArrival.java Normal file
View File

@@ -0,0 +1,374 @@
/*
* Copyright @ 2015 Atlassian Pty Ltd
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi_modified.impl.neomedia.rtp.remotebitrateestimator;
import org.jitsi.utils.TimestampUtils;
import org.jetbrains.annotations.*;
import org.jitsi.utils.logging.*;
import org.jitsi.utils.logging2.*;
import org.jitsi.utils.logging2.Logger;
/**
* Helper class to compute the inter-arrival time delta and the size delta
* between two timestamp groups. A timestamp is a 32 bit unsigned number with a
* client defined rate.
*
* webrtc/modules/remote_bitrate_estimator/inter_arrival.cc
* webrtc/modules/remote_bitrate_estimator/inter_arrival.h
*
* @author Lyubomir Marinov
* @author Julian Chukwu
* @author George Politis
*/
class InterArrival
{
private static final int kBurstDeltaThresholdMs = 5;
private final Logger logger;
private final TimeSeriesLogger timeSeriesLogger;
/**
* webrtc/modules/include/module_common_types.h
*
* @param timestamp1
* @param timestamp2
* @return
*/
private static long latestTimestamp(long timestamp1, long timestamp2)
{
return TimestampUtils.latestTimestamp(timestamp1, timestamp2);
}
private boolean burstGrouping;
private TimestampGroup currentTimestampGroup = new TimestampGroup();
private final long kTimestampGroupLengthTicks;
private final DiagnosticContext diagnosticContext;
private TimestampGroup prevTimestampGroup = new TimestampGroup();
private double timestampToMsCoeff;
private int numConsecutiveReorderedPackets;
// After this many packet groups received out of order InterArrival will
// reset, assuming that clocks have made a jump.
private static final int kReorderedResetThreshold = 3;
private static final int kArrivalTimeOffsetThresholdMs = 3000;
/**
* A timestamp group is defined as all packets with a timestamp which are at
* most {@code timestampGroupLengthTicks} older than the first timestamp in
* that group.
*
* @param timestampGroupLengthTicks
* @param timestampToMsCoeff
* @param enableBurstGrouping
*/
public InterArrival(
long timestampGroupLengthTicks,
double timestampToMsCoeff,
boolean enableBurstGrouping,
@NotNull DiagnosticContext diagnosticContext,
@NotNull Logger parentLogger)
{
kTimestampGroupLengthTicks = timestampGroupLengthTicks;
this.timestampToMsCoeff = timestampToMsCoeff;
burstGrouping = enableBurstGrouping;
numConsecutiveReorderedPackets = 0;
this.diagnosticContext = diagnosticContext;
this.logger = parentLogger.createChildLogger(getClass().getName());
this.timeSeriesLogger = TimeSeriesLogger.getTimeSeriesLogger(getClass());
}
private boolean belongsToBurst(long arrivalTimeMs, long timestamp)
{
if (!burstGrouping)
return false;
if (currentTimestampGroup.completeTimeMs < 0)
{
throw new IllegalStateException(
"currentTimestampGroup.completeTimeMs");
}
long arrivalTimeDeltaMs
= arrivalTimeMs - currentTimestampGroup.completeTimeMs;
long timestampDiff = TimestampUtils
.subtractAsUnsignedInt32(timestamp, currentTimestampGroup.timestamp);
long tsDeltaMs = (long) (timestampToMsCoeff * timestampDiff + 0.5);
if (tsDeltaMs == 0)
return true;
long propagationDeltaMs = arrivalTimeDeltaMs - tsDeltaMs;
return
propagationDeltaMs < 0
&& arrivalTimeDeltaMs <= kBurstDeltaThresholdMs;
}
/**
* Returns {@code true} if a delta was computed, or {@code false} if the
* current group is still incomplete or if only one group has been
* completed.
*
* @param timestamp is the timestamp.
* @param arrivalTimeMs is the local time at which the packet arrived.
* @param packetSize is the size of the packet.
* @param deltas {@code timestampDelta} is the computed timestamp delta,
* {@code arrivalTimeDeltaMs} is the computed arrival-time delta,
* {@code packetSizeDelta} is the computed size delta.
* @return
*
* @Note: We have two {@code computeDeltas}.
* One with a valid {@code systemTimeMs} according to webrtc
* implementation as of June 12,2017 and a previous one
* with a default systemTimeMs (-1L). the later may be removed or
* deprecated.
*/
public boolean computeDeltas(
long timestamp,
long arrivalTimeMs,
int packetSize,
long[] deltas){
return computeDeltas(timestamp, arrivalTimeMs, packetSize, deltas, -1L);
}
public boolean computeDeltas(
long timestamp,
long arrivalTimeMs,
int packetSize,
long[] deltas,
long systemTimeMs)
{
if (deltas == null)
throw new NullPointerException("deltas");
if (deltas.length != 3)
throw new IllegalArgumentException("deltas.length");
boolean calculatedDeltas = false;
if (currentTimestampGroup.isFirstPacket())
{
// We don't have enough data to update the filter, so we store it
// until we have two frames of data to process.
currentTimestampGroup.timestamp = timestamp;
currentTimestampGroup.firstTimestamp = timestamp;
}
else if (!isPacketInOrder(timestamp))
{
return false;
}
else if (isNewTimestampGroup(arrivalTimeMs, timestamp))
{
// First packet of a later frame, the previous frame sample is
// ready.
if (prevTimestampGroup.completeTimeMs >= 0)
{
/* long timestampDelta */ deltas[0]
= TimestampUtils.subtractAsUnsignedInt32(
currentTimestampGroup.timestamp,
prevTimestampGroup.timestamp);
long arrivalTimeDeltaMs
= deltas[1]
= currentTimestampGroup.completeTimeMs
- prevTimestampGroup.completeTimeMs;
// Check system time differences to see if we have an unproportional jump
// in arrival time. In that case reset the inter-arrival computations.
long systemTimeDeltaMs =
currentTimestampGroup.lastSystemTimeMs -
prevTimestampGroup.lastSystemTimeMs;
if (prevTimestampGroup.lastSystemTimeMs != -1L &&
currentTimestampGroup.lastSystemTimeMs != -1L &&
arrivalTimeDeltaMs - systemTimeDeltaMs >=
kArrivalTimeOffsetThresholdMs)
{
logger.warn( "The arrival time clock offset has changed (diff = "
+ (arrivalTimeDeltaMs - systemTimeDeltaMs)
+ " ms), resetting.");
Reset();
return false;
}
if (arrivalTimeDeltaMs < 0)
{
++numConsecutiveReorderedPackets;
if (numConsecutiveReorderedPackets >= kReorderedResetThreshold)
{
// The group of packets has been reordered since receiving
// its local arrival timestamp.
logger.warn(
"Packets are being reordered on the path from the "
+ "socket to the bandwidth estimator. Ignoring "
+ "this packet for bandwidth estimation.");
Reset();
}
return false;
}
else
{
numConsecutiveReorderedPackets = 0;
}
/* int packetSizeDelta */ deltas[2]
= (int)
(currentTimestampGroup.size - prevTimestampGroup.size);
if (timeSeriesLogger.isTraceEnabled())
{
timeSeriesLogger.trace(diagnosticContext
.makeTimeSeriesPoint("computed_deltas")
.addField("inter_arrival", hashCode())
.addField("arrival_time_ms", arrivalTimeMs)
.addField("timestamp_delta", deltas[0])
.addField("arrival_time_ms_delta", deltas[1])
.addField("payload_size_delta", deltas[2]));
}
calculatedDeltas = true;
}
prevTimestampGroup.copy(currentTimestampGroup);
// The new timestamp is now the current frame.
currentTimestampGroup.firstTimestamp = timestamp;
currentTimestampGroup.timestamp = timestamp;
currentTimestampGroup.size = 0;
}
else
{
currentTimestampGroup.timestamp = latestTimestamp(
currentTimestampGroup.timestamp, timestamp);
}
// Accumulate the frame size.
currentTimestampGroup.size += packetSize;
currentTimestampGroup.completeTimeMs = arrivalTimeMs;
currentTimestampGroup.lastSystemTimeMs = systemTimeMs;
return calculatedDeltas;
}
/**
* Returns {@code true} if the last packet was the end of the current batch
* and the packet with {@code timestamp} is the first of a new batch.
*
* @param arrivalTimeMs
* @param timestamp
* @return
*/
private boolean isNewTimestampGroup(long arrivalTimeMs, long timestamp)
{
if (currentTimestampGroup.isFirstPacket())
{
return false;
}
else if (belongsToBurst(arrivalTimeMs, timestamp))
{
return false;
}
else
{
long timestampDiff
= TimestampUtils.subtractAsUnsignedInt32(
timestamp, currentTimestampGroup.firstTimestamp);
return timestampDiff > kTimestampGroupLengthTicks;
}
}
/**
* Returns {@code true} if the packet with timestamp {@code timestamp}
* arrived in order.
*
* @param timestamp
* @return
*/
private boolean isPacketInOrder(long timestamp)
{
if (currentTimestampGroup.isFirstPacket())
{
return true;
}
else
{
// Assume that a diff which is bigger than half the timestamp
// interval (32 bits) must be due to reordering. This code is almost
// identical to that in isNewerTimestamp() in module_common_types.h.
long timestampDiff
= TimestampUtils.subtractAsUnsignedInt32(
timestamp, currentTimestampGroup.firstTimestamp);
long tsDeltaMs = (long) (timestampToMsCoeff * timestampDiff + 0.5);
boolean inOrder = timestampDiff < 0x80000000L;
if (!inOrder && timeSeriesLogger.isTraceEnabled())
{
timeSeriesLogger.trace(diagnosticContext
.makeTimeSeriesPoint("reordered_packet")
.addField("inter_arrival", hashCode())
.addField("ts_delta_ms", tsDeltaMs));
}
return inOrder;
}
}
private static class TimestampGroup
{
public long completeTimeMs = -1L;
public long size = 0L;
public long firstTimestamp = 0L;
public long timestamp = 0L;
public long lastSystemTimeMs = -1L;
/**
* Assigns the values of the fields of <tt>source</tt> to the respective
* fields of this {@code TimestampGroup}.
*
* @param source the {@code TimestampGroup} the values of the fields of
* which are to be assigned to the respective fields of this
* {@code TimestampGroup}
*/
public void copy(TimestampGroup source)
{
completeTimeMs = source.completeTimeMs;
firstTimestamp = source.firstTimestamp;
size = source.size;
timestamp = source.timestamp;
}
public boolean isFirstPacket()
{
return completeTimeMs == -1L;
}
}
public void Reset() {
numConsecutiveReorderedPackets = 0;
currentTimestampGroup = new TimestampGroup();
prevTimestampGroup = new TimestampGroup();
}
}

44
jitsi-media-transform/src/main/java/org/jitsi_modified/impl/neomedia/rtp/remotebitrateestimator/OverUseDetectorOptions.java Normal file
View File

@@ -0,0 +1,44 @@
/*
* Copyright @ 2015 Atlassian Pty Ltd
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi_modified.impl.neomedia.rtp.remotebitrateestimator;
import org.jitsi_modified.impl.neomedia.rtp.remotebitrateestimator.config.OveruseDetectorConfig;
/**
* Bandwidth over-use detector options. These are used to drive experimentation
* with bandwidth estimation parameters.
*
* webrtc/common_types.h
*
* @author Lyubomir Marinov
*/
class OverUseDetectorOptions
{
public double initialAvgNoise = 0.0D;
public final double[][] initialE
= new double[][] { { 100, 0 }, { 0, 1e-1 } };
public double initialOffset = 0.0D;
public final double[] initialProcessNoise = new double[] { 1e-10, 1e-2 };
public double initialSlope = 8.0D / 512.0D;
public double initialThreshold = OveruseDetectorConfig.Companion.getInitialThreshold();
public double initialVarNoise = 50.0D;
}

203
jitsi-media-transform/src/main/java/org/jitsi_modified/impl/neomedia/rtp/remotebitrateestimator/OveruseDetector.java Normal file
View File

@@ -0,0 +1,203 @@
/*
* Copyright @ 2015 Atlassian Pty Ltd
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi_modified.impl.neomedia.rtp.remotebitrateestimator;
import edu.umd.cs.findbugs.annotations.*;
import org.jetbrains.annotations.*;
import org.jitsi.utils.logging.*;
import org.jitsi_modified.impl.neomedia.rtp.remotebitrateestimator.config.OveruseDetectorConfig;
import java.util.*;
/**
* webrtc/modules/remote_bitrate_estimator/overuse_detector.cc
* webrtc/modules/remote_bitrate_estimator/overuse_detector.h
*
* @author Lyubomir Marinov
*/
@SuppressFBWarnings(
value = {"FE_FLOATING_POINT_EQUALITY", "DB_DUPLICATE_BRANCHES"},
justification = "We only compare timeOverUsing with the special value -1")
class OveruseDetector
{
private final TimeSeriesLogger timeSeriesLogger;
private static final double kMaxAdaptOffsetMs = 15.0;
private BandwidthUsage hypothesis = BandwidthUsage.kBwNormal;
private static final double kDown = 0D;
private static final double kUp = 0D;
private long lastUpdateMs = -1L;
private int overuseCounter;
private static final double overusingTimeThreshold = OveruseDetectorConfig.Companion.getOverusingTimeThreshold();
private double prevOffset;
private double threshold;
private double timeOverUsing = -1D;
private final DiagnosticContext diagnosticContext;
public OveruseDetector(
OverUseDetectorOptions options,
@NotNull DiagnosticContext diagnosticContext)
{
Objects.requireNonNull(options, "options");
threshold = options.initialThreshold;
timeSeriesLogger = TimeSeriesLogger.getTimeSeriesLogger(getClass());
this.diagnosticContext = diagnosticContext;
}
/**
* Update the detection state based on the estimated inter-arrival time
* delta offset. {@code timestampDelta} is the delta between the last
* timestamp which the estimated offset is based on and the last timestamp
* on which the last offset was based on, representing the time between
* detector updates. {@code numOfDeltas} is the number of deltas the offset
* estimate is based on. Returns the state after the detection update.
*
* @param offset
* @param tsDelta
* @param numOfDeltas
* @param nowMs
* @return
*/
public BandwidthUsage detect(
double offset,
double tsDelta,
int numOfDeltas,
long nowMs)
{
if (numOfDeltas < 2)
return BandwidthUsage.kBwNormal;
double prev_offset = this.prevOffset;
this.prevOffset = offset;
double T = Math.min(numOfDeltas, 60) * offset;
boolean newHypothesis = false;
if (T > threshold)
{
if (timeOverUsing == -1)
{
// Initialize the timer. Assume that we've been
// over-using half of the time since the previous
// sample.
timeOverUsing = tsDelta / 2;
}
else
{
// Increment timer
timeOverUsing += tsDelta;
}
overuseCounter++;
if (timeOverUsing > overusingTimeThreshold && overuseCounter > 1)
{
if (offset >= prev_offset)
{
timeOverUsing = 0;
overuseCounter = 0;
hypothesis = BandwidthUsage.kBwOverusing;
newHypothesis = true;
}
}
}
else if (T < -threshold)
{
timeOverUsing = -1;
overuseCounter = 0;
hypothesis = BandwidthUsage.kBwUnderusing;
newHypothesis = true;
}
else
{
timeOverUsing = -1;
overuseCounter = 0;
hypothesis = BandwidthUsage.kBwNormal;
newHypothesis = true;
}
if (newHypothesis && timeSeriesLogger.isTraceEnabled())
{
timeSeriesLogger.trace(diagnosticContext
.makeTimeSeriesPoint("utilization_hypothesis", nowMs)
.addField("detector", hashCode())
.addField("offset", offset)
.addField("prev_offset", prev_offset)
.addField("T", T)
.addField("threshold", threshold)
.addField("hypothesis", hypothesis.getValue()));
}
updateThreshold(T, nowMs);
return hypothesis;
}
/**
* Returns the current detector state.
*
* @return
*/
public BandwidthUsage getState()
{
return hypothesis;
}
private void updateThreshold(double modifiedOffset, long nowMs)
{
if (lastUpdateMs == -1)
lastUpdateMs = nowMs;
if (Math.abs(modifiedOffset) > threshold + kMaxAdaptOffsetMs)
{
// Avoid adapting the threshold to big latency spikes, caused e.g.,
// by a sudden capacity drop.
lastUpdateMs = nowMs;
return;
}
double k = Math.abs(modifiedOffset) < threshold ? kDown : kUp;
threshold
+= k
* (Math.abs(modifiedOffset) - threshold)
* (nowMs - lastUpdateMs);
final double kMinThreshold = 6;
final double kMaxThreshold = 600;
threshold = Math.min(Math.max(threshold, kMinThreshold), kMaxThreshold);
lastUpdateMs = nowMs;
}
public double getThreshold()
{
return threshold;
}
}

330
jitsi-media-transform/src/main/java/org/jitsi_modified/impl/neomedia/rtp/remotebitrateestimator/OveruseEstimator.java Normal file
View File

@@ -0,0 +1,330 @@
/*
* Copyright @ 2015 Atlassian Pty Ltd
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi_modified.impl.neomedia.rtp.remotebitrateestimator;
import org.jetbrains.annotations.*;
import org.jitsi.utils.logging.*;
import org.jitsi.utils.logging2.*;
import org.jitsi.utils.logging2.Logger;
import java.util.*;
/**
* webrtc/modules/remote_bitrate_estimator/overuse_estimator.cc
* webrtc/modules/remote_bitrate_estimator/overuse_estimator.h
*
* @author Lyubomir Marinov
*/
class OveruseEstimator
{
/**
* The <tt>Logger</tt> used by
* <tt>OveruseEstimator</tt> instances for
* logging output.
*/
private final Logger logger;
/**
* The <tt>TimeSeriesLogger</tt> used for time series.
*/
private final TimeSeriesLogger timeSeriesLogger;
private static final int kDeltaCounterMax = 1000;
private static final int kMinFramePeriodHistoryLength = 60;
/**
* Creates and returns a deep copy of a <tt>double</tt> two-dimensional
* matrix.
*
* @param matrix the <tt>double</tt> two-dimensional matrix to create and
* return a deep copy of
* @return a deep copy of <tt>matrix</tt>
*/
private static double[][] clone(double[][] matrix)
{
int length = matrix.length;
double[][] clone;
clone = new double[length][];
for (int i = 0; i < length; i++)
clone[i] = matrix[i].clone();
return clone;
}
private double avgNoise;
private final double[][] E;
private DiagnosticContext diagnosticContext;
/**
* Reduces the effects of allocations and garbage collection of the method
* {@code update}.
*/
private final double[] Eh = new double[2];
/**
* Reduces the effects of allocations and garbage collection of the method
* {@code update}.
*/
private final double[] h = new double[2];
/**
* Reduces the effects of allocations and garbage collection of the method
* {@code update}.
*/
private final double[][] IKh = new double[][] { { 0, 0 }, { 0, 0 } };
/**
* Reduces the effects of allocations and garbage collection of the method
* {@code update}.
*/
private final double[] K = new double[2];
private int numOfDeltas;
private double offset;
private double prevOffset;
private final double[] processNoise;
private double slope;
/**
* Store the tsDelta history into a {@code double[]} used as a circular buffer
* Original c++ code uses std::list<double> but in Java this translate
* to {@code List<Double>} which causes a lot of autoboxing
*/
private final double[] tsDeltaHist = new double[kMinFramePeriodHistoryLength];
/**
* Index to insert next value into {@link #tsDeltaHist}
*/
private int tsDeltaHistInsIdx;
private double varNoise;
public OveruseEstimator(
OverUseDetectorOptions options,
@NotNull DiagnosticContext diagnosticContext,
@NotNull Logger parentLogger)
{
logger = parentLogger.createChildLogger(getClass().getName());
timeSeriesLogger = TimeSeriesLogger.getTimeSeriesLogger(getClass());
this.diagnosticContext = diagnosticContext;
slope = options.initialSlope;
offset = options.initialOffset;
prevOffset = offset;
avgNoise = options.initialAvgNoise;
varNoise = options.initialVarNoise;
E = clone(options.initialE);
processNoise = options.initialProcessNoise.clone();
/**
* Initialize {@link tsDeltaHist} with {@code Double.MAX_VALUE}
* to simplify {@link updateMinFramePeriod}
*/
Arrays.fill(tsDeltaHist, Double.MAX_VALUE);
}
/**
* Returns the number of deltas which the current over-use estimator state
* is based on.
*
* @return
*/
public int getNumOfDeltas()
{
return numOfDeltas;
}
/**
* Returns the estimated inter-arrival time delta offset in ms.
*
* @return
*/
public double getOffset()
{
return offset;
}
/**
* Returns the estimated noise/jitter variance in ms^2.
*
* @return
*/
public double getVarNoise()
{
return varNoise;
}
/**
* Update the estimator with a new sample. The deltas should represent
* deltas between timestamp groups as defined by the InterArrival class.
* {@code currentHypothesis} should be the hypothesis of the over-use
* detector at this time.
*
* @param tDelta
* @param tsDelta
* @param sizeDelta
* @param currentHypothesis
*/
public void update(
long tDelta,
double tsDelta,
int sizeDelta,
BandwidthUsage currentHypothesis, long systemTimeMs)
{
double minFramePeriod = updateMinFramePeriod(tsDelta);
double tTsDelta = tDelta - tsDelta;
++numOfDeltas;
if (numOfDeltas > kDeltaCounterMax)
numOfDeltas = kDeltaCounterMax;
// Update the Kalman filter
E[0][0] += processNoise[0];
E[1][1] += processNoise[1];
if ((currentHypothesis == BandwidthUsage.kBwOverusing
&& offset < prevOffset)
|| (currentHypothesis == BandwidthUsage.kBwUnderusing
&& offset > prevOffset))
{
E[1][1] += 10D * processNoise[1];
}
double[] h = this.h;
double[] Eh = this.Eh;
h[0] = sizeDelta;
h[1] = 1D;
Eh[0] = E[0][0] * h[0] + E[0][1] * h[1];
Eh[1] = E[1][0] * h[0] + E[1][1] * h[1];
double residual = tTsDelta - slope * h[0] - offset;
boolean inStableState = (currentHypothesis == BandwidthUsage.kBwNormal);
// We try to filter out very late frames. For instance periodic key
// frames doesn't fit the Gaussian model well.
double maxResidual = 3D * Math.sqrt(varNoise);
double residualForUpdateNoiseEstimate
= (Math.abs(residual) < maxResidual)
? residual
: (residual < 0 ? -maxResidual : maxResidual);
updateNoiseEstimate(
residualForUpdateNoiseEstimate,
minFramePeriod,
inStableState);
double denom = varNoise + h[0]*Eh[0] + h[1]*Eh[1];
double[] K = this.K;
double[][] IKh = this.IKh;
K[0] = Eh[0] / denom;
K[1] = Eh[1] / denom;
IKh[0][0] = 1D - K[0]*h[0];
IKh[0][1] = -K[0]*h[1];
IKh[1][0] = -K[1]*h[0];
IKh[1][1] = 1D - K[1]*h[1];
double e00 = E[0][0];
double e01 = E[0][1];
// Update state.
E[0][0] = e00 * IKh[0][0] + E[1][0] * IKh[0][1];
E[0][1] = e01 * IKh[0][0] + E[1][1] * IKh[0][1];
E[1][0] = e00 * IKh[1][0] + E[1][0] * IKh[1][1];
E[1][1] = e01 * IKh[1][0] + E[1][1] * IKh[1][1];
// Covariance matrix, must be positive semi-definite.
boolean positiveSemiDefinite
= E[0][0] + E[1][1] >= 0
&& E[0][0] * E[1][1] - E[0][1] * E[1][0] >= 0
&& E[0][0] >= 0;
if (!positiveSemiDefinite)
throw new IllegalStateException("positiveSemiDefinite");
slope = slope + K[0] * residual;
prevOffset = offset;
offset = offset + K[1] * residual;
if (timeSeriesLogger.isTraceEnabled())
{
timeSeriesLogger.trace(diagnosticContext
.makeTimeSeriesPoint("delay_variation_estimation", systemTimeMs)
.addField("estimator", hashCode())
.addField("time_delta", tDelta)
.addField("ts_delta", tsDelta)
.addField("tts_delta", tTsDelta)
.addField("offset", offset)
.addField("hypothesis", currentHypothesis.getValue()));
}
}
private double updateMinFramePeriod(double tsDelta)
{
/**
* Change from C++ version:
* We use {@link tsDeltaHist} as a circular buffer initialized
* with {@code Double.MAX_VALUE}, so we insert new {@link tsDelta}
* value at {@link tsDeltaHistInsIdx} and we search for the
* minimum value in {@link tsDeltaHist}
*/
tsDeltaHist[tsDeltaHistInsIdx] = tsDelta;
tsDeltaHistInsIdx = (tsDeltaHistInsIdx + 1) % tsDeltaHist.length;
double min = tsDelta;
for (double d : tsDeltaHist)
{
if (d < min)
min = d;
}
return min;
}
private void updateNoiseEstimate(
double residual,
double tsDelta,
boolean stableState)
{
if (!stableState)
return;
// Faster filter during startup to faster adapt to the jitter level of
// the network alpha is tuned for 30 frames per second, but is scaled
// according to tsDelta.
double alpha = 0.01D;
if (numOfDeltas > 10 * 30)
alpha = 0.002D;
// Only update the noise estimate if we're not over-using beta is a
// function of alpha and the time delta since the previous update.
double beta = Math.pow(1 - alpha, tsDelta * 30D / 1000D);
avgNoise = beta * avgNoise + (1 - beta) * residual;
varNoise
= beta * varNoise
+ (1 - beta) * (avgNoise - residual) * (avgNoise - residual);
if (varNoise < 1)
varNoise = 1;
}
}

58
jitsi-media-transform/src/main/java/org/jitsi_modified/impl/neomedia/rtp/remotebitrateestimator/RateControlInput.java Normal file
View File

@@ -0,0 +1,58 @@
/*
* Copyright @ 2015 Atlassian Pty Ltd
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi_modified.impl.neomedia.rtp.remotebitrateestimator;
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
/**
* webrtc/modules/remote_bitrate_estimator/include/bwe_defines.h
*
* @author Lyubomir Marinov
*/
@SuppressFBWarnings("PA_PUBLIC_PRIMITIVE_ATTRIBUTE")
class RateControlInput
{
public BandwidthUsage bwState;
public long incomingBitRate;
public double noiseVar;
public RateControlInput(
BandwidthUsage bwState,
long incomingBitRate,
double noiseVar)
{
this.bwState = bwState;
this.incomingBitRate = incomingBitRate;
this.noiseVar = noiseVar;
}
/**
* Assigns the values of the fields of <tt>source</tt> to the respective
* fields of this <tt>RateControlInput</tt>.
*
* @param source the <tt>RateControlInput</tt> the values of the fields of
* which are to be assigned to the respective fields of this
* <tt>RateControlInput</tt>
*/
public void copy(RateControlInput source)
{
bwState = source.bwState;
incomingBitRate = source.incomingBitRate;
noiseVar = source.noiseVar;
}
}

28
jitsi-media-transform/src/main/java/org/jitsi_modified/impl/neomedia/rtp/remotebitrateestimator/RateControlRegion.java Normal file
View File

@@ -0,0 +1,28 @@
/*
* Copyright @ 2015 Atlassian Pty Ltd
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi_modified.impl.neomedia.rtp.remotebitrateestimator;
/**
* webrtc/modules/remote_bitrate_estimator/include/bwe_defines.h
*
* @author Lyubomir Marinov
*/
enum RateControlRegion
{
kRcNearMax,
kRcAboveMax,
kRcMaxUnknown
}

28
jitsi-media-transform/src/main/java/org/jitsi_modified/impl/neomedia/rtp/remotebitrateestimator/RateControlState.java Normal file
View File

@@ -0,0 +1,28 @@
/*
* Copyright @ 2015 Atlassian Pty Ltd
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi_modified.impl.neomedia.rtp.remotebitrateestimator;
/**
* webrtc/modules/remote_bitrate_estimator/include/bwe_defines.h
*
* @author Lyubomir Marinov
*/
enum RateControlState
{
kRcHold,
kRcIncrease,
kRcDecrease
}

463
jitsi-media-transform/src/main/java/org/jitsi_modified/impl/neomedia/rtp/remotebitrateestimator/RemoteBitrateEstimatorAbsSendTime.java Normal file
View File

@@ -0,0 +1,463 @@
/*
* Copyright @ 2015 Atlassian Pty Ltd
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi_modified.impl.neomedia.rtp.remotebitrateestimator;
import org.jetbrains.annotations.*;
import org.jitsi.nlj.util.*;
import org.jitsi.utils.logging.DiagnosticContext;
import org.jitsi.utils.logging.TimeSeriesLogger;
import org.jitsi.utils.logging2.*;
import java.time.*;
/**
* webrtc.org abs_send_time implementation as of June 26, 2017.
* commit ID: 23fbd2aa2c81d065b84d17b09b747e75672e1159
*
* @author Julian Chukwu
* @author George Politis
*/
public class RemoteBitrateEstimatorAbsSendTime
{
/**
* webrtc/modules/remote_bitrate_estimator/include/bwe_defines.h
*/
static final private int kBitrateWindowMs = 1000;
static final int kDefaultMinBitrateBps = 30000;
static final private int kStreamTimeOutMs = 2000;
/**
* The {@link TimeSeriesLogger} to be used by this instance to print time
* series.
*/
private static final TimeSeriesLogger timeSeriesLogger
= TimeSeriesLogger.getTimeSeriesLogger(
RemoteBitrateEstimatorAbsSendTime.class);
/**
* Defines the number of digits in the AST representation (24 bits, 6.18
* fixed point) after the radix.
*/
private final static int kAbsSendTimeFraction = 18;
/**
* Defines the upshift (left bit-shift) to apply to AST (24 bits, 6.18 fixed
* point) to make it inter-arrival compatible (expanded AST, 32 bits, 6.26
* fixed point).
*/
private final static int kAbsSendTimeInterArrivalUpshift = 8;
/**
* This is used in the {@link InterArrival} computations. In this estimator
* a timestamp group is defined as all packets with a timestamp which are at
* most 5ms older than the first timestamp in that group.
*/
private final static int kTimestampGroupLengthMs = 5;
/**
* Defines the number of digits in the expanded AST representation (32 bits,
* 6.26 fixed point) after the radix.
*/
private final static int kInterArrivalShift
= kAbsSendTimeFraction + kAbsSendTimeInterArrivalUpshift;
/**
* Converts the {@link #kTimestampGroupLengthMs} into "ticks" for use with
* the {@link InterArrival}.
*/
private static final long kTimestampGroupLengthTicks
= (kTimestampGroupLengthMs << kInterArrivalShift) / 1000;
/**
* Defines the expanded AST (32 bits) to millis conversion rate. Units are
* ms per timestamp
*/
private static final double
kTimestampToMs = (double) 1000 / (1 << kInterArrivalShift);
/**
* Reduces the effects of allocations and garbage collection of the method
* {@code incomingPacket}.
*/
private final long[] deltas = new long[3];
/**
* Reduces the effects of allocations and garbage collection of the method
* {@link #incomingPacketInfo(long, long, long, int)}} by promoting the
* {@code RateControlInput} instance from a local variable to a field and
* reusing the same instance across method invocations. (Consequently, the
* default values used to initialize the field are of no importance because
* they will be overwritten before they are actually used.)
*/
private final RateControlInput input
= new RateControlInput(BandwidthUsage.kBwNormal, 0L, 0D);
private long lastPacketTimeMs;
/**
* The time (in millis) when we saw the first packet. Useful to determine
* the probing period.
*/
private long firstPacketTimeMs;
/**
* Keeps track of the last time (in millis) that we updated the bitrate
* estimate.
*/
private long lastUpdateMs;
/**
* The rate control implementation based on additive increases of bitrate
* when no over-use is detected and multiplicative decreases when over-uses
* are detected.
*/
private final AimdRateControl remoteRate;
/**
* Holds the {@link InterArrival}, {@link OveruseEstimator} and
* {@link OveruseDetector} instances of this RBE.
*/
private Detector detector;
/**
* Keeps track of how much data we're receiving.
*/
private BitrateTracker incomingBitrate;
/**
* Determines whether or not the incoming bitrate is initialized or not.
*/
private boolean incomingBitrateInitialized;
/**
* The {@link DiagnosticContext} of this instance.
*/
private final DiagnosticContext diagnosticContext;
/**
* The instance logger.
*/
private final Logger logger;
/**
* Ctor.
*
* @param diagnosticContext the {@link DiagnosticContext} of this instance.
*/
public RemoteBitrateEstimatorAbsSendTime(
@NotNull DiagnosticContext diagnosticContext,
@NotNull Logger parentLogger)
{
this.logger = parentLogger.createChildLogger(getClass().getName());
this.diagnosticContext = diagnosticContext;
this.remoteRate = new AimdRateControl(diagnosticContext);
this.incomingBitrate = new BitrateTracker(Duration.ofMillis(kBitrateWindowMs));
this.incomingBitrateInitialized = false;
this.firstPacketTimeMs = -1;
this.lastPacketTimeMs = -1;
this.lastUpdateMs = -1;
}
/**
* Reset back to the state immediately after construction.
*/
public synchronized void reset()
{
remoteRate.reset();
this.incomingBitrate = new BitrateTracker(Duration.ofMillis(kBitrateWindowMs));
this.incomingBitrateInitialized = false;
this.firstPacketTimeMs = -1;
this.lastPacketTimeMs = -1;
this.lastUpdateMs = -1;
this.detector = null;
}
/**
* Notifies this instance of an incoming packet.
*
* @param nowMs the current time when this method is called
* @param arrivalTimeMs the arrival time of the packet in millis.
* @param sendTimeMs the send time of the packet in millis
* @param payloadSize the payload size of the packet.
*/
public synchronized void incomingPacketInfo(
long nowMs,
long sendTimeMs,
long arrivalTimeMs,
int payloadSize)
{
long sendTime24bits = convertMsTo24Bits(sendTimeMs);
// Shift up send time to use the full 32 bits that inter_arrival
// works with, so wrapping works properly.
long timestamp = sendTime24bits << kAbsSendTimeInterArrivalUpshift;
if (timeSeriesLogger.isTraceEnabled())
{
timeSeriesLogger.trace(diagnosticContext
.makeTimeSeriesPoint("in_pkt", nowMs)
.addField("rbe_id", hashCode())
.addField("recv_ts_ms", arrivalTimeMs)
.addField("timestamp", timestamp)
.addField("pkt_sz_bytes", payloadSize));
}
// should be broken out from here.
// Check if incoming bitrate estimate is valid, and if it
// needs to be reset.
long incomingBitrate_ = incomingBitrate.getRateBps(arrivalTimeMs);
if (incomingBitrate_ != 0)
{
incomingBitrateInitialized = true;
}
else if (incomingBitrateInitialized)
{
// Incoming bitrate had a previous valid value, but now not
// enough data point are left within the current window.
// Reset incoming bitrate estimator so that the window
// size will only contain new data points.
incomingBitrate = new BitrateTracker(Duration.ofMillis(kBitrateWindowMs));
incomingBitrateInitialized = false;
}
incomingBitrate.update(DataSizeKt.getBytes(payloadSize), arrivalTimeMs);
if (firstPacketTimeMs == -1)
{
firstPacketTimeMs = nowMs;
}
boolean updateEstimate = false;
checkTimeouts(nowMs);
lastPacketTimeMs = nowMs;
long[] deltas = this.deltas;
/* long timestampDelta */ deltas[0] = 0;
/* long timeDelta */ deltas[1] = 0;
/* int sizeDelta */ deltas[2] = 0;
if (detector == null)
{
detector = new Detector(new OverUseDetectorOptions(), true, logger);
}
if (detector.interArrival.computeDeltas(
timestamp, arrivalTimeMs, payloadSize, deltas, nowMs))
{
double tsDeltaMs = deltas[0] * kTimestampToMs;
detector.estimator.update(
/* timeDelta */ deltas[1],
/* timestampDelta */ tsDeltaMs,
/* sizeDelta */ (int) deltas[2],
detector.detector.getState(), nowMs);
detector.detector.detect(
detector.estimator.getOffset(), tsDeltaMs,
detector.estimator.getNumOfDeltas(), arrivalTimeMs);
}
// Check if it's time for a periodic update or if we
// should update because of an over-use.
if (lastUpdateMs == -1
|| nowMs - lastUpdateMs > remoteRate.getFeedBackInterval())
{
updateEstimate = true;
}
else if (
detector.detector.getState() == BandwidthUsage.kBwOverusing)
{
long incomingRate_ = incomingBitrate.getRateBps(arrivalTimeMs);
if (incomingRate_ > 0 && remoteRate
.isTimeToReduceFurther(nowMs, incomingBitrate_))
{
updateEstimate = true;
}
}
if (updateEstimate)
{
// The first overuse should immediately trigger a new estimate.
// We also have to update the estimate immediately if we are
// overusing and the target bitrate is too high compared to
// what we are receiving.
input.bwState = detector.detector.getState();
input.incomingBitRate = incomingBitrate.getRateBps(arrivalTimeMs);
input.noiseVar = detector.estimator.getVarNoise();
remoteRate.update(input, nowMs);
remoteRate.updateBandwidthEstimate(nowMs);
updateEstimate = remoteRate.isValidEstimate();
}
if (updateEstimate)
{
lastUpdateMs = nowMs;
}
}
/**
* Timeouts SSRCs that have not received any data for
* kTimestampGroupLengthMs millis.
*
* @param nowMs the current time in millis.
*/
private synchronized void checkTimeouts(long nowMs)
{
if (nowMs - lastPacketTimeMs > kStreamTimeOutMs)
{
detector = null;
// We deliberately don't reset the first_packet_time_ms_
// here for now since we only probe for bandwidth in the
// beginning of a call right now.
}
}
/**
* Called when an RTP sender has a new round-trip time estimate.
*/
public synchronized void onRttUpdate(long nowMs, long avgRttMs)
{
if (timeSeriesLogger.isTraceEnabled())
{
timeSeriesLogger.trace(diagnosticContext
.makeTimeSeriesPoint("new_rtt", nowMs)
.addField("avg_ms", avgRttMs));
}
remoteRate.setRtt(nowMs, avgRttMs);
}
public synchronized long getLatestEstimate()
{
long bitrateBps;
if (!remoteRate.isValidEstimate())
{
return -1;
}
if (detector == null)
{
bitrateBps = 0;
}
else
{
bitrateBps = remoteRate.getLatestEstimate();
}
return bitrateBps;
}
/**
* Sets the minimum bitrate for this instance.
*
* @param minBitrateBps the minimum bitrate in bps.
*/
public synchronized void setMinBitrate(int minBitrateBps)
{
// Called from both the configuration thread and the network thread.
// Shouldn't be called from the network thread in the future.
remoteRate.setMinBitrate(minBitrateBps);
}
/**
* Get various statistics about the estimation process. [Local addition, not in original C++.]
*/
public synchronized @Nullable Statistics getStatistics()
{
if (detector == null)
{
return null;
}
return new Statistics(
detector.estimator.getOffset(),
detector.detector.getThreshold(),
detector.detector.getState()
);
}
/**
* Holds the {@link InterArrival}, {@link OveruseEstimator} and
* {@link OveruseDetector} instances that estimate the remote bitrate of a
* stream.
*/
private class Detector
{
/**
* Computes the send-time and recv-time deltas to feed to the estimator.
*/
private final InterArrival interArrival;
/**
* The Kalman filter implementation that estimates the jitter.
*/
private final OveruseEstimator estimator;
/**
* The overuse detector that compares the jitter to an adaptive threshold.
*/
private final OveruseDetector detector;
/**
* Ctor.
*
* @param options the over-use detector options.
* @param enableBurstGrouping true to activate burst detection, false
* otherwise
*/
Detector(OverUseDetectorOptions options,
boolean enableBurstGrouping,
Logger parentLogger)
{
this.interArrival = new InterArrival(
kTimestampGroupLengthTicks, kTimestampToMs,
enableBurstGrouping, diagnosticContext, parentLogger);
this.estimator = new OveruseEstimator(options, diagnosticContext, logger);
this.detector = new OveruseDetector(options, diagnosticContext);
}
}
/**
* Converts rtp timestamps to 24bit timestamp equivalence
* @param timeMs is the RTP timestamp e.g System.currentTimeMillis().
* @return time stamp representation in 24 bit representation.
*/
public static long convertMsTo24Bits(long timeMs)
{
return (((timeMs << kAbsSendTimeFraction) + 500) / 1000) & 0x00FFFFFF;
}
/**
* Various statistics about the estimation process. [Local addition, not in original C++.]
*/
public static class Statistics {
public double offset;
public double threshold;
public BandwidthUsage hypothesis;
public Statistics(double o, double t, BandwidthUsage h)
{
offset = o;
threshold = t;
hypothesis = h;
}
}
}

38
jitsi-media-transform/src/main/java/org/jitsi_modified/impl/neomedia/rtp/remotebitrateestimator/RemoteBitrateObserver.java Normal file
View File

@@ -0,0 +1,38 @@
/*
* Copyright @ 2015 Atlassian Pty Ltd
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi_modified.impl.neomedia.rtp.remotebitrateestimator;
import java.util.*;
/**
* <tt>RemoteBitrateObserver</tt> is used to signal changes in bitrate estimates
* for the incoming streams.
*
* webrtc/modules/remote_bitrate_estimator/include/remote_bitrate_estimator.h
*
* @author Lyubomir Marinov
*/
public interface RemoteBitrateObserver
{
/**
* Called when a receive channel group has a new bitrate estimate for the
* incoming streams.
*
* @param ssrcs
* @param bitrate
*/
void onReceiveBitrateChanged(Collection<Long> ssrcs, long bitrate);
}

882
jitsi-media-transform/src/main/java/org/jitsi_modified/impl/neomedia/rtp/sendsidebandwidthestimation/SendSideBandwidthEstimation.java Normal file
View File

@@ -0,0 +1,882 @@
/*
* Copyright @ 2018 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi_modified.impl.neomedia.rtp.sendsidebandwidthestimation;
import org.jetbrains.annotations.NotNull;
import org.jitsi.utils.logging.DiagnosticContext;
import org.jitsi.utils.logging.TimeSeriesLogger;
import org.jitsi.utils.logging2.Logger;
import org.jitsi_modified.impl.neomedia.rtp.sendsidebandwidthestimation.config.SendSideBandwidthEstimationConfig;
import java.time.Duration;
import java.util.*;
/**
* Implements the send-side bandwidth estimation described in
* https://tools.ietf.org/html/draft-ietf-rmcat-gcc-01
* Heavily based on code from webrtc.org (send_side_bandwidth_estimation.cc,
* commit ID 7ad9e661f8a035d49d049ccdb87c77ae8ecdfa35).
*
* @author Boris Grozev
*/
public class SendSideBandwidthEstimation
{
/**
* send_side_bandwidth_estimation.cc
*/
private static final int kBweIncreaseIntervalMs = 1000;
/**
* send_side_bandwidth_estimation.cc
*/
private static final long kBweDecreaseIntervalMs = 300;
/**
* send_side_bandwidth_estimation.cc
*/
private static final int kDefaultMinBitrateBps = 10000;
/**
* send_side_bandwidth_estimation.cc
*/
private static final int kDefaultMaxBitrateBps = 1000000000;
/**
* send_side_bandwidth_estimation.cc
*/
private static final int kStartPhaseMs = 2000;
/**
* send_side_bandwidth_estimation.cc
*/
private static final int kLimitNumPackets = 20;
/**
* send_side_bandwidth_estimation.cc
*
* Expecting that RTCP feedback is sent uniformly within [0.5, 1.5]s
* intervals.
*/
private static final long kFeedbackIntervalMs = 1500;
/**
* send_side_bandwidth_estimation.cc
*/
private static final double kPacketReportTimeoutIntervals = 1.2;
/**
* send_side_bandwidth_estimation.cc
*/
private static final long kFeedbackTimeoutIntervals = 3;
/**
* send_side_bandwidth_estimation.cc
*/
private static final long kTimeoutIntervalMs = 1000;
/**
* The random number generator for all instances of this class.
*/
private static final Random kRandom = new Random();
/**
* The <tt>Logger</tt> used by this instance for logging output.
*/
private final Logger logger;
/**
* The {@link TimeSeriesLogger} to be used by this instance to print time
* series.
*/
private static final TimeSeriesLogger timeSeriesLogger
= TimeSeriesLogger.getTimeSeriesLogger(
SendSideBandwidthEstimation.class);
/**
* send_side_bandwidth_estimation.h
*/
private final double low_loss_threshold_;
/**
* send_side_bandwidth_estimation.h
*/
private final double high_loss_threshold_;
/**
* send_side_bandwidth_estimation.h
*/
private final int bitrate_threshold_bps_;
/**
* send_side_bandwidth_estimation.h
*/
private long first_report_time_ms_ = -1;
/**
* send_side_bandwidth_estimation.h
*/
private int lost_packets_since_last_loss_update_Q8_ = 0;
/**
* send_side_bandwidth_estimation.h
*/
private int expected_packets_since_last_loss_update_ = 0;
/**
* send_side_bandwidth_estimation.h
*/
private boolean has_decreased_since_last_fraction_loss_ = false;
/**
* send_side_bandwidth_estimation.h
*
* uint8_t last_fraction_loss_;
*/
private int last_fraction_loss_ = 0;
/**
* send_side_bandwidth_estimation.h
*/
private long last_feedback_ms_ = -1;
/**
* send_side_bandwidth_estimation.h
*/
private long last_packet_report_ms_ = -1;
/**
* send_side_bandwidth_estimation.h
*/
private long last_timeout_ms_ = -1;
/**
* send_side_bandwidth_estimation.h
*/
private final boolean in_timeout_experiment_;
/**
* send_side_bandwidth_estimation.h
*/
private int min_bitrate_configured_ = kDefaultMinBitrateBps;
/**
* send_side_bandwidth_estimation.h
*/
private int max_bitrate_configured_ = kDefaultMaxBitrateBps;
/**
* send_side_bandwidth_estimation.h
*/
private long time_last_decrease_ms_= 0;
/**
* send_side_bandwidth_estimation.h
*/
private long bwe_incoming_ = 0;
/**
* send_side_bandwidth_estimation.h
*/
private long bitrate_;
/**
* send_side_bandwidth_estimation.h
*/
private Deque<BitrateHistoryRecord> min_bitrate_history_ = new LinkedList<>();
/**
* The {@link DiagnosticContext} of this instance.
*/
private final DiagnosticContext diagnosticContext;
/**
* The most recent RTT calculation we've received for our connection with
* the remote endpoint in milliseconds.
*/
private long rttMs;
/**
* The instance that holds stats for this instance.
*/
private final Statistics statistics = new Statistics();
public SendSideBandwidthEstimation(DiagnosticContext diagnosticContext, long startBitrate,
@NotNull Logger parentLogger)
{
logger = parentLogger.createChildLogger(getClass().getName());
this.diagnosticContext = diagnosticContext;
double lossExperimentProbability = SendSideBandwidthEstimationConfig.lossExperimentProbability();
if (kRandom.nextFloat() < lossExperimentProbability)
{
low_loss_threshold_ = SendSideBandwidthEstimationConfig.experimentalLowLossThreshold();
high_loss_threshold_ = SendSideBandwidthEstimationConfig.experimentalHighLossThreshold();
bitrate_threshold_bps_ = (int) SendSideBandwidthEstimationConfig.experimentalBitrateThresholdBps();
}
else
{
low_loss_threshold_ = SendSideBandwidthEstimationConfig.defaultLowLossThreshold();
high_loss_threshold_ = SendSideBandwidthEstimationConfig.defaultHighLossThreshold();
bitrate_threshold_bps_ = (int) SendSideBandwidthEstimationConfig.defaultBitrateThresholdBps();
}
double timeoutExperimentProbability = SendSideBandwidthEstimationConfig.timeoutExperimentProbability();
in_timeout_experiment_
= kRandom.nextFloat() < timeoutExperimentProbability;
setBitrate(startBitrate);
}
public synchronized void reset(long startBitrate)
{
first_report_time_ms_ = -1;
lost_packets_since_last_loss_update_Q8_ = 0;
expected_packets_since_last_loss_update_ = 0;
has_decreased_since_last_fraction_loss_ = false;
last_fraction_loss_ = 0;
last_feedback_ms_ = -1;
last_packet_report_ms_ = -1;
last_timeout_ms_ = -1;
time_last_decrease_ms_= 0;
bwe_incoming_ = 0;
min_bitrate_history_.clear();
setBitrate(startBitrate);
statistics.reset();
}
/**
* bool SendSideBandwidthEstimation::IsInStartPhase(int64_t now_ms)
*/
private synchronized boolean isInStartPhase(long now)
{
return first_report_time_ms_ == -1 ||
now - first_report_time_ms_ < kStartPhaseMs;
}
/**
* int SendSideBandwidthEstimation::CapBitrateToThresholds
*/
private synchronized long capBitrateToThresholds(long bitrate)
{
if (bwe_incoming_ > 0 && bitrate > bwe_incoming_)
{
bitrate = bwe_incoming_;
}
if (bitrate > max_bitrate_configured_)
{
bitrate = max_bitrate_configured_;
}
if (bitrate < min_bitrate_configured_)
{
bitrate = min_bitrate_configured_;
}
return bitrate;
}
/**
* void SendSideBandwidthEstimation::UpdateEstimate(int64_t now_ms)
*/
synchronized public void updateEstimate(long now)
{
long bitrate = bitrate_;
// We trust the REMB during the first 2 seconds if we haven't had any
// packet loss reported, to allow startup bitrate probing.
if (last_fraction_loss_ == 0 && isInStartPhase(now) &&
bwe_incoming_ > bitrate)
{
setBitrate(capBitrateToThresholds(bwe_incoming_));
min_bitrate_history_.clear();
min_bitrate_history_.addLast(new BitrateHistoryRecord(now, bitrate_));
return;
}
updateMinHistory(now);
if (last_packet_report_ms_ == -1)
{
// No feedback received.
bitrate_ = capBitrateToThresholds(bitrate_);
return;
}
long time_since_packet_report_ms = now - last_packet_report_ms_;
long time_since_feedback_ms = now - last_feedback_ms_;
if (time_since_packet_report_ms
< kPacketReportTimeoutIntervals * kFeedbackIntervalMs)
{
// We only care about loss above a given bitrate threshold.
float loss = last_fraction_loss_ / 256.0f;
// We only make decisions based on loss when the bitrate is above a
// threshold. This is a crude way of handling loss which is
// uncorrelated to congestion.
if (bitrate_ < bitrate_threshold_bps_ || loss <= low_loss_threshold_)
{
// Loss < 2%: Increase rate by 8% of the min bitrate in the last
// kBweIncreaseIntervalMs.
// Note that by remembering the bitrate over the last second one can
// rampup up one second faster than if only allowed to start ramping
// at 8% per second rate now. E.g.:
// If sending a constant 100kbps it can rampup immediatly to 108kbps
// whenever a receiver report is received with lower packet loss.
// If instead one would do: bitrate_ *= 1.08^(delta time), it would
// take over one second since the lower packet loss to achieve 108kbps.
bitrate = (long) (min_bitrate_history_.getFirst().bitrate * 1.08 + 0.5);
// Add 1 kbps extra, just to make sure that we do not get stuck
// (gives a little extra increase at low rates, negligible at higher
// rates).
bitrate += 1000;
statistics.update(now, false, LossRegion.LossFree);
}
else if (bitrate_ > bitrate_threshold_bps_)
{
if (loss <= high_loss_threshold_)
{
// Loss between 2% - 10%: Do nothing.
statistics.update(now, false, LossRegion.LossLimited);
}
else
{
// Loss > 10%: Limit the rate decreases to once a kBweDecreaseIntervalMs +
// rtt.
if (!has_decreased_since_last_fraction_loss_ &&
(now - time_last_decrease_ms_) >=
(kBweDecreaseIntervalMs + getRttMs()))
{
time_last_decrease_ms_ = now;
// Reduce rate:
// newRate = rate * (1 - 0.5*lossRate);
// where packetLoss = 256*lossRate;
bitrate = (long) (
(bitrate * (512 - last_fraction_loss_)) / 512.0);
has_decreased_since_last_fraction_loss_ = true;
statistics.update(now, false, LossRegion.LossDegraded);
}
}
}
}
else
{
statistics.update(now, true, null);
if (time_since_feedback_ms >
kFeedbackTimeoutIntervals * kFeedbackIntervalMs
&& (last_timeout_ms_ == -1
|| now - last_timeout_ms_ > kTimeoutIntervalMs))
{
if (in_timeout_experiment_)
{
bitrate_ *= 0.8;
// Reset accumulators since we've already acted on missing
// feedback and shouldn't to act again on these old lost
// packets.
lost_packets_since_last_loss_update_Q8_ = 0;
expected_packets_since_last_loss_update_ = 0;
last_timeout_ms_ = now;
}
}
}
setBitrate(capBitrateToThresholds(bitrate));
}
/**
* Report that one packet has arrived.
*/
public void reportPacketArrived(long now, boolean previouslyReportedLost)
{
if (previouslyReportedLost)
{
lostPacketArrived();
}
else
{
updateReceiverBlock(0, 1, now);
}
}
/**
* Report that one packet was lost.
*/
public void reportPacketLost(long now)
{
updateReceiverBlock(256, 1, now);
}
/**
* void SendSideBandwidthEstimation::UpdateReceiverBlock
*/
synchronized private void updateReceiverBlock(long fraction_lost, long number_of_packets, long now)
{
last_feedback_ms_ = now;
if (first_report_time_ms_ == -1)
{
first_report_time_ms_ = now;
}
// Check sequence number diff and weight loss report
if (number_of_packets > 0)
{
// Calculate number of lost packets.
long num_lost_packets_Q8 = fraction_lost * number_of_packets;
// Accumulate reports.
lost_packets_since_last_loss_update_Q8_ += num_lost_packets_Q8;
expected_packets_since_last_loss_update_ += number_of_packets;
// Don't generate a loss rate until it can be based on enough packets.
if (expected_packets_since_last_loss_update_ < kLimitNumPackets)
return;
has_decreased_since_last_fraction_loss_ = false;
last_fraction_loss_ =
lost_packets_since_last_loss_update_Q8_ /
expected_packets_since_last_loss_update_;
// Reset accumulators.
lost_packets_since_last_loss_update_Q8_ = 0;
expected_packets_since_last_loss_update_ = 0;
last_packet_report_ms_ = now;
}
}
synchronized private void lostPacketArrived()
{
/* This algorithm can only cancel out losses that have occurred
since the last loss update, unfortunately. */
if (lost_packets_since_last_loss_update_Q8_ >= 256)
{
lost_packets_since_last_loss_update_Q8_ -= 256;
}
}
private synchronized void updateMinHistory(long now_ms)
{
// Remove old data points from history.
// Since history precision is in ms, add one so it is able to increase
// bitrate if it is off by as little as 0.5ms.
while (!min_bitrate_history_.isEmpty() &&
now_ms - min_bitrate_history_.getFirst().timestampMs + 1 >
kBweIncreaseIntervalMs)
{
min_bitrate_history_.removeFirst();
}
// Typical minimum sliding-window algorithm: Pop values higher than current
// bitrate before pushing it.
while (!min_bitrate_history_.isEmpty() &&
bitrate_ <= min_bitrate_history_.getLast().bitrate)
{
min_bitrate_history_.removeLast();
}
min_bitrate_history_.addLast(new BitrateHistoryRecord(now_ms, bitrate_));
}
/**
* {@link SendSideBandwidthEstimation#updateReceiverEstimate}
* This is the entry/update point for the estimated bitrate in the
* REMBPacket or a Delay Based Controller estimated bitrate when the
* Delay based controller and the loss based controller lives on the
* send side. see internet draft on "Congestion Control for RTCWEB"
*/
public synchronized void updateReceiverEstimate(long bandwidth)
{
if (timeSeriesLogger.isTraceEnabled())
{
timeSeriesLogger.trace(diagnosticContext
.makeTimeSeriesPoint("bwe_incoming")
.addField("bitrate_bps", bandwidth));
}
bwe_incoming_ = bandwidth;
setBitrate(capBitrateToThresholds(bitrate_));
}
/**
* void SendSideBandwidthEstimation::SetMinMaxBitrate
*/
synchronized public void setMinMaxBitrate(int min_bitrate, int max_bitrate)
{
min_bitrate_configured_ = Math.max(min_bitrate, kDefaultMinBitrateBps);
if (max_bitrate > 0)
{
max_bitrate_configured_ =
Math.max(min_bitrate_configured_, max_bitrate);
}
else
{
max_bitrate_configured_ = kDefaultMaxBitrateBps;
}
}
/**
* Sets the value of {@link #bitrate_}.
* @param newValue the value to set
*/
private synchronized void setBitrate(long newValue)
{
bitrate_ = newValue;
}
/**
* @return the latest estimate.
*/
public long getLatestEstimate()
{
return bitrate_;
}
/**
* @return the latest values of the Receiver Estimated Maximum Bandwidth.
*/
public long getLatestREMB()
{
return bwe_incoming_;
}
/**
* @return the latest effective fraction loss calculated.
* The value is between 0 and 256 (corresponding
* to 0% and 100% respectively).
*/
public int getLatestFractionLoss()
{
return last_fraction_loss_;
}
/**
* @return the statistics specific to this bandwidth estimator.
*/
public Statistics getStatistics()
{
return statistics;
}
public synchronized void onRttUpdate(Duration newRtt)
{
this.rttMs = newRtt.toMillis();
}
/**
* Returns the last calculated RTT to the endpoint in milliseconds.
* @return the last calculated RTT to the endpoint in milliseconds.
*/
private synchronized long getRttMs()
{
if (rttMs < 0)
{
logger.warn("RTT not calculated, using the default of 100ms.");
rttMs = 100;
}
else if (rttMs > 1000)
{
logger.warn(
"RTT suspiciously high (" + rttMs + "ms), capping to 1000ms.");
rttMs = 1000;
}
return rttMs;
}
private static final class BitrateHistoryRecord
{
final long timestampMs;
final long bitrate;
BitrateHistoryRecord(long now, long br)
{
timestampMs = now;
bitrate = br;
}
}
/**
* This class records statistics information about how much time we spend
* in different loss-states (loss-free, loss-limited and loss-degraded).
*/
public class Statistics
{
/**
* The current state {@link LossRegion}.
*/
private LossRegion currentState = null;
/**
* Keeps the time (in millis) of the last transition (including a loop).
*/
private long lastTransitionTimestampMs = -1;
/**
* The cumulative duration (in millis) of the current state
* {@link #currentState} after having looped
* {@link #currentStateConsecutiveVisits} times.
*/
private long currentStateCumulativeDurationMs;
/**
* The number of loops over the current state {@link #currentState}.
*/
private int currentStateConsecutiveVisits;
/**
* The bitrate when we entered the current state {@link #currentState}.
*/
private long currentStateStartBitrateBps;
/**
* Computes the min/max/avg/sd of the bitrate while in
* {@link #currentState}.
*/
private LongSummaryStatistics currentStateBitrateStatistics
= new LongSummaryStatistics();
/**
* Computes the min/max/avg/sd of the loss while in
* {@link #currentState}.
*/
private IntSummaryStatistics currentStateLossStatistics
= new IntSummaryStatistics();
/**
* True when the fields of this class have changed from their default
* value. The purpose is to avoid creating new IntSummaryStatistics and
* LongSummaryStatistics when it's not needed.
*/
private boolean isDirty = false;
/**
* Computes the sum of the duration of the different states.
*/
private LongSummaryStatistics
lossFreeMsStats = new LongSummaryStatistics(),
lossDegradedMsStats = new LongSummaryStatistics(),
lossLimitedMsStats = new LongSummaryStatistics();
public void update(long nowMs)
{
synchronized (SendSideBandwidthEstimation.this)
{
long time_since_packet_report_ms
= nowMs - last_packet_report_ms_;
boolean currentStateHasTimedOut = time_since_packet_report_ms
< kPacketReportTimeoutIntervals * kFeedbackIntervalMs;
update(nowMs, currentStateHasTimedOut, null);
}
}
/**
* Records a state transition and updates the statistics information.
*
* @param nowMs the time (in millis) of the transition.
* @param currentStateHasTimedOut true if the current state has timed
* out, i.e. we haven't received receiver reports "in a while".
* @param nextState the that the bwe is transitioning to.
*/
void update(
long nowMs, boolean currentStateHasTimedOut, LossRegion nextState)
{
synchronized (SendSideBandwidthEstimation.this)
{
if (lastTransitionTimestampMs > -1 && !currentStateHasTimedOut)
{
isDirty = true;
currentStateCumulativeDurationMs
+= nowMs - lastTransitionTimestampMs;
}
lastTransitionTimestampMs = nowMs;
if (!currentStateHasTimedOut)
{
isDirty = true;
// If the current state has not timed out, then update the
// stats that we gather.
currentStateLossStatistics.accept(last_fraction_loss_);
currentStateConsecutiveVisits++; // we start counting from 0.
if (this.currentState == nextState)
{
currentStateBitrateStatistics.accept(bitrate_);
return;
}
}
if (this.currentState != null)
{
// This is not a loop, we're transitioning to another state.
// Record how much time we've spent on this state, how many
// times we've looped through it and what was the impact on
// the bitrate.
switch (this.currentState)
{
case LossDegraded:
lossDegradedMsStats.accept(
currentStateCumulativeDurationMs);
break;
case LossFree:
lossFreeMsStats.accept(currentStateCumulativeDurationMs);
break;
case LossLimited:
lossLimitedMsStats.accept(
currentStateCumulativeDurationMs);
break;
}
if (timeSeriesLogger.isTraceEnabled())
{
timeSeriesLogger.trace(diagnosticContext
.makeTimeSeriesPoint("loss_estimate")
.addField("state", currentState.name())
.addField("max_loss",
currentStateLossStatistics.getMax() / 256.0f)
.addField("min_loss",
currentStateLossStatistics.getMin() / 256.0f)
.addField("avg_loss",
currentStateLossStatistics.getAverage()/256.0f)
.addField("max_bps",
currentStateBitrateStatistics.getMax())
.addField("min_bps",
currentStateBitrateStatistics.getMin())
.addField("avg_bps",
currentStateBitrateStatistics.getAverage())
.addField("duration_ms",
currentStateCumulativeDurationMs)
.addField("consecutive_visits",
currentStateConsecutiveVisits)
.addField("bitrate_threshold",
bitrate_threshold_bps_)
.addField("low_loss_threshold",
low_loss_threshold_)
.addField("high_loss_threshold",
high_loss_threshold_)
.addField("delta_bps",
bitrate_ - currentStateStartBitrateBps)
.addField("bitrate",
bitrate_)
.addField("bwe_incoming",
bwe_incoming_)
.addField("rtt_ms",
rttMs));
}
}
currentState = nextState;
currentStateStartBitrateBps = bitrate_;
if (isDirty)
{
currentStateLossStatistics = new IntSummaryStatistics();
currentStateBitrateStatistics = new LongSummaryStatistics();
currentStateConsecutiveVisits = 0;
currentStateCumulativeDurationMs = 0;
isDirty = false;
}
currentStateBitrateStatistics.accept(bitrate_);
}
}
/** Return to state immediately after construction. */
void reset()
{
synchronized (SendSideBandwidthEstimation.this)
{
currentState = null;
lastTransitionTimestampMs = -1;
lossFreeMsStats = new LongSummaryStatistics();
lossDegradedMsStats = new LongSummaryStatistics();
lossLimitedMsStats = new LongSummaryStatistics();
if (isDirty)
{
currentStateLossStatistics = new IntSummaryStatistics();
currentStateBitrateStatistics = new LongSummaryStatistics();
currentStateConsecutiveVisits = 0;
currentStateCumulativeDurationMs = 0;
isDirty = false;
}
}
}
/**
* @return the number of millis spent in the loss-limited state.
*/
public long getLossLimitedMs()
{
synchronized (SendSideBandwidthEstimation.this)
{
return lossLimitedMsStats.getSum();
}
}
/**
* @return the number of millis spent in the loss-degraded state.
*/
public long getLossDegradedMs()
{
synchronized (SendSideBandwidthEstimation.this)
{
return lossDegradedMsStats.getSum();
}
}
/**
* @return the number of millis spent in the loss-free state.
*/
public long getLossFreeMs()
{
synchronized (SendSideBandwidthEstimation.this)
{
return lossFreeMsStats.getSum();
}
}
}
/**
* Represents the loss-based controller states.
*/
private enum LossRegion
{
/**
* Loss is between 2% and 10%.
*/
LossLimited,
/**
* Loss is above 10%.
*/
LossDegraded,
/**
* Loss is bellow 2%.
*/
LossFree
}
}

53
jitsi-media-transform/src/main/java/org/jitsi_modified/impl/neomedia/transform/PacketTransformer.java Normal file
View File

@@ -0,0 +1,53 @@
/*
* Copyright @ 2018 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi_modified.impl.neomedia.transform;
import org.jitsi.rtp.*;
/**
* Encapsulate the concept of packet transformation. Given an array of packets,
* <tt>PacketTransformer</tt> can either "transform" each one of them, or
* "reverse transform" (e.g. restore) each one of them.
*
* @author Bing SU (nova.su@gmail.com)
* @author Boris Grozev
*/
public interface PacketTransformer
{
/**
* Closes this <tt>PacketTransformer</tt> i.e. releases the resources
* allocated by it and prepares it for garbage collection.
*/
public void close();
/**
* Reverse-transforms each packet in an array of packets. Null values
* must be ignored.
*
* @param pkts the transformed packets to be restored.
* @return the restored packets.
*/
public Packet[] reverseTransform(Packet[] pkts);
/**
* Transforms each packet in an array of packets. Null values must be
* ignored.
*
* @param pkts the packets to be transformed
* @return the transformed packets
*/
public Packet[] transform(Packet[] pkts);
}

29
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/AudioLevelListener.kt Normal file
View File

@@ -0,0 +1,29 @@
/*
* Copyright @ 2018 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
interface AudioLevelListener {
/**
* Process the audio level from a received audio RTP packet.
*
* @param sourceSsrc The SSRC identifying the source of the audio stream.
* @param level The audio level.
* @return A boolean set to {@code true} if this packet should be
* discarded without being forwarded to other endpoints.
*/
fun onLevelReceived(sourceSsrc: Long, level: Long): Boolean
}

27
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/DebugStateMode.kt Normal file
View File

@@ -0,0 +1,27 @@
/*
* Copyright @ 2025 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
enum class DebugStateMode {
/** Include full details, mostly for manually debugging */
FULL,
/** Include all information needed for stats/metric collection (e.g. rtcstats) */
STATS,
/** Include only the most important information, e.g. listing conferences and endpoints. */
SHORT
}

27
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/Event.kt Normal file
View File

@@ -0,0 +1,27 @@
/*
* Copyright @ 2018 - Present, 8x8 Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
import org.jitsi.utils.MediaType
interface Event
class SetMediaSourcesEvent(
val mediaSourceDescs: Array<MediaSourceDesc>,
val signaledMediaSourceDescs: Array<MediaSourceDesc>
) : Event
class SetLocalSsrcEvent(val mediaType: MediaType, val ssrc: Long) : Event

20
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/EventHandler.kt Normal file
View File

@@ -0,0 +1,20 @@
/*
* Copyright @ 2018 - Present, 8x8 Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
interface EventHandler {
fun handleEvent(event: Event)
}

21
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/Features.kt Normal file
View File

@@ -0,0 +1,21 @@
/*
* Copyright @ 2018 - Present, 8x8 Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
enum class Features {
TRANSCEIVER_PCAP_DUMP
}

222
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/MediaSourceDesc.kt Normal file
View File

@@ -0,0 +1,222 @@
/*
* Copyright @ 2018 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
import org.jitsi.nlj.rtp.VideoRtpPacket
import org.jitsi.nlj.util.Bandwidth
import org.jitsi.nlj.util.bps
import org.jitsi.rtp.rtp.RtpPacket
import org.jitsi.utils.ArrayUtils
import java.util.Collections
import java.util.NavigableMap
import java.util.TreeMap
/**
* Represents a collection of [RtpLayerDesc]s that encode the same
* media source. This specific implementation provides webrtc simulcast stream
* suspension detection.
*
* We take the definition of "Media Source" from RFC 7656. It takes a single
* logical source of media, which might be represented by multiple encoded streams.
*
* @author George Politis
*/
class MediaSourceDesc
@JvmOverloads constructor(
/**
* The [RtpEncodingDesc]s that this [MediaSourceDesc]
* possesses, ordered by their subjective quality from low to high.
*/
val rtpEncodings: Array<RtpEncodingDesc>,
/**
* A string which identifies the owner of this source (e.g. the endpoint
* which is the sender of the source).
*/
val owner: String,
/**
* A string which identifies this source.
*/
val sourceName: String,
/**
* The {@link VideoType} signaled for this media source (defaulting to {@code CAMERA} if nothing has been signaled).
*/
var videoType: VideoType = VideoType.CAMERA,
/**
* Whether this is a synthetic source, i.e. one generated by the bridge rather than sent by an endpoint.
*/
val synthetic: Boolean = false,
) {
/**
* Current single-list view of all the encodings' layers.
*/
private lateinit var layers: List<RtpLayerDesc>
/**
* Allow the lookup of a layer by the encoding id of a received packet.
*/
private val layersById: MutableMap<Long, RtpLayerDesc> = HashMap()
/**
* Allow the lookup of a layer by index.
*/
private val layersByIndex: NavigableMap<Int, RtpLayerDesc> = TreeMap()
/**
* Get a view of the source's RTP layers, in quality order.
*/
val rtpLayers: List<RtpLayerDesc>
@Synchronized
get() = layers
/**
* Update the layer cache. Should be synchronized on [this].
*/
private fun updateLayerCache() {
layersById.clear()
layersByIndex.clear()
val tempLayers = ArrayList<RtpLayerDesc>()
for (encoding in rtpEncodings) {
for (layer in encoding.layers) {
layersById[encoding.encodingId(layer)] = layer
layersByIndex[layer.index] = layer
tempLayers.add(layer)
}
}
layers = Collections.unmodifiableList(tempLayers)
}
init {
updateLayerCache()
}
/**
* Gets the last "stable" bitrate (in bps) of the encoding of the specified
* index. The "stable" bitrate is measured on every new frame and with a
* 5000ms window.
*
* If the bitrate for the specified index is 0, return bitrate of the highest-
* index layer less than the index with a non-zero bitrate.
*
* @return the last "stable" bitrate (bps) of the encoding with a non-zero rate
* at or below the specified index.
*/
@Synchronized
fun getBitrate(nowMs: Long, idx: Int): Bandwidth {
for (entry in layersByIndex.headMap(idx, true).descendingMap()) {
val bitrate = entry.value.getBitrate(nowMs)
if (bitrate.bps > 0) {
return bitrate
}
}
return 0.bps
}
@Synchronized
fun hasRtpLayers(): Boolean = layers.isNotEmpty()
@Synchronized
fun numRtpLayers(): Int = layersByIndex.size
val primarySSRC: Long
get() = rtpEncodings[0].primarySSRC
@Synchronized
fun getRtpLayerByQualityIdx(idx: Int): RtpLayerDesc? = layersByIndex[idx]
@Synchronized
fun findRtpLayerDescs(videoRtpPacket: VideoRtpPacket): Collection<RtpLayerDesc> {
if (ArrayUtils.isNullOrEmpty(rtpEncodings)) {
return emptyList()
}
return videoRtpPacket.getEncodingIds().mapNotNull { layersById[it] }
}
@Synchronized
fun findRtpEncodingDesc(ssrc: Long): RtpEncodingDesc? = rtpEncodings.find { it.hasSsrc(ssrc) }
@Synchronized
fun getEncodingLayers(ssrc: Long): Array<RtpLayerDesc> {
val enc = findRtpEncodingDesc(ssrc) ?: return emptyArray()
return Array(enc.layers.size) { i ->
enc.layers[i].copy()
}
}
@Synchronized
fun setEncodingLayers(layers: Array<RtpLayerDesc>, ssrc: Long) {
val enc = findRtpEncodingDesc(ssrc) ?: return
enc.layers = layers
updateLayerCache()
}
/**
* Clone an existing media source desc, inheriting layer descs' statistics.
*/
@Synchronized
fun copy() = MediaSourceDesc(
Array(this.rtpEncodings.size) { i -> this.rtpEncodings[i].copy() },
this.owner,
this.sourceName,
this.videoType,
this.synthetic
)
override fun toString(): String = "MediaSourceDesc[name=$sourceName owner=$owner, videoType=$videoType, " +
"synthetic=$synthetic, encodings=${rtpEncodings.joinToString(",")}]"
/**
* Checks whether the given SSRC matches this source's [primarySSRC].
* This is mostly useful only for determining quickly whether two source
* descriptions describe the same source; other functions (probably [hasSsrc]) should be used
* to match received media packets.
*
* @param ssrc the SSRC to match.
* @return `true` if the specified `ssrc` is the primary SSRC
* for this source.
*/
fun matches(ssrc: Long) = rtpEncodings.getOrNull(0)?.primarySSRC == ssrc
/**
* Checks whether any encoding of this source has this [ssrc]
*/
@Synchronized
fun hasSsrc(ssrc: Long) = rtpEncodings.any { it.hasSsrc(ssrc) }
}
/**
* Clone an array of media source descriptions.
*/
fun Array<MediaSourceDesc>.copy() = Array(this.size) { i -> this[i].copy() }
fun Array<MediaSourceDesc>.findRtpLayerDescs(packet: VideoRtpPacket): Collection<RtpLayerDesc> = this.flatMap {
it.findRtpLayerDescs(packet)
}
fun Array<MediaSourceDesc>.findRtpSourceByPrimary(ssrc: Long): MediaSourceDesc? = this.find { it.matches(ssrc) }
fun Array<MediaSourceDesc>.findRtpSource(ssrc: Long): MediaSourceDesc? = this.find { it.hasSsrc(ssrc) }
fun Array<MediaSourceDesc>.findRtpSource(packet: RtpPacket): MediaSourceDesc? = findRtpSource(packet.ssrc)
fun Array<MediaSourceDesc>.findRtpEncodingId(packet: VideoRtpPacket): Int? {
for (source in this) {
source.findRtpEncodingDesc(packet.ssrc)?.let {
return it.eid
}
}
return null
}

75
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/MediaSources.kt Normal file
View File

@@ -0,0 +1,75 @@
/*
* Copyright @ 2018 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
import com.fasterxml.jackson.databind.node.JsonNodeFactory
import com.fasterxml.jackson.databind.node.ObjectNode
/**
* Maintains an array of [MediaSourceDesc]. The set method preserves the existing sources that match one of the new
* sources, because [MediaSourceDesc] holds some local state (rate statistics) that we would like to keep. Ideally
* this state should be out of the source descriptor, in which case this class would be obsolete.
*
* @author Boris Grozev
*/
class MediaSources {
private var sources: Array<MediaSourceDesc> = arrayOf()
fun setMediaSources(newSources: Array<MediaSourceDesc>): Boolean {
val oldSources = sources
if (oldSources.isEmpty() || newSources.isEmpty()) {
sources = newSources
return oldSources.size != newSources.size
}
var cntMatched = 0
val mergedSources: Array<MediaSourceDesc> = Array(newSources.size) { i ->
val newPrimarySSRC = newSources[i].primarySSRC
for (j in 0 until oldSources.size) {
if (oldSources[j].matches(newPrimarySSRC)) {
cntMatched++
// NOTE: we deliberately do not update the old source instance
// with the encodings of the new one. Values set on the
// source are more likely to be correct than ones generated
// from signaling about the source. (Revisit this if we
// encounter a scenario where this isn't true...)
return@Array oldSources[j]
}
}
newSources[i]
}
sources = mergedSources
return oldSources.size != newSources.size || cntMatched != oldSources.size
}
fun getMediaSources(): Array<MediaSourceDesc> = sources
fun debugState(): ObjectNode = JsonNodeFactory.instance.objectNode().apply {
sources.forEach { source ->
val sourceNode = JsonNodeFactory.instance.objectNode().apply {
put("owner", source.owner)
put("video_type", source.videoType.toString())
source.rtpEncodings.forEach {
set<ObjectNode>("rtp_encoding_${it.primarySSRC}", it.debugState())
}
}
set<ObjectNode>(source.sourceName, sourceNode)
}
}
}

81
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/PacketHandler.kt Normal file
View File

@@ -0,0 +1,81 @@
/*
* Copyright @ 2018 - Present, 8x8 Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
import org.jitsi.nlj.stats.NodeStatsBlock
import org.jitsi.nlj.transform.NodeStatsProducer
import org.jitsi.nlj.util.addMbps
interface PacketHandler {
/**
* Process the given packets
*/
fun processPacket(packetInfo: PacketInfo)
}
/**
* A [PacketHandler] which tracks statistics about the packets it handles
*/
abstract class StatsKeepingPacketHandler : PacketHandler, NodeStatsProducer {
private val statistics = Statistics()
final override fun processPacket(packetInfo: PacketInfo) {
val now = System.currentTimeMillis()
statistics.packetsReceived++
statistics.bytesReceived += packetInfo.packet.length
if (statistics.firstPacketReceivedTime == 0L) {
statistics.firstPacketReceivedTime = now
}
statistics.lastPacketReceivedTime = now
doProcessPacket(packetInfo)
}
/**
* We override [processPacket] above and prevent subclasses from doing so
* to make sure this class sees all packets that pass through it and it
* can track stats correctly. This method should be implemented by
* subclasses as a stand-in for [processPacket], as it is called once
* all the stat-tracking has been done.
*/
protected abstract fun doProcessPacket(packetInfo: PacketInfo)
override fun getNodeStats(): NodeStatsBlock = statistics.getNodeStats()
/**
* Statistics common to [PacketHandler]s which are interesting to track
*/
data class Statistics(
var firstPacketReceivedTime: Long = 0,
var lastPacketReceivedTime: Long = 0,
var bytesReceived: Long = 0,
var packetsReceived: Long = 0
) : NodeStatsProducer {
override fun getNodeStats(): NodeStatsBlock = NodeStatsBlock("Packet handler stats").apply {
addNumber(RECEIVED_PACKETS, packetsReceived)
addNumber(RECEIVED_BYTES, bytesReceived)
addNumber(RECEIVED_DURATION_MS, lastPacketReceivedTime - firstPacketReceivedTime)
addMbps(RECEIVED_BITRATE_MBPS, RECEIVED_BYTES, RECEIVED_DURATION_MS)
}
companion object {
private const val RECEIVED_PACKETS = "received_packets"
private const val RECEIVED_BYTES = "received_bytes"
private const val RECEIVED_DURATION_MS = "received_duration_ms"
private const val RECEIVED_BITRATE_MBPS = "received_bitrate_mbps"
}
}
}

298
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/PacketInfo.kt Normal file
View File

@@ -0,0 +1,298 @@
/*
* Copyright @ 2018 - Present, 8x8 Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings
import org.jitsi.config.JitsiConfig
import org.jitsi.metaconfig.config
import org.jitsi.nlj.format.PayloadType
import org.jitsi.rtp.Packet
import org.jitsi.utils.logging2.createLogger
import java.time.Clock
import java.time.Duration
import java.time.Instant
import java.util.Collections
@SuppressFBWarnings("CN_IMPLEMENTS_CLONE_BUT_NOT_CLONEABLE")
class EventTimeline(
/**
* We want this thread safe, because while PacketInfo objects are only handled by a single thread at a time,
* [StatsKeepingNode] may add an "exit" event after the packets has been added to another queue potentially handled
* by a different thread. This is not critical as it only affects the timeline and the result is just some "exit"
* events missing from the trace logs.
*/
timelineArg: MutableList<Pair<String, Duration>> = mutableListOf(),
private val clock: Clock = Clock.systemUTC()
) : Iterable<Pair<String, Duration>> {
private val timeline = Collections.synchronizedList(timelineArg)
/**
* The [referenceTime] refers to the first timestamp we have
* in the timeline. In the timeline this is used as time "0" and
* all other times are represented as deltas from this 0.
*/
var referenceTime: Instant? = null
val size: Int
get() = timeline.size
fun addEvent(desc: String) {
val now = clock.instant()
if (referenceTime == null) {
referenceTime = now
}
timeline.add(desc to Duration.between(referenceTime, now))
}
fun clone(): EventTimeline {
val clone = EventTimeline(timeline.toMutableList(), clock)
clone.referenceTime = referenceTime
return clone
}
override fun iterator(): Iterator<Pair<String, Duration>> = timeline.iterator()
/**
* Return the total time between this packet's first event and last event
* or -1 if there is no reference time
*/
fun totalDelay(): Duration {
return referenceTime?.let {
return timeline.last().second
} ?: Duration.ofMillis(-1)
}
override fun toString(): String = with(StringBuffer()) {
referenceTime?.let {
append("Reference time: $referenceTime; ")
synchronized(timeline) {
append(timeline.joinToString(separator = "; "))
}
} ?: run {
append("[No timeline]")
}
toString()
}
}
/**
* The origin of a packet in the system; used to track outgoing packets in [OutgoingStatisticsTracker]
* to measure the bitrates of each type of data.
*
* Currently only used for RTP, so RTCP, SCTP, and Datachannel will all be either Routed or Misc.
*/
enum class PacketOrigin {
Routed,
Retransmission,
Probing,
Padding,
Synthesized,
Misc
/* TODO: Add RTCP, SCTP, and datachannel if needed */
}
/**
* [PacketInfo] is a wrapper around a [Packet] instance to be passed through
* a pipeline. Since the [Packet] can change as it moves through the pipeline
* (as it is parsed into different types), the wrapping [PacketInfo] stays consistent
* and allows for metadata to be passed along with a packet.
*/
@SuppressFBWarnings("CN_IMPLEMENTS_CLONE_BUT_NOT_CLONEABLE")
open class PacketInfo @JvmOverloads constructor(
var packet: Packet,
/** The original length of the packet, i.e. before decryption. Stays unchanged even if the packet is updated. */
val originalLength: Int = packet.length,
val timeline: EventTimeline? = if (enableTimeline) EventTimeline() else null
) {
/**
* An explicit tag for when this packet was originally received (assuming it
* was an incoming packet and not one created by jvb itself).
*/
var receivedTime: Instant? = null
set(value) {
field = value
if (timeline != null && timeline.referenceTime == null) {
timeline.referenceTime = value
}
}
/** Whether the packet originally had cryptex RTP header extensions. */
var originalHadCryptex: Boolean = false
/**
* Whether this packet has been recognized to contain only shouldDiscard.
*/
var shouldDiscard: Boolean = false
/**
* The ID of the endpoint associated with this packet (i.e. the source endpoint).
*/
var endpointId: String? = null
/**
* Whether this packet indicates a point in which its stream's layering changed, in
* a way that indicates that bitrate allocation may need to be recomputed.
*/
var layeringChanged = false
var payloadType: PayloadType? = null
/**
* The RFC 6464 audio level (0-127, i.e. -dBov: 0 is loudest, 127 is silence) reported by the sender in the
* ssrc-audio-level RTP header extension, or null if the packet had no such extension. Populated on the incoming
* path by [org.jitsi.nlj.transform.node.incoming.AudioLevelReader].
*/
var audioLevel: Int? = null
/**
* The RFC 6464 Voice Activity Detection flag reported by the sender in the ssrc-audio-level RTP header extension,
* or null if the packet had no such extension. Populated alongside [audioLevel].
*/
var vad: Boolean? = null
/**
* The payload verification string for the packet, or 'null' if payload verification is disabled. Calculating the
* it is expensive, thus we only do it when the flag is enabled.
*/
var payloadVerification = if (enablePayloadVerification) packet.payloadVerification else null
/**
* Information about whether this packet is used for probing by the transport-cc engine.
* The type is internal to that object.
*/
var probingInfo: Any? = null
/**
* The origin of the packet, used for tracking the sources of media being routed.
*/
var packetOrigin: PacketOrigin = PacketOrigin.Misc
/**
* Re-calculates the expected payload verification string. This should be called any time that the code
* intentionally modifies the packet in a way that could change the verification string (for example, re-creates
* it with a new type (parsing), or intentionally modifies the payload (SRTP)).
*/
fun resetPayloadVerification() {
payloadVerification = if (enablePayloadVerification) packet.payloadVerification else null
}
/**
* Get the contained packet cast to [ExpectedPacketType]
*/
@Suppress("UNCHECKED_CAST")
fun <ExpectedPacketType : Packet> packetAs(): ExpectedPacketType = packet as ExpectedPacketType
/**
* Create a deep clone of this PacketInfo (both the contained packet and the metadata map
* will be copied for the cloned PacketInfo).
*/
fun clone(): PacketInfo {
val clone = PacketInfo(packet.clone(), originalLength, timeline?.clone())
clone.receivedTime = receivedTime
clone.originalHadCryptex = originalHadCryptex
clone.shouldDiscard = shouldDiscard
clone.endpointId = endpointId
clone.payloadType = payloadType
clone.audioLevel = audioLevel
clone.vad = vad
clone.layeringChanged = layeringChanged
clone.payloadVerification = payloadVerification
clone.probingInfo = probingInfo
clone.packetOrigin = packetOrigin
@Suppress("UNCHECKED_CAST") // ArrayList.clone() really does return ArrayList, not Object.
clone.onSentActions = onSentActions?.clone() as ArrayList<(PacketInfo) -> Unit>?
return clone
}
fun addEvent(desc: String) = timeline?.addEvent(desc)
/**
* The list of pending actions, or [null] if none.
*/
private var onSentActions: ArrayList<(PacketInfo) -> Unit>? = null
/**
* Add an action to be performed when the packet is sent (i.e. when this packet's
* [sent] method is called).
*
* If this [PacketInfo] object is cloned, the action will be called for every
* cloned instance. If packet is dropped (i.e. [sent] is never called), the
* action will not be called.
*/
fun onSent(action: (PacketInfo) -> Unit) {
synchronized(this) {
if (onSentActions == null) {
onSentActions = ArrayList(1)
}
onSentActions!!.add(action)
}
}
/**
* Invoke any actions previously registered with this [PacketInfo]'s [onSent]
* method. This should be called just before, or after, this packet is sent.
*/
fun sent() {
var actions: List<(PacketInfo) -> Unit> = Collections.emptyList()
synchronized(this) {
onSentActions?.let {
actions = it
onSentActions = null
} ?: run { return@sent }
}
for (action in actions) {
action.invoke(this)
}
}
companion object {
private val enableTimeline: Boolean by config {
"jmt.debug.packet-timeline.enabled".from(JitsiConfig.newConfig)
}
private val enablePayloadVerificationDefault: Boolean by config {
"jmt.debug.payload-verification.enabled".from(JitsiConfig.newConfig)
}
/**
* If this is enabled all [Node]s will verify that the payload didn't unexpectedly change. This is expensive.
*/
@field:Suppress("ktlint:standard:property-naming")
var enablePayloadVerification = enablePayloadVerificationDefault
init {
if (enableTimeline) {
createLogger().info("Packet timeline is enabled.")
}
if (enablePayloadVerification) {
createLogger().info("Payload verification is enabled.")
}
}
}
}
/**
* This is a specialization of [org.jitsi.nlj.util.forEachAs] method which makes it easier
* to operate on lists of [PacketInfo] when the caller wants to treat the contained [Packet]
* as a specific packet type. This method iterates over the iterable of [PacketInfo]s and calls
* the given lambda with the [PacketInfo] instance and the contained [Packet] instance, cast
* as [ExpectedPacketType]. This will throw if the cast attempt is unsuccessful.
*/
@Suppress("UNCHECKED_CAST")
inline fun <ExpectedPacketType> Iterable<PacketInfo>.forEachAs(action: (PacketInfo, ExpectedPacketType) -> Unit) {
for (element in this) action(element, element.packet as ExpectedPacketType)
}

203
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/RtpEncodingDesc.kt Normal file
View File

@@ -0,0 +1,203 @@
/*
* Copyright @ 2018 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
import com.fasterxml.jackson.databind.node.JsonNodeFactory
import com.fasterxml.jackson.databind.node.ObjectNode
import org.jitsi.nlj.rtp.SsrcAssociationType
import org.jitsi.nlj.rtp.VideoRtpPacket
import org.jitsi.nlj.stats.NodeStatsBlock
/**
* Keeps track of information specific to an RTP encoded stream
* (and its associated secondary sources).
*
* @author Jonathan Lennox
*/
class RtpEncodingDesc
@JvmOverloads
constructor(
/**
* The primary SSRC for this encoding.
*/
val primarySSRC: Long,
/**
* The [RtpLayerDesc]s describing the encoding's layers.
*/
initialLayers: Array<RtpLayerDesc>,
/**
* The ID of this encoding.
*/
val eid: Int = requireNotNull(initialLayers.getOrNull(0)?.eid) {
"initialLayers may not be empty if no explicit EID is provided"
}
) {
constructor(primarySSRC: Long, eid: Int) : this(primarySSRC, arrayOf(), eid)
/**
* The ssrcs associated with this encoding (for example, RTX or FLEXFEC)
* Maps ssrc -> type [SsrcAssociationType] (rtx, etc.)
*/
private val secondarySsrcs: MutableMap<Long, SsrcAssociationType> = HashMap()
fun addSecondarySsrc(ssrc: Long, type: SsrcAssociationType) {
secondarySsrcs[ssrc] = type
}
/**
* All SSRCs (primary and secondary) associated with this encoding.
*/
val ssrcs: Collection<Long>
get() = HashSet<Long>().also { set ->
set.add(primarySSRC)
set.addAll(secondarySsrcs.keys)
}
private fun validateLayerEids(layers: Array<RtpLayerDesc>) {
for (layer in layers) {
require(layer.eid == eid) { "Cannot add layer with EID ${layer.eid} to encoding with EID $eid" }
}
}
init {
validateLayerEids(initialLayers)
}
private var nominalHeight = initialLayers.getNominalHeight()
internal var layers = initialLayers
set(newLayers) {
validateLayerEids(newLayers)
/* Check if the new layer set is a single spatial layer that doesn't specify a height - if so, we
* want to apply the nominal height to them.
*/
val useNominalHeight = nominalHeight != RtpLayerDesc.NO_HEIGHT &&
newLayers.all { it.sid == 0 } &&
newLayers.all { it.height == RtpLayerDesc.NO_HEIGHT }
/* Copy the rate statistics objects from the old layers to the new layers
* with matching layer IDs.
*/
/* Note: because layer arrays are sorted by ID we could avoid creating this
* intermediate map object, and do this in a single pass in O(1) space.
* The number of layers is small enough that this more complicated code
* is probably unnecessary, though.
*/
val oldLayerMap = field.associateBy { it.layerId }
for (newLayer in newLayers) {
oldLayerMap[newLayer.layerId]?.let {
newLayer.inheritFrom(it)
}
if (useNominalHeight) {
newLayer.height = nominalHeight
}
}
if (!useNominalHeight) {
val newNominalHeight = newLayers.getNominalHeight()
if (newNominalHeight != RtpLayerDesc.NO_HEIGHT) {
nominalHeight = newNominalHeight
}
}
field = newLayers
}
/**
* @return the "id" of a layer within this source, across all encodings. This is a server-side id and should
* not be confused with any encoding id defined in the client (such as the
* rid). This server-side id is used in the layer lookup table that is
* maintained in [MediaSourceDesc].
*/
fun encodingId(layer: RtpLayerDesc): Long = calcEncodingId(primarySSRC, layer.layerId)
/**
* Get the secondary ssrc for this encoding that corresponds to the given
* type
* @param type the type of the secondary ssrc (e.g. RTX)
* @return the ssrc for the encoding that corresponds to the given type,
* if it exists; otherwise -1
*/
fun getSecondarySsrc(type: SsrcAssociationType): Long {
for ((key, value) in secondarySsrcs) {
if (value == type) {
return key
}
}
return -1
}
/**
* Clone an existing encoding desc, inheriting layer descs' statistics,
* modifying only specific values.
*/
fun copy(
primarySSRC: Long = this.primarySSRC,
layers: Array<RtpLayerDesc> = Array(this.layers.size) { i -> this.layers[i].copy() }
) = RtpEncodingDesc(primarySSRC, layers, eid).also {
this.secondarySsrcs.forEach { (ssrc, type) -> it.addSecondarySsrc(ssrc, type) }
}
/**
* {@inheritDoc}
*/
override fun toString(): String = "primary_ssrc=$primarySSRC,secondary_ssrcs=$secondarySsrcs," +
"layers=\n ${layers.joinToString(separator = "\n ")}"
/**
* Gets a boolean indicating whether the SSRC specified in the
* arguments is used by this encoding.
*
* @param ssrc the SSRC to match.
*/
fun hasSsrc(ssrc: Long): Boolean = if (primarySSRC == ssrc) {
true
} else {
secondarySsrcs.containsKey(ssrc)
}
/**
* Extracts a [NodeStatsBlock] from an [RtpEncodingDesc].
*/
fun debugState(): ObjectNode = JsonNodeFactory.instance.objectNode().apply {
put("rtx_ssrc", getSecondarySsrc(SsrcAssociationType.RTX))
put("fec_ssrc", getSecondarySsrc(SsrcAssociationType.FEC))
put("eid", eid)
put("nominal_height", nominalHeight)
for (layer in layers) {
set<ObjectNode>(layer.indexString(), layer.debugState())
}
}
companion object {
fun calcEncodingId(ssrc: Long, layerId: Int) = ssrc or (layerId.toLong() shl 32)
}
}
fun VideoRtpPacket.getEncodingIds(): Collection<Long> = this.layerIds.map { RtpEncodingDesc.calcEncodingId(ssrc, it) }
/**
* Get the "nominal" height of a set of layers - if they all indicate the same spatial layer and same height.
*/
private fun Array<RtpLayerDesc>.getNominalHeight(): Int {
if (isEmpty()) {
return RtpLayerDesc.NO_HEIGHT
}
val firstHeight = first().height
if (!(all { it.sid == 0 } || all { it.sid == -1 })) {
return RtpLayerDesc.NO_HEIGHT
}
if (any { it.height != firstHeight }) {
return RtpLayerDesc.NO_HEIGHT
}
return firstHeight
}

280
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/RtpLayerDesc.kt Normal file
View File

@@ -0,0 +1,280 @@
/*
* Copyright @ 2018 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
import com.fasterxml.jackson.databind.node.JsonNodeFactory
import com.fasterxml.jackson.databind.node.ObjectNode
import org.jitsi.nlj.transform.node.incoming.BitrateCalculator
import org.jitsi.nlj.util.Bandwidth
import org.jitsi.nlj.util.BitrateTracker
import org.jitsi.nlj.util.DataSize
/**
* Keeps track of its subjective quality index,
* its last stable bitrate and other useful things for adaptivity/routing.
*
* @author George Politis
*/
abstract class RtpLayerDesc(
/**
* The index of this instance's encoding in the source encoding array.
*/
val eid: Int,
/**
* The temporal layer ID of this instance.
*/
val tid: Int,
/**
* The spatial layer ID of this instance.
*/
val sid: Int,
/**
* The max "height" of the bitstream that this instance represents. The actual
* height may be less due to bad network or system load. [NO_HEIGHT] for unknown.
*
* In order to handle portrait-mode video correctly, this should actually be the smaller of
* the bitstream's height or width.
*
* Where possible we sniff the actual height from the RTP packets.
*/
var height: Int,
/**
* The max frame rate (in fps) of the bitstream that this instance
* represents. The actual frame rate may be less due to bad network or
* system load. [NO_FRAME_RATE] for unknown.
*/
var frameRate: Double,
) {
abstract fun copy(height: Int = this.height, tid: Int = this.tid, inherit: Boolean = true): RtpLayerDesc
/**
* The [BitrateTracker] instance used to calculate the receiving bitrate of this RTP layer.
*/
protected var bitrateTracker = BitrateCalculator.createBitrateTracker()
/**
* A second [BitrateTracker] with a longer window, for sources whose bitrate is bursty. It is only created once
* [enableSmoothedBitrate] is called, because it costs memory per layer and an update per packet per layer, and
* only screen sharing sources use it. See [BitrateCalculator.createSmoothedBitrateTracker].
*
* Volatile because it is enabled from the threads which run bandwidth allocation (one per receiver, so possibly
* several at once), but read and updated from the thread which reads packets.
*/
@Volatile
private var smoothedBitrateTracker: BitrateTracker? = null
/**
* The [clock] time at which [smoothedBitrateTracker] was created. Only meaningful while the tracker is non-null.
* Writes to it are made visible by the subsequent volatile write to [smoothedBitrateTracker].
*/
private var smoothedBitrateTrackerCreatedMs: Long = -1
/**
* Start tracking a smoothed bitrate for this layer. Idempotent. Until the tracker created here has a full window
* of history, [getSmoothedBitrate] falls back to [getBitrate], so that a source is not priced at close to zero
* while the longer window fills up.
*/
fun enableSmoothedBitrate(nowMs: Long) {
if (smoothedBitrateTracker == null) {
synchronized(this) {
if (smoothedBitrateTracker == null) {
smoothedBitrateTrackerCreatedMs = nowMs
smoothedBitrateTracker = BitrateCalculator.createSmoothedBitrateTracker()
}
}
}
}
/**
* Stop tracking a smoothed bitrate for this layer, dropping the tracker and its history. Called when the source
* stops being bursty (e.g. a screen share reverting to camera), so that the layer stops paying for the tracker,
* and so that a later re-enable does not price the source from the previous mode's traffic.
*/
fun disableSmoothedBitrate() {
if (smoothedBitrateTracker != null) {
synchronized(this) {
smoothedBitrateTracker = null
}
}
}
/**
* The rate of this layer alone, either over the estimator-matched window or over the longer window used for
* bursty sources. The smoothed rate falls back to the other one while it is not being tracked, and while its
* tracker does not yet have a full window of history (since it always divides by the full window, a young tracker
* would under-report the rate, in the extreme pricing the source at zero at the moment it is enabled).
*/
protected fun layerRate(nowMs: Long, smoothed: Boolean): Bandwidth {
val smoothedTracker = if (smoothed) smoothedBitrateTracker else null
return if (smoothedTracker != null && nowMs - smoothedBitrateTrackerCreatedMs >= smoothedWindowMs) {
smoothedTracker.getRateOverFullWindow(nowMs)
} else {
bitrateTracker.getRate(nowMs)
}
}
var targetBitrate: Bandwidth? = null
/**
* @return the "id" of this layer within this encoding. This is a server-side id and should
* not be confused with any encoding id defined in the client (such as the
* rid).
*/
abstract val layerId: Int
/**
* A local index of this track.
*/
abstract val index: Int
/**
* Inherit a [BitrateTracker] object
*/
internal fun inheritStatistics(tracker: BitrateTracker) {
bitrateTracker = tracker
}
/**
* Inherit another layer description's [BitrateTracker] object.
*/
internal open fun inheritFrom(other: RtpLayerDesc) {
inheritStatistics(other.bitrateTracker)
// Only take the other layer's smoothed tracker if it has one: layer arrays are replaced whenever the
// scalability structure is re-signaled, and dropping an established tracker would restart its window.
other.smoothedBitrateTracker?.let {
smoothedBitrateTrackerCreatedMs = other.smoothedBitrateTrackerCreatedMs
smoothedBitrateTracker = it
}
targetBitrate = other.targetBitrate
}
/**
*
* @param packetSize
* @param nowMs
* @return true if this caused the rate being tracked to transition from zero to nonzero
*/
fun updateBitrate(packetSize: DataSize, nowMs: Long): Boolean {
val wasInactive = hasZeroBitrate(nowMs)
// Update rate stats (this should run after padding termination).
bitrateTracker.update(packetSize, nowMs)
smoothedBitrateTracker?.update(packetSize, nowMs)
return wasInactive && packetSize.bits > 0L
}
/**
* Gets the cumulative bitrate (in bps) of this [RtpLayerDesc] and its dependencies.
*
* @param nowMs
* @return the cumulative bitrate (in bps) of this [RtpLayerDesc] and its dependencies.
*/
abstract fun getBitrate(nowMs: Long): Bandwidth
/**
* Like [getBitrate], but measured over the longer window of [smoothedBitrateTracker]. Used for sources whose
* bitrate is bursty, where the instantaneous rate over-states the bandwidth the source needs.
*/
abstract fun getSmoothedBitrate(nowMs: Long): Bandwidth
/**
* Recursively checks this layer and its dependencies to see if the bitrate is zero.
* Note that unlike [calcBitrate] this does not avoid double-visiting layers; the overhead
* of the hash table is usually more than the cost of any double-visits.
*/
abstract fun hasZeroBitrate(nowMs: Long): Boolean
open fun debugState(): ObjectNode = JsonNodeFactory.instance.objectNode().apply {
put("frameRate", frameRate)
put("height", height)
put("index", index)
put("bitrate_bps", getBitrate(System.currentTimeMillis()).bps)
put("smoothed_bitrate_bps", getSmoothedBitrate(System.currentTimeMillis()).bps)
put("target_bitrate", targetBitrate?.bps ?: 0)
put("indexString", indexString())
}
abstract fun indexString(): String
companion object {
/** The length of the smoothed trackers' window, cached (it is fixed by configuration). */
private val smoothedWindowMs: Long by lazy { BitrateCalculator.smoothedWindowSize.toMillis() }
/**
* The index value that is used to represent that forwarding is suspended.
*/
const val SUSPENDED_INDEX = -1
/**
* The encoding ID value that is used to represent that forwarding is suspended.
*/
const val SUSPENDED_ENCODING_ID = -1
/**
* A value used to designate the absence of height information.
*/
const val NO_HEIGHT = -1
/**
* A value used to designate the absence of frame rate information.
*/
const val NO_FRAME_RATE = -1.0
/**
* Calculate the "index" of a layer based on its encoding, spatial, and temporal ID.
* This is a server-side id and should not be confused with any encoding id defined
* in the client (such as the rid) or the encodingId. This is used by the videobridge's
* adaptive source projection for filtering.
*/
@JvmStatic
fun getIndex(eid: Int, sid: Int, tid: Int): Int {
val e = if (eid < 0) 0 else eid
val s = if (sid < 0) 0 else sid
val t = if (tid < 0) 0 else tid
return (e shl 6) or (s shl 3) or t
}
/**
* Get an encoding ID from a layer index. If the index is [SUSPENDED_INDEX],
* [SUSPENDED_ENCODING_ID] will be returned.
*/
@JvmStatic
fun getEidFromIndex(index: Int) = index shr 6
/**
* Get a spatial ID from a layer index. If the index is [SUSPENDED_INDEX],
* the value is unspecified.
*/
@JvmStatic
fun getSidFromIndex(index: Int) = (index and 0x38) shr 3
/**
* Get a temporal ID from a layer index. If the index is [SUSPENDED_INDEX],
* the value is unspecified.
*/
@JvmStatic
fun getTidFromIndex(index: Int) = index and 0x7
/**
* Get a string description of a layer index.
*/
fun indexString(index: Int): String = if (index == SUSPENDED_INDEX) {
"SUSP"
} else {
"E${getEidFromIndex(index)}S${getSidFromIndex(index)}T${getTidFromIndex(index)}"
}
}
}

79
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/RtpReceiver.kt Normal file
View File

@@ -0,0 +1,79 @@
/*
* Copyright @ 2018 - Present, 8x8 Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
import com.fasterxml.jackson.databind.node.ObjectNode
import org.jitsi.nlj.rtp.LossListener
import org.jitsi.nlj.srtp.SrtpTransformers
import org.jitsi.nlj.stats.EndpointConnectionStats
import org.jitsi.nlj.stats.RtpReceiverStats
import org.jitsi.nlj.util.Bandwidth
abstract class RtpReceiver :
StatsKeepingPacketHandler(),
EventHandler,
Stoppable,
EndpointConnectionStats.EndpointConnectionStatsListener {
/**
* The handler which will be invoked for each RTP/RTCP packet received
* by this receiver (after it has gone through the receiver's
* input chain).
*/
abstract var packetHandler: PacketHandler?
/**
* Enqueue an incoming packet to be processed
*/
abstract fun enqueuePacket(p: PacketInfo)
abstract fun debugState(mode: DebugStateMode): ObjectNode
/**
* Set the SRTP transformers to be used for RTP/RTCP encryption and decryption
*/
abstract fun setSrtpTransformers(srtpTransformers: SrtpTransformers)
abstract fun getStats(): RtpReceiverStats
abstract fun tearDown()
abstract fun isReceivingAudio(): Boolean
abstract fun isReceivingVideo(): Boolean
abstract fun addLossListener(lossListener: LossListener)
abstract fun setFeature(feature: Features, enabled: Boolean)
abstract fun isFeatureEnabled(feature: Features): Boolean
/**
* Forcibly mute or unmute the incoming audio stream
*/
abstract fun forceMuteAudio(shouldMute: Boolean)
abstract fun forceMuteVideo(shouldMute: Boolean)
}
interface RtpReceiverEventHandler {
/**
* We received an audio level indication from the remote endpoint.
*/
fun audioLevelReceived(sourceSsrc: Long, level: Long): Boolean = false
/**
* The estimation of the available send bandwidth changed.
*/
fun bandwidthEstimationChanged(newValue: Bandwidth) {}
}

355
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/RtpReceiverImpl.kt Normal file
View File

@@ -0,0 +1,355 @@
/*
* Copyright @ 2018 - Present, 8x8 Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
import com.fasterxml.jackson.databind.node.JsonNodeFactory
import com.fasterxml.jackson.databind.node.ObjectNode
import org.jitsi.config.JitsiConfig
import org.jitsi.metaconfig.config
import org.jitsi.metaconfig.from
import org.jitsi.nlj.rtcp.CompoundRtcpParser
import org.jitsi.nlj.rtcp.RembHandler
import org.jitsi.nlj.rtcp.RtcpEventNotifier
import org.jitsi.nlj.rtcp.RtcpRrGenerator
import org.jitsi.nlj.rtp.AudioRtpPacket
import org.jitsi.nlj.rtp.LossListener
import org.jitsi.nlj.rtp.TransportCcEngine
import org.jitsi.nlj.rtp.VideoRtpPacket
import org.jitsi.nlj.srtp.SrtpTransformers
import org.jitsi.nlj.stats.RtpReceiverStats
import org.jitsi.nlj.transform.NodeDebugStateVisitor
import org.jitsi.nlj.transform.NodeEventVisitor
import org.jitsi.nlj.transform.NodeTeardownVisitor
import org.jitsi.nlj.transform.node.ConsumerNode
import org.jitsi.nlj.transform.node.Node
import org.jitsi.nlj.transform.node.PacketLossConfig
import org.jitsi.nlj.transform.node.PacketLossNode
import org.jitsi.nlj.transform.node.PacketStreamStatsNode
import org.jitsi.nlj.transform.node.RtpParser
import org.jitsi.nlj.transform.node.SrtcpDecryptNode
import org.jitsi.nlj.transform.node.SrtpDecryptNode
import org.jitsi.nlj.transform.node.ToggleablePcapWriter
import org.jitsi.nlj.transform.node.incoming.AudioLevelReader
import org.jitsi.nlj.transform.node.incoming.BitrateCalculator
import org.jitsi.nlj.transform.node.incoming.DiscardableDiscarder
import org.jitsi.nlj.transform.node.incoming.DuplicateTermination
import org.jitsi.nlj.transform.node.incoming.IncomingStatisticsTracker
import org.jitsi.nlj.transform.node.incoming.PaddingTermination
import org.jitsi.nlj.transform.node.incoming.RemoteBandwidthEstimator
import org.jitsi.nlj.transform.node.incoming.RetransmissionRequesterNode
import org.jitsi.nlj.transform.node.incoming.RtcpTermination
import org.jitsi.nlj.transform.node.incoming.RtxHandler
import org.jitsi.nlj.transform.node.incoming.TccGeneratorNode
import org.jitsi.nlj.transform.node.incoming.VideoBitrateCalculator
import org.jitsi.nlj.transform.node.incoming.VideoMuteNode
import org.jitsi.nlj.transform.node.incoming.VideoParser
import org.jitsi.nlj.transform.node.incoming.VideoQualityLayerLookup
import org.jitsi.nlj.transform.node.incoming.VlaReaderNode
import org.jitsi.nlj.transform.packetPath
import org.jitsi.nlj.transform.pipeline
import org.jitsi.nlj.util.Bandwidth
import org.jitsi.nlj.util.BufferPool
import org.jitsi.nlj.util.PacketInfoQueue
import org.jitsi.nlj.util.PacketPredicate
import org.jitsi.nlj.util.ReadOnlyStreamInformationStore
import org.jitsi.rtp.Packet
import org.jitsi.rtp.extensions.looksLikeRtcp
import org.jitsi.rtp.extensions.looksLikeRtp
import org.jitsi.rtp.rtcp.RtcpPacket
import org.jitsi.utils.logging.DiagnosticContext
import org.jitsi.utils.logging2.Logger
import org.jitsi.utils.logging2.cdebug
import org.jitsi.utils.logging2.createChildLogger
import org.jitsi.utils.queue.CountingErrorHandler
import java.util.concurrent.ExecutorService
import java.util.concurrent.ScheduledExecutorService
class RtpReceiverImpl @JvmOverloads constructor(
val id: String,
/**
* A function to be used when the receiver wants to send RTCP packets to the
* participant it's receiving data from (NACK packets, for example)
*/
private val rtcpSender: (RtcpPacket) -> Unit = {},
rtcpEventNotifier: RtcpEventNotifier,
/**
* The executor this class will use for its primary work (i.e. critical path
* packet processing). This [RtpReceiver] will execute a blocking queue read
* on this executor.
*/
private val executor: ExecutorService,
/**
* A [ScheduledExecutorService] which can be used for less important
* background tasks, or tasks that need to execute at some fixed delay/rate
*/
private val backgroundExecutor: ScheduledExecutorService,
streamInformationStore: ReadOnlyStreamInformationStore,
private val eventHandler: RtpReceiverEventHandler,
parentLogger: Logger,
diagnosticContext: DiagnosticContext = DiagnosticContext()
) : RtpReceiver() {
private val logger = createChildLogger(parentLogger)
private var running: Boolean = true
private val inputTreeRoot: Node
private val incomingPacketQueue = PacketInfoQueue(
"rtp-receiver-incoming-packet-queue",
executor,
this::handleIncomingPacket,
queueSize
)
private val srtpDecryptWrapper = SrtpDecryptNode()
private val srtcpDecryptWrapper = SrtcpDecryptNode()
private val tccGenerator = TccGeneratorNode(rtcpSender, streamInformationStore, logger)
private val remoteBandwidthEstimator = RemoteBandwidthEstimator(streamInformationStore, logger, diagnosticContext)
private val audioLevelReader = AudioLevelReader(streamInformationStore).apply {
audioLevelListener = object : AudioLevelListener {
override fun onLevelReceived(sourceSsrc: Long, level: Long): Boolean =
eventHandler.audioLevelReceived(sourceSsrc, level)
}
}
private val videoMuteNode = VideoMuteNode()
private val silenceDiscarder = DiscardableDiscarder("Silence discarder", false)
private val paddingOnlyDiscarder = DiscardableDiscarder("Padding-only discarder", true)
private val statsTracker = IncomingStatisticsTracker(streamInformationStore)
private val packetStreamStats = PacketStreamStatsNode(diagnosticContext, "receive")
private val rtcpRrGenerator = RtcpRrGenerator(backgroundExecutor, rtcpSender, statsTracker) {
remoteBandwidthEstimator.createRemb()?.let {
listOf(it)
} ?: emptyList()
}
private val rtcpTermination = RtcpTermination(rtcpEventNotifier, logger)
private val retransmissionRequester = RetransmissionRequesterNode(rtcpSender, backgroundExecutor, logger)
private val rembHandler = RembHandler(streamInformationStore, logger).apply {
addListener(object : TransportCcEngine.BandwidthListener {
override fun bandwidthEstimationChanged(newValue: Bandwidth) {
eventHandler.bandwidthEstimationChanged(newValue)
}
})
}
private val toggleablePcapWriter = ToggleablePcapWriter(logger, "$id-rx")
private val videoBitrateCalculator = VideoBitrateCalculator(parentLogger)
private val audioBitrateCalculator = BitrateCalculator("Audio bitrate calculator")
private val videoParser = VideoParser(streamInformationStore, logger, diagnosticContext)
override fun isReceivingAudio() = audioBitrateCalculator.active
override fun isReceivingVideo() = videoBitrateCalculator.active
override fun addLossListener(lossListener: LossListener) {
tccGenerator.addLossListener(lossListener)
}
companion object {
var queueErrorCounter = CountingErrorHandler()
private const val PACKET_QUEUE_ENTRY_EVENT = "Entered RTP receiver incoming queue"
private const val PACKET_QUEUE_EXIT_EVENT = "Exited RTP receiver incoming queue"
private val queueSize: Int by config("jmt.transceiver.recv.queue-size".from(JitsiConfig.newConfig))
/**
* Configuration for the packet loss to introduce in the receive pipeline (for debugging/testing purposes).
*/
private val packetLossConfig = PacketLossConfig("jmt.debug.packet-loss.incoming")
}
/**
* [packetHandler] will be invoked with RTP packets that have made
* it through the entire receive pipeline. Some external entity should
* assign it to a [PacketHandler] with appropriate logic.
*/
override var packetHandler: PacketHandler? = null
/**
* The [packetHandler] can be re-assigned at any time, but it should maintain
* its place in the receive pipeline. To support both keeping it in the same
* place and allowing it to be re-assigned, we wrap it with this.
*/
private val packetHandlerWrapper = object : ConsumerNode("Input pipeline termination node") {
override fun consume(packetInfo: PacketInfo) {
// When there's no handler set we're effectively dropping packets, so their buffers
// should be returned.
packetHandler?.processPacket(packetInfo) ?: packetDiscarded(packetInfo)
}
override val aggregationKey: String = name
override fun trace(f: () -> Unit) = f.invoke()
}
init {
logger.cdebug { "using executor ${executor.hashCode()}" }
if (packetLossConfig.enabled) {
logger.warn("Will simulate packet loss: $packetLossConfig")
}
rtcpEventNotifier.addRtcpEventListener(rtcpRrGenerator)
rtcpEventNotifier.addRtcpEventListener(rembHandler)
incomingPacketQueue.setErrorHandler(queueErrorCounter)
inputTreeRoot = pipeline {
node(packetStreamStats)
demux("SRTP/SRTCP") {
packetPath {
name = "SRTP"
predicate = PacketPredicate(Packet::looksLikeRtp)
path = pipeline {
node(PacketLossNode(packetLossConfig), condition = { packetLossConfig.enabled })
node(RtpParser(streamInformationStore, logger))
// TODO: temporarily putting the audioLevelReader node here such that we can determine whether
// or not a packet should be discarded before doing SRTP. audioLevelReader has been moved here
// (instead of introducing a different class to read audio levels) to avoid parsing the RTP
// header extensions twice (which is expensive). In the future we will parse and cache the
// header extensions to make this lookup more efficient, at which time we could move
// audioLevelReader back to where it was (in the audio path) and add a new node here which would
// check for different discard conditions (i.e. checking the audio level for silence)
node(audioLevelReader.preDecryptNode)
node(videoMuteNode)
node(srtpDecryptWrapper)
node(tccGenerator)
node(remoteBandwidthEstimator)
// This reads audio levels from packets that use cryptex. TODO: should it go in the Audio path?
node(audioLevelReader.postDecryptNode)
node(toggleablePcapWriter.newObserverNode(outbound = false, suffix = "rx_rtp"))
node(statsTracker)
node(PaddingTermination(logger))
demux("Media Type") {
packetPath {
name = "Audio"
predicate = PacketPredicate { it is AudioRtpPacket }
path = pipeline {
node(silenceDiscarder)
node(audioBitrateCalculator)
node(packetHandlerWrapper)
}
}
packetPath {
name = "Video"
predicate = PacketPredicate { it is VideoRtpPacket }
path = pipeline {
node(RtxHandler(streamInformationStore, logger))
node(DuplicateTermination())
node(retransmissionRequester)
node(paddingOnlyDiscarder)
node(videoParser)
node(VideoQualityLayerLookup(logger))
node(videoBitrateCalculator)
node(VlaReaderNode(streamInformationStore, logger))
node(packetHandlerWrapper)
}
}
}
}
}
packetPath {
name = "SRTCP"
predicate = PacketPredicate(Packet::looksLikeRtcp)
path = pipeline {
node(srtcpDecryptWrapper)
node(toggleablePcapWriter.newObserverNode(outbound = false, suffix = "rx_rtcp"))
node(CompoundRtcpParser(logger))
node(rtcpTermination)
node(packetHandlerWrapper)
}
}
}
}
}
private fun handleIncomingPacket(packet: PacketInfo): Boolean = if (running) {
packet.addEvent(PACKET_QUEUE_EXIT_EVENT)
processPacket(packet)
true
} else {
BufferPool.returnBuffer(packet.packet.buffer)
false
}
override fun doProcessPacket(packetInfo: PacketInfo) = inputTreeRoot.processPacket(packetInfo)
override fun debugState(mode: DebugStateMode): ObjectNode = JsonNodeFactory.instance.objectNode().apply {
NodeDebugStateVisitor(this, mode).visit(inputTreeRoot)
}
override fun enqueuePacket(p: PacketInfo) {
// logger.cinfo { "Receiver $id enqueing data" }
if (running) {
p.addEvent(PACKET_QUEUE_ENTRY_EVENT)
incomingPacketQueue.add(p)
} else {
BufferPool.returnBuffer(p.packet.buffer)
}
}
override fun setSrtpTransformers(srtpTransformers: SrtpTransformers) {
srtpDecryptWrapper.transformer = srtpTransformers.srtpDecryptTransformer
srtcpDecryptWrapper.transformer = srtpTransformers.srtcpDecryptTransformer
}
override fun handleEvent(event: Event) {
NodeEventVisitor(event).visit(inputTreeRoot)
}
override fun getStats(): RtpReceiverStats = RtpReceiverStats(
incomingStats = statsTracker.getSnapshot(),
packetStreamStats = packetStreamStats.snapshot(),
videoParserStats = videoParser.getStats()
)
override fun forceMuteAudio(shouldMute: Boolean) {
audioLevelReader.forceMute = shouldMute
}
override fun forceMuteVideo(shouldMute: Boolean) {
videoMuteNode.forceMute = shouldMute
}
override fun setFeature(feature: Features, enabled: Boolean) {
when (feature) {
Features.TRANSCEIVER_PCAP_DUMP -> {
if (enabled) {
toggleablePcapWriter.enable()
} else {
toggleablePcapWriter.disable()
}
}
}
}
override fun isFeatureEnabled(feature: Features): Boolean = when (feature) {
Features.TRANSCEIVER_PCAP_DUMP -> toggleablePcapWriter.isEnabled()
}
override fun stop() {
running = false
rtcpRrGenerator.running = false
retransmissionRequester.stop()
}
override fun tearDown() {
logger.debug("Tearing down")
NodeTeardownVisitor().visit(inputTreeRoot)
incomingPacketQueue.close()
toggleablePcapWriter.disable()
}
override fun onRttUpdate(newRttMs: Double) {
remoteBandwidthEstimator.onRttUpdate(newRttMs)
}
}

59
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/RtpSender.kt Normal file
View File

@@ -0,0 +1,59 @@
/*
* Copyright @ 2018 - Present, 8x8 Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
import com.fasterxml.jackson.databind.node.ObjectNode
import org.jitsi.nlj.rtp.LossListener
import org.jitsi.nlj.rtp.RtpExtensionType
import org.jitsi.nlj.rtp.TransportCcEngine
import org.jitsi.nlj.srtp.SrtpTransformers
import org.jitsi.nlj.stats.EndpointConnectionStats
import org.jitsi.nlj.stats.PacketStreamStats
import org.jitsi.nlj.transform.node.outgoing.OutgoingStatisticsSnapshot
/**
* Not an 'RtpSender' in the sense that it sends only RTP (and not
* RTCP) but in the sense of a webrtc 'RTCRTPSender' which handles
* all RTP and RTP control packets.
*/
abstract class RtpSender :
StatsKeepingPacketHandler(),
EventHandler,
Stoppable,
EndpointConnectionStats.EndpointConnectionStatsListener {
abstract fun sendProbing(mediaSsrcs: Collection<Long>, numBytes: Int): Int
abstract fun onOutgoingPacket(handler: PacketHandler)
abstract fun setSrtpTransformers(srtpTransformers: SrtpTransformers)
abstract fun getStreamStats(): OutgoingStatisticsSnapshot
abstract fun getPacketStreamStats(): PacketStreamStats.Snapshot
abstract fun addBandwidthListener(listener: TransportCcEngine.BandwidthListener)
abstract fun removeBandwidthListener(listener: TransportCcEngine.BandwidthListener)
abstract fun getTransportCcEngineStats(): TransportCcEngine.StatisticsSnapshot
abstract fun requestKeyframe(requesterID: String?, mediaSsrc: Long? = null)
abstract fun addLossListener(lossListener: LossListener)
abstract fun setFeature(feature: Features, enabled: Boolean)
abstract fun isFeatureEnabled(feature: Features): Boolean
abstract fun tearDown()
abstract fun addRtpExtensionToRetain(extensionType: RtpExtensionType)
/**
* An optional function to be executed for each RTP packet, as the first step of the send pipeline.
*/
var preProcesor: ((PacketInfo) -> PacketInfo?)? = null
abstract fun debugState(mode: DebugStateMode): ObjectNode
}

384
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/RtpSenderImpl.kt Normal file
View File

@@ -0,0 +1,384 @@
/*
* Copyright @ 2018 - Present, 8x8 Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
import com.fasterxml.jackson.databind.node.JsonNodeFactory
import com.fasterxml.jackson.databind.node.ObjectNode
import org.jitsi.config.JitsiConfig
import org.jitsi.metaconfig.config
import org.jitsi.metaconfig.from
import org.jitsi.nlj.rtcp.KeyframeRequester
import org.jitsi.nlj.rtcp.NackHandler
import org.jitsi.nlj.rtcp.RtcpEventNotifier
import org.jitsi.nlj.rtcp.RtcpSrUpdater
import org.jitsi.nlj.rtp.ClassicTransportCcEngine
import org.jitsi.nlj.rtp.LossListener
import org.jitsi.nlj.rtp.RtpExtensionType
import org.jitsi.nlj.rtp.TransportCcEngine
import org.jitsi.nlj.rtp.bandwidthestimation.BandwidthEstimatorConfig
import org.jitsi.nlj.rtp.bandwidthestimation.BandwidthEstimatorEngine
import org.jitsi.nlj.rtp.bandwidthestimation.GoogleCcEstimator
import org.jitsi.nlj.rtp.bandwidthestimation2.GoogCcTransportCcEngine
import org.jitsi.nlj.srtp.SrtpTransformers
import org.jitsi.nlj.transform.NodeDebugStateVisitor
import org.jitsi.nlj.transform.NodeEventVisitor
import org.jitsi.nlj.transform.NodeTeardownVisitor
import org.jitsi.nlj.transform.node.AudioRedHandler
import org.jitsi.nlj.transform.node.ConsumerNode
import org.jitsi.nlj.transform.node.Node
import org.jitsi.nlj.transform.node.ObserverNode
import org.jitsi.nlj.transform.node.PacketCacher
import org.jitsi.nlj.transform.node.PacketLossConfig
import org.jitsi.nlj.transform.node.PacketLossNode
import org.jitsi.nlj.transform.node.PacketStreamStatsNode
import org.jitsi.nlj.transform.node.PluggableTransformerNode
import org.jitsi.nlj.transform.node.SrtcpEncryptNode
import org.jitsi.nlj.transform.node.SrtpEncryptNode
import org.jitsi.nlj.transform.node.ToggleablePcapWriter
import org.jitsi.nlj.transform.node.outgoing.AbsSendTime
import org.jitsi.nlj.transform.node.outgoing.HeaderExtEncoder
import org.jitsi.nlj.transform.node.outgoing.HeaderExtStripper
import org.jitsi.nlj.transform.node.outgoing.MidStamper
import org.jitsi.nlj.transform.node.outgoing.OutgoingStatisticsTracker
import org.jitsi.nlj.transform.node.outgoing.ProbingDataSender
import org.jitsi.nlj.transform.node.outgoing.RetransmissionSender
import org.jitsi.nlj.transform.node.outgoing.SentRtcpStats
import org.jitsi.nlj.transform.node.outgoing.TccSeqNumTagger
import org.jitsi.nlj.transform.pipeline
import org.jitsi.nlj.util.BufferPool
import org.jitsi.nlj.util.PacketInfoQueue
import org.jitsi.nlj.util.StreamInformationStore
import org.jitsi.nlj.util.appendAll
import org.jitsi.rtp.rtcp.RtcpPacket
import org.jitsi.utils.MediaType
import org.jitsi.utils.logging.DiagnosticContext
import org.jitsi.utils.logging2.Logger
import org.jitsi.utils.logging2.cdebug
import org.jitsi.utils.logging2.createChildLogger
import org.jitsi.utils.queue.CountingErrorHandler
import java.time.Duration
import java.util.concurrent.ExecutorService
import java.util.concurrent.ScheduledExecutorService
class RtpSenderImpl(
val id: String,
private val rtcpEventNotifier: RtcpEventNotifier,
/**
* The executor this class will use for its primary work (i.e. critical path
* packet processing). This [RtpSender] will execute a blocking queue read
* on this executor.
*/
val executor: ExecutorService,
/**
* A [ScheduledExecutorService] which can be used for less important
* background tasks, or tasks that need to execute at some fixed delay/rate
*/
val backgroundExecutor: ScheduledExecutorService,
private val streamInformationStore: StreamInformationStore,
parentLogger: Logger,
diagnosticContext: DiagnosticContext = DiagnosticContext(),
/**
* Resolves the sdes:mid to stamp on an outgoing packet given its (rewritten) SSRC, or null to not stamp a mid.
* Used under SSRC rewriting for mid-based demuxing on the receiving client.
*/
private val getMidBySsrc: (Long) -> String? = { null }
) : RtpSender() {
private val logger = createChildLogger(parentLogger)
private val outgoingRtpRoot: Node
private val outgoingRtxRoot: Node
private val outgoingRtcpRoot: Node
private val incomingPacketQueue = PacketInfoQueue(
"rtp-sender-incoming-packet-queue",
executor,
this::handlePacket,
queueSize
)
var running = true
private var localVideoSsrc: Long? = null
private var localAudioSsrc: Long? = null
// TODO(brian): this is changed to a handler instead of a queue because we want to use
// a PacketQueue, and the handler for a PacketQueue must be set at the time of creation.
// since we want the handler to be another entity (something in jvb) we just use
// a generic handler here and then the bridge can put it into its PacketQueue and have
// its handler (likely in another thread) grab the packet and send it out
private var outgoingPacketHandler: PacketHandler? = null
private val transportCcEngine: TransportCcEngine =
when (BandwidthEstimatorConfig.engine) {
BandwidthEstimatorEngine.GoogleCc ->
ClassicTransportCcEngine(GoogleCcEstimator(diagnosticContext, logger), logger)
BandwidthEstimatorEngine.GoogleCc2 ->
GoogCcTransportCcEngine(diagnosticContext, logger, backgroundExecutor, { dataSize, probingData ->
return@GoogCcTransportCcEngine probingDataSender.sendProbing(
null,
dataSize.bytes.toInt(),
probingData
)
})
}.also { it.start() }
private val srtpEncryptWrapper = SrtpEncryptNode()
private val srtcpEncryptWrapper = SrtcpEncryptNode()
private val toggleablePcapWriter = ToggleablePcapWriter(logger, "$id-tx")
private val outgoingPacketCache = PacketCacher()
private val headerExtensionStripper = HeaderExtStripper(streamInformationStore)
private val absSendTime = AbsSendTime(streamInformationStore)
private val statsTracker = OutgoingStatisticsTracker(diagnosticContext)
private val packetStreamStats = PacketStreamStatsNode(diagnosticContext, "send")
private val rtcpSrUpdater = RtcpSrUpdater(statsTracker)
private val keyframeRequester = KeyframeRequester(streamInformationStore, logger)
private val probingDataSender: ProbingDataSender
private val nackHandler: NackHandler
private val outputPipelineTerminationNode = object : ConsumerNode("Output pipeline termination node") {
override fun consume(packetInfo: PacketInfo) {
// While there's no handler set we're effectively dropping packets, so their buffers
// should be returned.
outgoingPacketHandler?.processPacket(packetInfo) ?: packetDiscarded(packetInfo)
}
override val aggregationKey = name
override fun trace(f: () -> Unit) = f.invoke()
}
init {
logger.cdebug { "Sender $id using executor ${executor.hashCode()}" }
if (packetLossConfig.enabled) {
logger.warn("Will simulate packet loss: $packetLossConfig")
}
incomingPacketQueue.setErrorHandler(queueErrorCounter)
outgoingRtpRoot = pipeline {
node(PluggableTransformerNode("RTP pre-processor") { preProcesor })
node(AudioRedHandler(streamInformationStore, logger))
node(headerExtensionStripper)
node(outgoingPacketCache)
node(absSendTime)
// Placed right after absSendTime, which is where the RTX pipeline joins the main pipeline, so that both
// media packets and their retransmissions get the mid stamped.
node(MidStamper(streamInformationStore, getMidBySsrc))
node(statsTracker)
node(TccSeqNumTagger(transportCcEngine, streamInformationStore))
node(HeaderExtEncoder(streamInformationStore, logger))
node(toggleablePcapWriter.newObserverNode(outbound = true, suffix = "tx_rtp"))
node(srtpEncryptWrapper)
node(packetStreamStats.createNewNode())
node(PacketLossNode(packetLossConfig), condition = { packetLossConfig.enabled })
node(outputPipelineTerminationNode)
}
outgoingRtxRoot = pipeline {
node(RetransmissionSender(streamInformationStore, logger))
// We want RTX packets to hook into the main RTP pipeline starting at AbsSendTime
node(absSendTime)
}
nackHandler = NackHandler(outgoingPacketCache.getPacketCache(), outgoingRtxRoot, logger)
rtcpEventNotifier.addRtcpEventListener(nackHandler)
rtcpEventNotifier.addRtcpEventListener(transportCcEngine)
// TODO: are we setting outgoing rtcp sequence numbers correctly? just add a simple node here to rewrite them
outgoingRtcpRoot = pipeline {
node(PluggableTransformerNode("RTCP pre-processor") { preProcesor })
node(keyframeRequester)
node(SentRtcpStats())
// TODO(brian): not sure this is a great idea. it works as a catch-call but can also be error-prone
// (i found i was accidentally clobbering the sender ssrc for SRs which caused issues). I think
// it'd be better to notify everything creating RTCP the bridge SSRCs and then everything should be
// responsible for setting it themselves
simpleNode("RTCP sender ssrc setter") { packetInfo ->
val senderSsrc = localVideoSsrc ?: return@simpleNode packetInfo
val rtcpPacket = packetInfo.packetAs<RtcpPacket>()
if (rtcpPacket.senderSsrc == 0L) {
rtcpPacket.senderSsrc = senderSsrc
}
packetInfo
}
node(rtcpSrUpdater)
node(toggleablePcapWriter.newObserverNode(outbound = true, suffix = "tx_rtcp"))
node(object : ObserverNode("RTCP sent notifier") {
override fun observe(packetInfo: PacketInfo) {
val packet = packetInfo.packet
if (packet is RtcpPacket) {
rtcpEventNotifier.notifyRtcpSent(packet)
}
}
override fun trace(f: () -> Unit) {}
})
node(srtcpEncryptWrapper)
node(packetStreamStats.createNewNode())
node(PacketLossNode(packetLossConfig), condition = { packetLossConfig.enabled })
node(outputPipelineTerminationNode)
}
probingDataSender = ProbingDataSender(
packetCache = outgoingPacketCache.getPacketCache(),
rtxDataSender = outgoingRtxRoot,
garbageDataSender = absSendTime,
diagnosticContext = diagnosticContext,
streamInformationStore = streamInformationStore,
parentLogger = logger
)
}
override fun onRttUpdate(newRttMs: Double) {
nackHandler.onRttUpdate(newRttMs)
keyframeRequester.onRttUpdate(newRttMs)
transportCcEngine.onRttUpdate(Duration.ofNanos((newRttMs * 1e6).toLong()))
}
/**
* Insert packets into the incoming packet queue
*/
override fun doProcessPacket(packetInfo: PacketInfo) {
if (running) {
packetInfo.addEvent(PACKET_QUEUE_ENTRY_EVENT)
incomingPacketQueue.add(packetInfo)
} else {
BufferPool.returnBuffer(packetInfo.packet.buffer)
}
}
override fun sendProbing(mediaSsrcs: Collection<Long>, numBytes: Int): Int =
probingDataSender.sendProbing(mediaSsrcs, numBytes, null)
override fun onOutgoingPacket(handler: PacketHandler) {
outgoingPacketHandler = handler
}
override fun setSrtpTransformers(srtpTransformers: SrtpTransformers) {
srtpEncryptWrapper.transformer = srtpTransformers.srtpEncryptTransformer
srtcpEncryptWrapper.transformer = srtpTransformers.srtcpEncryptTransformer
}
override fun requestKeyframe(requesterID: String?, mediaSsrc: Long?) {
keyframeRequester.requestKeyframe(requesterID, mediaSsrc)
}
override fun addLossListener(lossListener: LossListener) {
transportCcEngine.addLossListener(lossListener)
}
override fun setFeature(feature: Features, enabled: Boolean) {
when (feature) {
Features.TRANSCEIVER_PCAP_DUMP -> {
if (enabled) {
toggleablePcapWriter.enable()
} else {
toggleablePcapWriter.disable()
}
}
}
}
override fun isFeatureEnabled(feature: Features): Boolean = when (feature) {
Features.TRANSCEIVER_PCAP_DUMP -> toggleablePcapWriter.isEnabled()
}
/**
* Handles packets that have gone through the incoming queue and sends them
* through the sender pipeline
*/
private fun handlePacket(packetInfo: PacketInfo): Boolean = if (running) {
packetInfo.addEvent(PACKET_QUEUE_EXIT_EVENT)
val root = when (packetInfo.packet) {
is RtcpPacket -> outgoingRtcpRoot
else -> outgoingRtpRoot
}
root.processPacket(packetInfo)
true
} else {
BufferPool.returnBuffer(packetInfo.packet.buffer)
false
}
override fun getStreamStats() = statsTracker.getSnapshot()
override fun getPacketStreamStats() = packetStreamStats.snapshot()
override fun getTransportCcEngineStats() = transportCcEngine.getStatistics()
override fun addBandwidthListener(listener: TransportCcEngine.BandwidthListener) =
transportCcEngine.addBandwidthListener(listener)
override fun removeBandwidthListener(listener: TransportCcEngine.BandwidthListener) =
transportCcEngine.removeBandwidthListener(listener)
override fun handleEvent(event: Event) {
when (event) {
is SetLocalSsrcEvent -> {
when (event.mediaType) {
MediaType.VIDEO -> localVideoSsrc = event.ssrc
MediaType.AUDIO -> localAudioSsrc = event.ssrc
else -> {}
}
}
}
NodeEventVisitor(event).reverseVisit(outputPipelineTerminationNode)
probingDataSender.handleEvent(event)
}
override fun debugState(mode: DebugStateMode): ObjectNode = JsonNodeFactory.instance.objectNode().apply {
if (mode == DebugStateMode.FULL) {
appendAll(super.getNodeStats().toJson())
set<ObjectNode>("packet_queue", incomingPacketQueue.debugState)
put("running", running)
}
appendAll(nackHandler.getNodeStats().toJson())
set<ObjectNode>("probing_data_sender", probingDataSender.debugState(mode))
put("local_video_ssrc", localVideoSsrc?.toString() ?: "null")
put("local_audio_ssrc", localAudioSsrc?.toString() ?: "null")
set<ObjectNode>("transport_cc_engine", transportCcEngine.getStatistics().toJson())
NodeDebugStateVisitor(this, mode).reverseVisit(outputPipelineTerminationNode)
}
override fun stop() {
running = false
}
override fun tearDown() {
logger.debug("Tearing down")
NodeTeardownVisitor().reverseVisit(outputPipelineTerminationNode)
incomingPacketQueue.close()
toggleablePcapWriter.disable()
}
override fun addRtpExtensionToRetain(extensionType: RtpExtensionType) {
headerExtensionStripper.addRtpExtensionToRetain(extensionType)
}
companion object {
var queueErrorCounter = CountingErrorHandler()
private const val PACKET_QUEUE_ENTRY_EVENT = "Entered RTP sender incoming queue"
private const val PACKET_QUEUE_EXIT_EVENT = "Exited RTP sender incoming queue"
private val queueSize: Int by config("jmt.transceiver.send.queue-size".from(JitsiConfig.newConfig))
/**
* Configuration for the packet loss to introduce in the send pipeline (for debugging/testing purposes).
*/
private val packetLossConfig = PacketLossConfig("jmt.debug.packet-loss.outgoing")
}
}

20
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/Stoppable.kt Normal file
View File

@@ -0,0 +1,20 @@
/*
* Copyright @ 2018 - Present, 8x8 Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
interface Stoppable {
fun stop()
}

397
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/Transceiver.kt Normal file
View File

@@ -0,0 +1,397 @@
/*
* Copyright @ 2018 - Present, 8x8 Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
import com.fasterxml.jackson.databind.node.JsonNodeFactory
import com.fasterxml.jackson.databind.node.ObjectNode
import org.jitsi.nlj.format.PayloadType
import org.jitsi.nlj.rtcp.RtcpEventNotifier
import org.jitsi.nlj.rtp.RtpExtension
import org.jitsi.nlj.rtp.RtpExtensionType
import org.jitsi.nlj.rtp.TransportCcEngine
import org.jitsi.nlj.srtp.SrtpTransformers
import org.jitsi.nlj.srtp.SrtpUtil
import org.jitsi.nlj.srtp.TlsRole
import org.jitsi.nlj.stats.EndpointConnectionStats
import org.jitsi.nlj.stats.PacketIOActivity
import org.jitsi.nlj.stats.TransceiverStats
import org.jitsi.nlj.util.Bandwidth
import org.jitsi.nlj.util.LocalSsrcAssociation
import org.jitsi.nlj.util.ReadOnlyStreamInformationStore
import org.jitsi.nlj.util.SsrcAssociation
import org.jitsi.nlj.util.StreamInformationStoreImpl
import org.jitsi.utils.MediaType
import org.jitsi.utils.logging.DiagnosticContext
import org.jitsi.utils.logging2.Logger
import org.jitsi.utils.logging2.cdebug
import org.jitsi.utils.logging2.cinfo
import org.jitsi.utils.logging2.createChildLogger
import java.time.Clock
import java.util.concurrent.ExecutorService
import java.util.concurrent.ScheduledExecutorService
// This is an API class, so its usages will largely be outside of this library
/**
* Handles all packets (incoming and outgoing) for a particular stream.
* (TODO: 'stream' defined as what, exactly, here?)
* Handles the DTLS negotiation
*
* Incoming packets should be written via [handleIncomingPacket]. Outgoing
* packets are put in [outgoingQueue] (and should be read by something)
* TODO: maybe we want to have this 'push' the outgoing packets somewhere
* else instead (then we could have all senders push to a single queue and
* have the one thread just read from the queue and send, rather than that thread
* having to read from a bunch of individual queues)
*/
@Suppress("unused")
class Transceiver(
private val id: String,
receiverExecutor: ExecutorService,
senderExecutor: ExecutorService,
/**
* A [ScheduledExecutorService] which can be used for less important
* background tasks, or tasks that need to execute at some fixed delay/rate
*/
backgroundExecutor: ScheduledExecutorService,
diagnosticContext: DiagnosticContext,
parentLogger: Logger,
/**
* The handler for events coming out of this [Transceiver]. Note that these events are fired synchronously,
* potentially in one of the threads in the CPU pool, and it is up to the user of the library to handle the
* transition to another thread if necessary.
*/
private val eventHandler: TransceiverEventHandler,
private val clock: Clock = Clock.systemUTC(),
/**
* Resolves the sdes:mid to stamp on an outgoing packet given its (rewritten) SSRC, or null to not stamp a mid.
*/
getMidBySsrc: (Long) -> String? = { null }
) : Stoppable {
private val logger = createChildLogger(parentLogger)
val packetIOActivity = PacketIOActivity()
private val endpointConnectionStats = EndpointConnectionStats(logger)
private val streamInformationStore = StreamInformationStoreImpl()
val readOnlyStreamInformationStore: ReadOnlyStreamInformationStore = streamInformationStore
/**
* A central place to subscribe to be notified on the reception or transmission of RTCP packets for
* this transceiver. This is intended to be used by internal entities: mainly logic for things like generating
* SRs and RRs and calculating RTT. Since it is used for both send and receive, it is held here and passed to
* the sender and receive so each can push or subscribe to updates.
*/
val rtcpEventNotifier = RtcpEventNotifier()
private var mediaSources = MediaSources()
var srtpTransformers: SrtpTransformers? = null
/** Whether the srtpTransformers were created inside this object, or passed in externally.
* For external transformers it's the external owner's responsibility to close them.
*/
var internalTransformers = false
/**
* Whether this [Transceiver] is receiving audio from the remote endpoint.
*/
fun isReceivingAudio(): Boolean = rtpReceiver.isReceivingAudio()
/**
* Whether this [Transceiver] is receiving video from the remote endpoint.
*/
fun isReceivingVideo(): Boolean = rtpReceiver.isReceivingVideo()
val rtpSender: RtpSender = RtpSenderImpl(
id,
rtcpEventNotifier,
senderExecutor,
backgroundExecutor,
streamInformationStore,
logger,
diagnosticContext,
getMidBySsrc
)
private val rtpReceiver: RtpReceiver =
RtpReceiverImpl(
id,
{ rtcpPacket ->
if (rtcpPacket.length >= 1500) {
logger.warn(
"Sending large locally-generated RTCP packet of size ${rtcpPacket.length}, " +
"first packet of type ${rtcpPacket.packetType} rc ${rtcpPacket.reportCount}."
)
}
rtpSender.processPacket(PacketInfo(rtcpPacket))
},
rtcpEventNotifier,
receiverExecutor,
backgroundExecutor,
streamInformationStore,
eventHandler,
logger,
diagnosticContext
)
init {
rtpSender.addBandwidthListener(
object : TransportCcEngine.BandwidthListener {
override fun bandwidthEstimationChanged(newValue: Bandwidth) {
eventHandler.bandwidthEstimationChanged(newValue)
}
}
)
rtpReceiver.addLossListener(endpointConnectionStats.incomingLossTracker)
rtpSender.addLossListener(endpointConnectionStats.outgoingLossTracker)
rtcpEventNotifier.addRtcpEventListener(endpointConnectionStats)
endpointConnectionStats.addListener(rtpSender)
endpointConnectionStats.addListener(rtpReceiver)
}
/**
* Handle an incoming [PacketInfo] (that is, a packet received by the endpoint
* this transceiver is associated with) to be processed by the receiver pipeline.
*/
fun handleIncomingPacket(p: PacketInfo) {
packetIOActivity.lastRtpPacketReceivedInstant = clock.instant()
rtpReceiver.enqueuePacket(p)
}
/**
* Send packets to the endpoint this transceiver is associated with by
* passing them out the sender's outgoing pipeline
*/
fun sendPacket(packetInfo: PacketInfo) {
packetIOActivity.lastRtpPacketSentInstant = clock.instant()
rtpSender.processPacket(packetInfo)
}
fun sendProbing(mediaSsrcs: Collection<Long>, numBytes: Int): Int = rtpSender.sendProbing(mediaSsrcs, numBytes)
/**
* Set a handler to be invoked when incoming RTP packets have finished
* being processed.
*/
fun setIncomingPacketHandler(rtpHandler: PacketHandler) {
rtpReceiver.packetHandler = rtpHandler
}
/**
* Set a handler to be invoked when outgoing packets have finished
* being processed (and are ready to be sent)
*/
fun setOutgoingPacketHandler(outgoingPacketHandler: PacketHandler) {
rtpSender.onOutgoingPacket(outgoingPacketHandler)
}
fun addReceiveSsrc(ssrc: Long, mediaType: MediaType) {
logger.cdebug { "${hashCode()} adding receive ssrc $ssrc of type $mediaType" }
streamInformationStore.addReceiveSsrc(ssrc, mediaType)
}
fun removeReceiveSsrc(ssrc: Long) {
logger.cinfo { "Transceiver ${hashCode()} removing receive ssrc $ssrc" }
streamInformationStore.removeReceiveSsrc(ssrc)
}
/**
* Set the 'local' bridge SSRC to [ssrc] for [mediaType]
*/
fun setLocalSsrc(mediaType: MediaType, ssrc: Long) {
val localSsrcSetEvent = SetLocalSsrcEvent(mediaType, ssrc)
rtpSender.handleEvent(localSsrcSetEvent)
rtpReceiver.handleEvent(localSsrcSetEvent)
}
fun addRtpExtensionToRetain(extensionType: RtpExtensionType) {
rtpSender.addRtpExtensionToRetain(extensionType)
}
fun receivesSsrc(ssrc: Long): Boolean = streamInformationStore.receiveSsrcs.contains(ssrc)
val receiveSsrcs: Set<Long>
get() = streamInformationStore.receiveSsrcs
fun setMediaSources(mediaSources: Array<MediaSourceDesc>): Boolean {
logger.cdebug { "$id setting media sources: ${mediaSources.joinToString()}" }
val ret = this.mediaSources.setMediaSources(mediaSources)
val mergedMediaSources = this.mediaSources.getMediaSources()
val signaledMediaSources = mediaSources.copy()
rtpReceiver.handleEvent(SetMediaSourcesEvent(mergedMediaSources, signaledMediaSources))
return ret
}
// TODO(brian): we should only expose an immutable version of this, but Array doesn't have that. Go in
// and change all the storage of the media sources to use a list
fun getMediaSources(): Array<MediaSourceDesc> = mediaSources.getMediaSources()
@JvmOverloads
fun requestKeyFrame(requesterID: String? = null, mediaSsrc: Long? = null) =
rtpSender.requestKeyframe(requesterID, mediaSsrc)
fun addPayloadType(payloadType: PayloadType) {
logger.cdebug { "Payload type added: $payloadType" }
streamInformationStore.addRtpPayloadType(payloadType)
}
fun clearPayloadTypes() {
logger.cinfo { "All payload types being cleared" }
streamInformationStore.clearRtpPayloadTypes()
}
fun addRtpExtension(rtpExtension: RtpExtension) {
logger.cdebug { "Adding RTP extension: $rtpExtension" }
streamInformationStore.addRtpExtensionMapping(rtpExtension)
}
fun clearRtpExtensions() {
logger.cinfo { "Clearing all RTP extensions" }
// TODO: ignoring this for now, since we'll have conflicts from each channel calling it
// val rtpExtensionClearEvent = RtpExtensionClearEvent()
// rtpReceiver.handleEvent(rtpExtensionClearEvent)
// rtpSender.handleEvent(rtpExtensionClearEvent)
// rtpExtensions.clear()
}
fun setExtmapAllowMixed(allow: Boolean) {
streamInformationStore.setExtmapAllowMixed(allow)
}
// TODO(brian): we may want to handle local and remote ssrc associations differently, as different parts of the
// code care about one or the other, but currently there is no issue treating them the same.
fun addSsrcAssociation(ssrcAssociation: SsrcAssociation) {
logger.cdebug {
val location = if (ssrcAssociation is LocalSsrcAssociation) "local" else "remote"
"Adding $location SSRC association: $ssrcAssociation"
}
streamInformationStore.addSsrcAssociation(ssrcAssociation)
}
fun setSrtpInformation(
chosenSrtpProtectionProfile: Int,
tlsRole: TlsRole,
keyingMaterial: ByteArray,
cryptex: Boolean
) {
val srtpProfileInfo =
SrtpUtil.getSrtpProfileInformationFromSrtpProtectionProfile(chosenSrtpProtectionProfile)
logger.cdebug {
"Transceiver $id creating transformers with:\n" +
"profile info:\n$srtpProfileInfo\n" +
"tls role: $tlsRole\n" +
"cryptex: $cryptex"
}
srtpTransformers = SrtpUtil.initializeTransformer(
srtpProfileInfo,
keyingMaterial,
tlsRole,
cryptex,
logger
).also { setSrtpInformationInternal(it, true) }
}
private fun setSrtpInformationInternal(srtpTransformers: SrtpTransformers, internal: Boolean) {
rtpReceiver.setSrtpTransformers(srtpTransformers)
rtpSender.setSrtpTransformers(srtpTransformers)
internalTransformers = internal
}
fun setSrtpInformation(srtpTransformers: SrtpTransformers) = setSrtpInformationInternal(srtpTransformers, false)
/**
* Forcibly mute or unmute the incoming audio stream
*/
fun forceMuteAudio(shouldMute: Boolean) {
when (shouldMute) {
true -> logger.info("Muting incoming audio")
false -> logger.info("Unmuting incoming audio")
}
rtpReceiver.forceMuteAudio(shouldMute)
}
fun forceMuteVideo(shouldMute: Boolean) {
when (shouldMute) {
true -> logger.info("Muting incoming video")
false -> logger.info("Unmuting incoming video")
}
rtpReceiver.forceMuteVideo(shouldMute)
}
fun debugState(mode: DebugStateMode): ObjectNode = JsonNodeFactory.instance.objectNode().apply {
set<ObjectNode>("stream_information_store", streamInformationStore.debugState(mode))
set<ObjectNode>("media_sources", mediaSources.debugState())
set<ObjectNode>("endpoint_connection_stats", endpointConnectionStats.getSnapshot().toJson())
set<ObjectNode>("receiver", rtpReceiver.debugState(mode))
set<ObjectNode>("sender", rtpSender.debugState(mode))
}
/**
* Get various media and network stats
*/
fun getTransceiverStats(): TransceiverStats = TransceiverStats(
endpointConnectionStats.getSnapshot(),
rtpReceiver.getStats(),
rtpSender.getStreamStats(),
rtpSender.getPacketStreamStats(),
rtpSender.getTransportCcEngineStats()
)
fun addEndpointConnectionStatsListener(listener: EndpointConnectionStats.EndpointConnectionStatsListener) =
endpointConnectionStats.addListener(listener)
fun removeEndpointConnectionStatsListener(listener: EndpointConnectionStats.EndpointConnectionStatsListener) =
endpointConnectionStats.removeListener(listener)
override fun stop() {
rtpReceiver.stop()
rtpSender.stop()
if (internalTransformers) {
srtpTransformers?.close()
}
}
fun teardown() {
logger.debug("Tearing down")
rtpReceiver.tearDown()
rtpSender.tearDown()
}
fun setFeature(feature: Features, enabled: Boolean) {
rtpReceiver.setFeature(feature, enabled)
rtpSender.setFeature(feature, enabled)
}
fun isFeatureEnabled(feature: Features): Boolean {
// As of now, the only feature we have (pcap) is always enabled on both
// the RTP sender and RTP receiver at the same time, so returning
// the state of one of them is sufficient. If that were to change
// in the future we'd have to rethink this API
return rtpReceiver.isFeatureEnabled(feature)
}
companion object {
init {
// Node.plugins.add(BufferTracePlugin)
// Node.PLUGINS_ENABLED = true
}
}
}
/**
* Interface for handling events coming from a [Transceiver]
* The intention is to extend if needed (e.g. merge with EndpointConnectionStats or a potential RtpSenderEventHandler).
*/
interface TransceiverEventHandler : RtpReceiverEventHandler

37
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/VideoType.kt Normal file
View File

@@ -0,0 +1,37 @@
/*
* Copyright @ 2021-Present 8x8, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj
enum class VideoType {
CAMERA,
DESKTOP,
DESKTOP_HIGH_FPS,
DISABLED,
// NONE was used in the context where an Endpoint has always one media source description. It used to cover both
// lack of the actual source and the source being temporarily disabled. With the support for multiple sources per
// endpoint DISABLED means a source is turned off. Lack of a MediaSourceDesc is equivalent to NONE.
@Deprecated("Use DISABLED instead", ReplaceWith("DISABLED"))
NONE;
fun isEnabled() = this != NONE && this != DISABLED
/**
* Whether this is a screen sharing type. Note that among other things screen sharing sources have bursty bitrates,
* because the encoder may leave long gaps between frames while the content is static.
*/
fun isScreenshare() = this == DESKTOP || this == DESKTOP_HIGH_FPS
}

94
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/codec/vp8/Vp8Utils.kt Normal file
View File

@@ -0,0 +1,94 @@
/*
* Copyright @ 2018 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj.codec.vp8
import org.jitsi.rtp.rtp.RtpPacket
import org.jitsi_modified.impl.neomedia.codec.video.vp8.DePacketizer
import java.nio.ByteBuffer
class Vp8Utils {
companion object {
private const val VP8_PAYLOAD_HEADER_LEN = 3
// TODO(brian): should move these elsewhere probably
private const val MIN_HD_HEIGHT = 540
private const val MIN_SD_HEIGHT = 360
private const val HD_LAYER_ID = 2
private const val SD_LAYER_ID = 1
private const val LD_LAYER_ID = 0
private const val SUSPENDED_LAYER_ID = -1
fun isKeyFrame(vp8Payload: ByteBuffer): Boolean =
DePacketizer.isKeyFrame(vp8Payload.array(), vp8Payload.arrayOffset(), vp8Payload.limit())
fun getSpatialLayerIndexFromKeyFrame(vp8Payload: ByteBuffer): Int {
// Copied from VP8QualityFilter#getSpatialLayerIndexFromKeyframe
val payloadDescriptorLen =
DePacketizer.VP8PayloadDescriptor.getSize(
vp8Payload.array(),
vp8Payload.arrayOffset(),
vp8Payload.limit()
)
val height = DePacketizer.VP8KeyframeHeader.getHeight(
vp8Payload.array(),
vp8Payload.arrayOffset() + payloadDescriptorLen + VP8_PAYLOAD_HEADER_LEN
)
return when {
height >= MIN_HD_HEIGHT -> HD_LAYER_ID
height >= MIN_SD_HEIGHT -> SD_LAYER_ID
height > -1 -> LD_LAYER_ID
else -> -1
}
}
fun getHeightFromKeyFrame(vp8Packet: RtpPacket): Int {
val payloadDescriptorLen =
DePacketizer.VP8PayloadDescriptor.getSize(
vp8Packet.buffer,
vp8Packet.payloadOffset,
vp8Packet.payloadLength
)
return DePacketizer.VP8KeyframeHeader.getHeight(
vp8Packet.buffer,
vp8Packet.payloadOffset + payloadDescriptorLen + VP8_PAYLOAD_HEADER_LEN
)
}
fun getSpatialLayerIndexFromKeyFrame(vp8Packet: RtpPacket): Int {
// Copied from VP8QUalityFilter#getSpatialLayerIndexFromKeyframe
val height = getHeightFromKeyFrame(vp8Packet)
return when {
height >= MIN_HD_HEIGHT -> HD_LAYER_ID
height >= MIN_SD_HEIGHT -> SD_LAYER_ID
height > -1 -> LD_LAYER_ID
else -> -1
}
}
fun getTemporalLayerIdOfFrame(vp8Payload: ByteBuffer) = DePacketizer.VP8PayloadDescriptor.getTemporalLayerIndex(
vp8Payload.array(),
vp8Payload.arrayOffset(),
vp8Payload.limit()
)
fun getTemporalLayerIdOfFrame(vp8Packet: RtpPacket) = DePacketizer.VP8PayloadDescriptor.getTemporalLayerIndex(
vp8Packet.buffer,
vp8Packet.payloadOffset,
vp8Packet.payloadLength
)
}
}

90
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/codec/vpx/VpxUtils.kt Normal file
View File

@@ -0,0 +1,90 @@
/*
* Copyright @ 2018 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj.codec.vpx
class VpxUtils {
companion object {
/**
* The bitmask for the TL0PICIDX field.
*/
const val TL0PICIDX_MASK = 0xff
/**
* The bitmask for the extended picture id field.
*/
const val EXTENDED_PICTURE_ID_MASK = 0x7fff
/**
* Returns the delta between two VP8/VP9 extended picture IDs, taking into account
* rollover. This will return the 'shortest' delta between the two
* picture IDs in the form of the number you'd add to b to get a. e.g.:
* getExtendedPictureIdDelta(1, 10) -> -9 (10 + -9 = 1)
* getExtendedPictureIdDelta(1, 32760) -> 9 (32760 + 9 = 1)
* @return the delta between two extended picture IDs (modulo 2^15).
*/
@JvmStatic
fun getExtendedPictureIdDelta(a: Int, b: Int): Int {
val diff = a - b
return when {
diff < -(1 shl 14) -> diff + (1 shl 15)
diff > (1 shl 14) -> diff - (1 shl 15)
else -> diff
}
}
/**
* Apply a delta to a given extended picture ID and return the result (taking
* rollover into account)
* @param start the starting extended picture ID
* @param delta the delta to be applied
* @return the extended picture ID resulting from doing "start + delta"
*/
@JvmStatic
fun applyExtendedPictureIdDelta(start: Int, delta: Int): Int = (start + delta) and EXTENDED_PICTURE_ID_MASK
/**
* Returns the delta between two VP8/VP9 Tl0PicIdx values, taking into account
* rollover. This will return the 'shortest' delta between the two
* picture IDs in the form of the number you'd add to b to get a. e.g.:
* getTl0PicIdxDelta(1, 10) -> -9 (10 + -9 = 1)
* getTl0PicIdxDelta(1, 250) -> 7 (250 + 7 = 1)
*
* If either value is -1 (meaning tl0picidx not found) return 0.
* @return the delta between two extended picture IDs (modulo 2^8).
*/
@JvmStatic
fun getTl0PicIdxDelta(a: Int, b: Int): Int {
if (a < 0 || b < 0) return 0
val diff = a - b
return when {
diff < -(1 shl 7) -> diff + (1 shl 8)
diff > (1 shl 7) -> diff - (1 shl 8)
else -> diff
}
}
/**
* Apply a delta to a given Tl0PidIcx and return the result (taking
* rollover into account)
* @param start the starting Tl0PicIdx
* @param delta the delta to be applied
* @return the Tl0PicIdx resulting from doing "start + delta"
*/
@JvmStatic
fun applyTl0PicIdxDelta(start: Int, delta: Int): Int = (start + delta) and TL0PICIDX_MASK
}
}

34
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/dtls/CertificateInfo.kt Normal file
View File

@@ -0,0 +1,34 @@
/*
* Copyright @ 2018 - Present, 8x8 Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj.dtls
import java.security.KeyPair
/**
* Store various information about a generated Certificate, including:
* The [KeyPair] used to build and sign it, as well as the fingerprint
* hash function and the calculated fingerprint we use (which is transmitted
* over the signaling channel so the certificate can be verified). We also
* store the time at which this certificate was created so we can refresh it
* appropriately.
*/
data class CertificateInfo(
val keyPair: KeyPair,
val certificate: org.bouncycastle.tls.Certificate,
val localFingerprintHashFunction: String,
val localFingerprint: String,
val creationTimestampMs: Long
)

58
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/dtls/DtlsClient.kt Normal file
View File

@@ -0,0 +1,58 @@
/*
* Copyright @ 2018 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj.dtls
import org.bouncycastle.tls.Certificate
import org.bouncycastle.tls.DTLSClientProtocol
import org.bouncycastle.tls.DTLSTransport
import org.bouncycastle.tls.DatagramTransport
import org.jitsi.nlj.srtp.TlsRole
import org.jitsi.utils.logging2.Logger
import org.jitsi.utils.logging2.cdebug
import org.jitsi.utils.logging2.cerror
import org.jitsi.utils.logging2.createChildLogger
class DtlsClient(
private val datagramTransport: DatagramTransport,
certificateInfo: CertificateInfo,
private val handshakeCompleteHandler: (Int, TlsRole, ByteArray) -> Unit = { _, _, _ -> },
verifyAndValidateRemoteCertificate: (Certificate?) -> Unit = {},
parentLogger: Logger,
private val dtlsClientProtocol: DTLSClientProtocol = DTLSClientProtocol()
) : DtlsRole {
private val logger = createChildLogger(parentLogger)
private val tlsClient: TlsClientImpl = TlsClientImpl(certificateInfo, verifyAndValidateRemoteCertificate, logger)
override fun start(): DTLSTransport = connect()
fun connect(): DTLSTransport {
try {
return dtlsClientProtocol.connect(this.tlsClient, datagramTransport).also {
logger.cdebug { "DTLS handshake finished" }
handshakeCompleteHandler(
tlsClient.chosenSrtpProtectionProfile,
TlsRole.CLIENT,
tlsClient.srtpKeyingMaterial
)
}
} catch (e: Exception) {
logger.cerror { "Error during DTLS connection: $e" }
throw e
}
}
}

76
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/dtls/DtlsConfig.kt Normal file
View File

@@ -0,0 +1,76 @@
/*
* Copyright @ 2019 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj.dtls
import org.bouncycastle.operator.DefaultDigestAlgorithmIdentifierFinder
import org.bouncycastle.tls.CipherSuite
import org.jitsi.config.JitsiConfig
import org.jitsi.metaconfig.ConfigException
import org.jitsi.metaconfig.config
import java.time.Duration
class DtlsConfig private constructor() {
val handshakeTimeout: Duration by config {
"jmt.dtls.handshake-timeout".from(JitsiConfig.newConfig)
}
val cipherSuites: List<Int> by config {
"jmt.dtls.cipher-suites".from(JitsiConfig.newConfig).convertFrom<List<String>> { list ->
val ciphers = list.map { it.toBcCipherSuite() }
if (ciphers.isEmpty()) {
throw ConfigException.UnableToRetrieve.ConditionNotMet("cipher-suites must not be empty")
}
ciphers
}
}
val localFingerprintHashFunction: String by config {
"jmt.dtls.local-fingerprint-hash-function".from(JitsiConfig.newConfig).transformedBy {
validateHashFunction(it)
}
}
val acceptedFingerprintHashFunctions: List<String> by config {
"jmt.dtls.accepted-fingerprint-hash-functions".from(JitsiConfig.newConfig).convertFrom<List<String>> { list ->
if (list.isEmpty()) {
throw ConfigException.UnableToRetrieve.ConditionNotMet(
"accepted-fingerprint-hash-functions must not be empty"
)
}
list.map { validateHashFunction(it) }
}
}
companion object {
val config = DtlsConfig()
}
}
private fun validateHashFunction(func: String): String {
val ucFunc = func.uppercase()
DefaultDigestAlgorithmIdentifierFinder().find(ucFunc)
?: throw ConfigException.UnableToRetrieve.WrongType("Unknown hash function $func")
if (ucFunc == "MD5" || ucFunc == "MD2") {
throw ConfigException.UnableToRetrieve.WrongType("Forbidden hash function $func")
}
return func.lowercase()
}
private fun String.toBcCipherSuite(): Int = try {
CipherSuite::class.java.getDeclaredField(this).getInt(null)
} catch (e: Exception) {
throw ConfigException.UnableToRetrieve.ConditionNotMet("Value is not a valid BouncyCastle cipher suite name: $this")
}

28
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/dtls/DtlsProtocolPacket.kt Normal file
View File

@@ -0,0 +1,28 @@
/*
* Copyright @ 2018 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj.dtls
import org.jitsi.rtp.Packet
class DtlsProtocolPacket(
buf: ByteArray,
offset: Int,
length: Int
) : Packet(buf, offset, length) {
override fun clone(): DtlsProtocolPacket = DtlsProtocolPacket(cloneBuffer(0), 0, length)
}

32
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/dtls/DtlsRole.kt Normal file
View File

@@ -0,0 +1,32 @@
/*
* Copyright @ 2018 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj.dtls
import org.bouncycastle.tls.DTLSTransport
interface DtlsRole {
/**
* 'Starts' a DTLS connection. Blocks until the connection
* has been successfully established (an exception is thrown
* upon failure). Returns the resulting [DTLSTransport].
*
* The definition of 'start' varies between roles. For a client
* this means 'connect' to a remote server, for a server it means
* 'accept' and wait for an incoming connection.
*/
fun start(): DTLSTransport
}

58
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/dtls/DtlsServer.kt Normal file
View File

@@ -0,0 +1,58 @@
/*
* Copyright @ 2018 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj.dtls
import org.bouncycastle.tls.Certificate
import org.bouncycastle.tls.DTLSServerProtocol
import org.bouncycastle.tls.DTLSTransport
import org.bouncycastle.tls.DatagramTransport
import org.jitsi.nlj.srtp.TlsRole
import org.jitsi.utils.logging2.Logger
import org.jitsi.utils.logging2.cdebug
import org.jitsi.utils.logging2.cerror
import org.jitsi.utils.logging2.createChildLogger
class DtlsServer(
private val datagramTransport: DatagramTransport,
certificateInfo: CertificateInfo,
private val handshakeCompleteHandler: (Int, TlsRole, ByteArray) -> Unit = { _, _, _ -> },
verifyAndValidateRemoteCertificate: (Certificate?) -> Unit = {},
parentLogger: Logger,
private val dtlsServerProtocol: DTLSServerProtocol = DTLSServerProtocol()
) : DtlsRole {
private val logger = createChildLogger(parentLogger)
private val tlsServer: TlsServerImpl = TlsServerImpl(certificateInfo, verifyAndValidateRemoteCertificate, logger)
override fun start(): DTLSTransport = accept()
fun accept(): DTLSTransport {
try {
return dtlsServerProtocol.accept(tlsServer, datagramTransport).also {
logger.cdebug { "DTLS handshake finished" }
handshakeCompleteHandler(
tlsServer.chosenSrtpProtectionProfile,
TlsRole.SERVER,
tlsServer.srtpKeyingMaterial
)
}
} catch (t: Throwable) {
logger.cerror { "Error during DTLS connection: $t" }
throw t
}
}
}

355
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/dtls/DtlsStack.kt Normal file
View File

@@ -0,0 +1,355 @@
/*
* Copyright @ 2018 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj.dtls
import com.fasterxml.jackson.databind.node.JsonNodeFactory
import com.fasterxml.jackson.databind.node.ObjectNode
import org.bouncycastle.tls.Certificate
import org.bouncycastle.tls.DTLSTransport
import org.bouncycastle.tls.DatagramTransport
import org.jitsi.nlj.srtp.TlsRole
import org.jitsi.nlj.util.BufferPool
import org.jitsi.utils.concurrent.ArrayBlockingQueueWithShutdown
import org.jitsi.utils.logging2.Logger
import org.jitsi.utils.logging2.cdebug
import org.jitsi.utils.logging2.createChildLogger
import java.nio.ByteBuffer
import java.time.Duration
import java.util.concurrent.CountDownLatch
import java.util.concurrent.TimeUnit
import kotlin.concurrent.thread
import kotlin.math.min
/**
* A DTLS stack implementation, which can act as either the DTLS server or DTLS client, and can be used
* for negotiating the DTLS connection and sending and receiving data over that connection.
*
* This class does not directly communicate with the network in any way. Raw DTLS packets must be fed into
* this stack via the [processIncomingProtocolData] method. [incomingDataHandler] will be invoked with the
* decrypted, application data sent over the connection.
*
* Data can be sent through the DTLS connection via [sendApplicationData]; it will be encrypted by the stack and
* then sent out via the [outgoingDataHandler], which must be set to have the data go anywhere interesting :)
*
* [eventHandler] will be invoked when any events occur.
*
* After wiring up all the handlers, to start a connection the stack must be told which role to fullfil: client
* or server. This can be done by calling either the [actAsClient] or [actAsServer] methods. Once the role has
* been set, [start] can be called to start the negotiation.
*/
class DtlsStack(
parentLogger: Logger
) {
private val logger = createChildLogger(parentLogger)
private val roleSet = CountDownLatch(1)
/**
* The certificate info for this particular [DtlsStack] instance. We save it in a local val because the global one
* might be refreshed.
*/
private val certificateInfo = DtlsStack.certificateInfo
val localFingerprintHashFunction: String
get() = certificateInfo.localFingerprintHashFunction
val localFingerprint: String
get() = certificateInfo.localFingerprint
/**
* The remote fingerprints sent to us over the signaling path.
*/
var remoteFingerprints: Map<String, List<String>> = mapOf()
/**
* A handler which will be invoked when DTLS application data is received
*/
var incomingDataHandler: IncomingDataHandler? = null
/**
* The method [DtlsStack] will invoke when it wants to send DTLS data out onto the network.
*/
var outgoingDataHandler: OutgoingDataHandler? = null
/**
* Handle to be invoked when events occur
*/
var eventHandler: EventHandler? = null
private val incomingProtocolData = ArrayBlockingQueueWithShutdown<ByteBuffer>(QUEUE_SIZE, true)
private var numPacketDropsQueueFull = 0
/**
* The [DtlsRole] 'plugin' that will determine how this stack operates (as a client
* or a server). A call to [actAsClient] or [actAsServer] must be made to fill out
* this role and successfully call [start]
*/
var role: DtlsRole? = null
private set
/**
* A buffer we'll use to receive data from [dtlsTransport].
*/
private val dtlsAppDataBuf = ByteArray(1500)
/**
* The negotiated DTLS transport. This is used to read and write DTLS application data.
*/
private var dtlsTransport: DTLSTransport? = null
/**
* The [DatagramTransport] implementation we use for this stack.
*/
private val datagramTransport: DatagramTransport = DatagramTransportImpl(logger)
/**
* Set the role to server, unless it already is one.
*
* The role is re-applied whenever the peer signals its `setup` attribute again, which it does on every
* transport update and not only when something about DTLS changed. Building a new [DtlsRole] each time would
* replace the one that has already negotiated the connection with one that never will, so repeating the role
* we already have is a no-op.
*
* @return true if the role changed.
*/
fun actAsServer(): Boolean {
if (role is DtlsServer) {
return false
}
warnIfRoleChangesAfterStart("server")
role = DtlsServer(
datagramTransport,
certificateInfo,
{ chosenSrtpProfile, tlsRole, keyingMaterial ->
eventHandler?.handshakeComplete(chosenSrtpProfile, tlsRole, keyingMaterial)
},
this::verifyAndValidateRemoteCertificate,
logger
)
roleSet.countDown()
return true
}
/** Set the role to client, unless it already is one. See [actAsServer]. */
fun actAsClient(): Boolean {
if (role is DtlsClient) {
return false
}
warnIfRoleChangesAfterStart("client")
role = DtlsClient(
datagramTransport,
certificateInfo,
{ chosenSrtpProfile, tlsRole, keyingMaterial ->
eventHandler?.handshakeComplete(chosenSrtpProfile, tlsRole, keyingMaterial)
},
this::verifyAndValidateRemoteCertificate,
logger
)
roleSet.countDown()
return true
}
/**
* The stack has been started with the role it has now, so replacing that role leaves it with one that was
* never started while [dtlsTransport] keeps running the old one. Nothing in the stack does this today, and
* the peer is not supposed to change its `setup` after the handshake, so log loudly if it happens.
*/
private fun warnIfRoleChangesAfterStart(newRole: String) {
if (dtlsTransport != null) {
logger.warn("Changing the DTLS role to $newRole after the stack was started with ${role?.javaClass}.")
}
}
/**
* 'start' this stack, in whatever role it has been told to operate (client or server). If a role
* has not yet been yet (via [actAsServer] or [actAsClient]), then it will block until the role
* has been set.
*/
fun start() {
roleSet.await()
dtlsTransport = role?.start()
// There is a bit of a race here: It's technically possible the
// far side could finish the handshake and send a message before
// this side assigns dtlsTransport here. If so, that message
// would be passed to #processIncomingProtocolData and put in
// incomingProtocolData, but, since dtlsTransport won't be set
// yet, we won't 'receive' it yet. Check for any incoming packets
// here, to handle this case.
processIncomingProtocolData()
}
fun close() {
datagramTransport.close()
incomingProtocolData.shutdown()
incomingProtocolData.forEach {
BufferPool.returnBuffer(it.array())
}
incomingProtocolData.clear()
}
/**
* Checks that a specific [Certificate] matches the remote fingerprints sent to us over the signaling path.
*/
private fun verifyAndValidateRemoteCertificate(remoteCertificate: Certificate?) {
remoteCertificate?.let {
DtlsUtils.verifyAndValidateCertificate(it, remoteFingerprints)
// The above throws an exception if the checks fail.
logger.cdebug { "Fingerprints verified." }
} ?: run {
throw DtlsUtils.DtlsException("Remote certificate was null")
}
}
fun sendApplicationData(data: ByteArray, off: Int, len: Int) {
dtlsTransport?.send(data, off, len)
}
private fun processIncomingProtocolData() {
var bytesReceived: Int
do {
val bufCopy2: ByteArray? = synchronized(dtlsAppDataBuf) {
bytesReceived = dtlsTransport?.receive(dtlsAppDataBuf, 0, 1500, 1) ?: -1
if (bytesReceived > 0) {
// Copy again to copy out of dtlsAppDataBuf, which we re-use.
BufferPool.getBuffer(bytesReceived).apply {
System.arraycopy(dtlsAppDataBuf, 0, this, 0, bytesReceived)
}
} else {
null
}
}
if (bufCopy2 != null) {
incomingDataHandler?.dataReceived(bufCopy2, 0, bytesReceived)
}
} while (bytesReceived > 0)
}
/**
* We get 'pushed' the data from a lower transport layer, but bouncycastle wants to 'pull' the data
* itself. To mimic this, we put the received data into a queue, and then 'pull' it through ourselves by
* calling 'receive' on the negotiated [DTLSTransport].
*
* Note: the data we get here may be a DTLS protocol packet and therefore won't generate anything
* to be received by [dtlsTransport] or a DTLS app packet (data sent over DTLS after the handshake has
* completed) and will result in data being received through [dtlsTransport]. It's possible, though,
* that the handshake has finished and the far end has sent application data but we've not yet set
* [dtlsTransport], so we "miss" it. We don't lose this data, but it will sit inside of [incomingProtocolData]
* until the next packet comes through. This means that we must make a copy of the buffer we receive, as we
* won't necessarily be done with it by the time this method completes.
*/
fun processIncomingProtocolData(data: ByteArray, off: Int, len: Int) {
val bufCopy = BufferPool.getBuffer(len).apply {
System.arraycopy(data, off, this, 0, len)
}
if (!incomingProtocolData.offer(ByteBuffer.wrap(bufCopy, 0, len))) {
BufferPool.returnBuffer(bufCopy)
if (!incomingProtocolData.isShutdown) {
logger.warn("DTLS stack queue full, dropping packet")
numPacketDropsQueueFull++
}
}
processIncomingProtocolData()
}
fun getDebugState(): ObjectNode = JsonNodeFactory.instance.objectNode().apply {
put("localFingerprintHashFunction", certificateInfo.localFingerprint)
put("remoteFingerprints", remoteFingerprints.map { (hash, fp) -> "$hash: $fp" }.joinToString())
put("role", (role?.javaClass ?: "null").toString())
put("num_packet_drops_queue_full", numPacketDropsQueueFull)
}
companion object {
private const val QUEUE_SIZE = 50
/**
* Because generating the certificateInfo can be expensive, we generate a single
* one to be used everywhere which expires in 24 hours (when we'll generate
* another one).
*/
private val syncRoot: Any = Any()
private var certificateInfo: CertificateInfo = DtlsUtils.generateCertificateInfo()
get() = synchronized(syncRoot) {
val expirationPeriodMs = Duration.ofDays(1).toMillis()
if (field.creationTimestampMs + expirationPeriodMs < System.currentTimeMillis()) {
// TODO: avoid creating our own thread
thread { field = DtlsUtils.generateCertificateInfo() }
}
return field
}
}
/**
* An implementation of [DatagramTransport] which 'receives' from the queue in [DtlsStack] and sends out
* via [DtlsStack]'s [outgoingDataHandler]
*/
inner class DatagramTransportImpl(parentLogger: Logger) : DatagramTransport {
private val logger = createChildLogger(parentLogger)
override fun receive(buf: ByteArray, off: Int, len: Int, waitMillis: Int): Int {
val data = try {
incomingProtocolData.poll(waitMillis.toLong(), TimeUnit.MILLISECONDS) ?: return -1
} catch (ie: InterruptedException) {
return -1
}
val length = min(len, data.limit())
if (length < data.limit()) {
logger.warn(
"Passed buffer size ($len) was too small to hold incoming data size (${data.limit()}); " +
"data was truncated"
)
}
System.arraycopy(data.array(), data.arrayOffset(), buf, off, length)
BufferPool.returnBuffer(data.array())
return length
}
override fun send(buf: ByteArray, off: Int, len: Int) {
outgoingDataHandler?.sendData(buf, off, len)
}
/**
* Receive limit computation copied from [org.bouncycastle.tls.UDPTransport]
*/
override fun getReceiveLimit(): Int = 1500 - 20 - 8
/**
* Send limit computation copied from [org.bouncycastle.tls.UDPTransport]
*/
override fun getSendLimit(): Int = 1500 - 84 - 8
override fun close() {}
}
interface IncomingDataHandler {
/**
* Notify the handler that data has been received. The handler takes ownership of the passed
* buffer, and it should be returned to the buffer pool when done with it.
*/
fun dataReceived(data: ByteArray, off: Int, len: Int)
}
interface OutgoingDataHandler {
fun sendData(data: ByteArray, off: Int, len: Int)
}
interface EventHandler {
fun handshakeComplete(chosenSrtpProtectionProfile: Int, tlsRole: TlsRole, keyingMaterial: ByteArray)
}
}

323
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/dtls/DtlsUtils.kt Normal file
View File

@@ -0,0 +1,323 @@
/*
* Copyright @ 2018 - Present, 8x8 Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj.dtls
import org.bouncycastle.asn1.ASN1Encoding
import org.bouncycastle.asn1.x500.X500Name
import org.bouncycastle.asn1.x500.X500NameBuilder
import org.bouncycastle.asn1.x500.style.BCStyle
import org.bouncycastle.asn1.x509.Certificate
import org.bouncycastle.cert.jcajce.JcaX509v3CertificateBuilder
import org.bouncycastle.jce.ECNamedCurveTable
import org.bouncycastle.jce.provider.BouncyCastleProvider
import org.bouncycastle.operator.DefaultDigestAlgorithmIdentifierFinder
import org.bouncycastle.operator.bc.BcDefaultDigestProvider
import org.bouncycastle.operator.jcajce.JcaContentSignerBuilder
import org.bouncycastle.tls.AlertDescription
import org.bouncycastle.tls.TlsContext
import org.bouncycastle.tls.TlsUtils
import org.bouncycastle.tls.crypto.TlsSecret
import org.bouncycastle.tls.crypto.impl.bc.BcTlsCertificate
import org.bouncycastle.tls.crypto.impl.bc.BcTlsCrypto
import org.jitsi.utils.logging2.Logger
import org.jitsi.utils.logging2.cdebug
import org.jitsi.utils.logging2.cerror
import org.jitsi.utils.logging2.cinfo
import java.math.BigInteger
import java.security.KeyPair
import java.security.KeyPairGenerator
import java.security.SecureRandom
import java.security.Security
import java.time.Duration
import java.util.Date
import java.util.NoSuchElementException
private val SECURE_RANDOM = SecureRandom()
val BC_TLS_CRYPTO = BcTlsCrypto(SECURE_RANDOM)
/**
* Various helper utilities for DTLS
*
* https://tools.ietf.org/html/draft-ietf-rtcweb-security-arch-18
*/
class DtlsUtils {
companion object {
init {
Security.addProvider(BouncyCastleProvider())
}
fun generateCertificateInfo(): CertificateInfo {
val cn = generateCN("TODO-APP-NAME", "TODO-APP-VERSION")
val keyPair = generateEcKeyPair()
val x509certificate = generateCertificate(cn, keyPair)
val localFingerprintHashFunction = DtlsConfig.config.localFingerprintHashFunction
val localFingerprint = x509certificate.getFingerprint(localFingerprintHashFunction)
val certificate = org.bouncycastle.tls.Certificate(
arrayOf(BcTlsCertificate(BC_TLS_CRYPTO, x509certificate))
)
return CertificateInfo(
keyPair,
certificate,
localFingerprintHashFunction,
localFingerprint,
System.currentTimeMillis()
)
}
/**
* A helper which finds an SRTP protection profile present in both
* [ours] and [theirs]. Throws [DtlsException] if no common profile is found.
*/
fun chooseSrtpProtectionProfile(ours: Iterable<Int>, theirs: Iterable<Int>): Int = try {
ours.first(theirs::contains)
} catch (e: NoSuchElementException) {
throw DtlsException(
"No common SRTP protection profile found. Ours: ${ours.joinToString()} " +
"Theirs: ${theirs.joinToString()}"
)
}
/**
* Generate an x509 certificate valid from 1 day ago until 7 days from now.
*
* TODO: make the algorithm dynamic (passed in) to support older dtls versions/clients
*/
private fun generateCertificate(subject: X500Name, keyPair: KeyPair): Certificate {
val now = System.currentTimeMillis()
val startDate = Date(now - Duration.ofDays(1).toMillis())
val expiryDate = Date(now + Duration.ofDays(7).toMillis())
val serialNumber = BigInteger.valueOf(now)
val certBuilder = JcaX509v3CertificateBuilder(
subject,
serialNumber,
startDate,
expiryDate,
subject,
keyPair.public
)
val signer = JcaContentSignerBuilder("SHA256withECDSA").build(keyPair.private)
return certBuilder.build(signer).toASN1Structure()
}
/**
* Generate an eliptic-curve keypair using the secp256r1 named curve:
* "All Implementations MUST implement DTLS 1.2 with the
* TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 cipher suite and the P-256
* curve"
*
* --https://tools.ietf.org/html/draft-ietf-rtcweb-security-arch-18#section-6.5
*
* NOTE(brian): I used 'secp256r1' specifically because it's what I saw in wireshark traces from chrome
*/
private fun generateEcKeyPair(): KeyPair {
val keyGen = KeyPairGenerator.getInstance("EC", "BC")
val ecCurveSpec = ECNamedCurveTable.getParameterSpec("secp256r1")
keyGen.initialize(ecCurveSpec)
return keyGen.generateKeyPair()
}
/**
* Generate an [X500Name] using the given [appName] and [appVersion]
*/
private fun generateCN(appName: String, appVersion: String): X500Name {
val builder = X500NameBuilder(BCStyle.INSTANCE)
val rdn = "$appName $appVersion"
builder.addRDN(BCStyle.CN, rdn)
return builder.build()
}
/**
* Verifies and validates a specific certificate against the fingerprints
* presented by the remote endpoint via the signaling path.
*
* @param certificateInfo the certificate to be verified and validated against
* the fingerprints presented by the remote endpoint via the signaling path
* @throws [DtlsException] if [certificateInfo] fails validation
*/
fun verifyAndValidateCertificate(
certificateInfo: org.bouncycastle.tls.Certificate,
remoteFingerprints: Map<String, List<String>>
) {
if (certificateInfo.certificateList.isEmpty()) {
throw DtlsException("No remote fingerprints.")
}
for (currCertificate in certificateInfo.certificateList) {
val x509Cert = Certificate.getInstance(currCertificate.encoded)
verifyAndValidateCertificate(x509Cert, remoteFingerprints)
}
}
/**
* Verifies and validates a specific certificate against the fingerprints
* presented by the remote endpoint via the signaling path.
*
* @param certificate the certificate to be verified and validated against
* the fingerprints presented by the remote endpoint via the signaling path.
* @throws DtlsException if the specified [certificate] failed to verify
* and validate against the fingerprints presented by the remote endpoint
* via the signaling path.
*/
private fun verifyAndValidateCertificate(
certificate: Certificate,
remoteFingerprints: Map<String, List<String>>
) {
/* RFC 8122:
* An endpoint MUST select the set of fingerprints that use its most
* preferred hash function (out of those offered by the peer) and verify
* that each certificate used matches one fingerprint out of that set.
*/
DtlsConfig.config.acceptedFingerprintHashFunctions.forEach { hashFunction ->
val fingerprints = remoteFingerprints[hashFunction] ?: return@forEach
val certificateFingerprint = certificate.getFingerprint(hashFunction)
if (fingerprints.none { it == certificateFingerprint }) {
throw DtlsException(
"None of the fingerprints ${fingerprints.joinToString()} match the $hashFunction-hashed " +
"certificate $certificateFingerprint"
)
}
return@verifyAndValidateCertificate
}
/* If we got here none of our accepted fingerprint functions were listed. */
throw DtlsException(
"No fingerprint declared over the signaling path with any of the accepted hash functions: " +
DtlsConfig.config.acceptedFingerprintHashFunctions.joinToString()
)
}
/**
* Computes the fingerprint of a [org.bouncycastle.asn1.x509.Certificate] using [hashFunction] and returns it
* as a [String]
*/
private fun Certificate.getFingerprint(hashFunction: String): String {
val digAlgId = DefaultDigestAlgorithmIdentifierFinder().find(hashFunction.uppercase())
val digest = BcDefaultDigestProvider.INSTANCE.get(digAlgId)
val input: ByteArray = getEncoded(ASN1Encoding.DER)
val output = ByteArray(digest.digestSize)
digest.update(input, 0, input.size)
digest.doFinal(output, 0)
return output.toFingerprint()
}
private val HEX_CHARS = "0123456789ABCDEF".toCharArray()
/**
* Helper function to convert a [ByteArray] to a colon-delimited hex string
*/
private fun ByteArray.toFingerprint(): String {
val buf = StringBuffer()
for (i in 0 until size) {
val octet = get(i).toInt()
val firstIndex = (octet and 0xF0).ushr(4)
val secondIndex = octet and 0x0F
buf.append(HEX_CHARS[firstIndex])
buf.append(HEX_CHARS[secondIndex])
if (i < size - 1) {
buf.append(":")
}
}
return buf.toString()
}
/*
* Copied from TlsContext#exportKeyingMaterial and modified to work with
* an externally provided masterSecret value.
*/
fun exportKeyingMaterial(
context: TlsContext,
asciiLabel: String,
context_value: ByteArray?,
length: Int,
masterSecret: TlsSecret
): ByteArray {
if (context_value != null && !TlsUtils.isValidUint16(context_value.size)) {
throw IllegalArgumentException("'context_value' must have a length less than 2^16 (or be null)")
}
val sp = context.securityParameters
val cr = sp.clientRandom
val sr = sp.serverRandom
var seedLength = cr.size + sr.size
if (context_value != null) {
seedLength += (2 + context_value.size)
}
val seed = ByteArray(seedLength)
var seedPos = 0
System.arraycopy(cr, 0, seed, seedPos, cr.size)
seedPos += cr.size
System.arraycopy(sr, 0, seed, seedPos, sr.size)
seedPos += sr.size
if (context_value != null) {
TlsUtils.writeUint16(context_value.size, seed, seedPos)
seedPos += 2
System.arraycopy(context_value, 0, seed, seedPos, context_value.size)
seedPos += context_value.size
}
if (seedPos != seedLength) {
throw IllegalStateException("error in calculation of seed for export")
}
return TlsUtils.PRF(sp, masterSecret, asciiLabel, seed, length).extract()
}
}
class DtlsException(msg: String) : Exception(msg)
}
@Suppress("NOTHING_TO_INLINE") // Avoid adding to the trace in the log file
inline fun Logger.notifyAlertRaised(alertLevel: Short, alertDescription: Short, message: String?, cause: Throwable?) {
when (alertDescription) {
AlertDescription.close_notify -> cdebug { "close_notify raised, connection closing" }
else -> {
val stack = with(StringBuffer()) {
val e = Exception()
for (el in e.stackTrace) {
appendLine(el.toString())
}
toString()
}
cinfo {
"Alert raised: level=$alertLevel, description=$alertDescription, message=$message " +
"cause=$cause $stack"
}
}
}
}
@Suppress("NOTHING_TO_INLINE") // Avoid adding to the trace in the log file
inline fun Logger.notifyAlertReceived(alertLevel: Short, alertDescription: Short) {
when (alertDescription) {
AlertDescription.close_notify -> cinfo { "close_notify received, connection closing" }
else -> cerror {
"Alert received: level=$alertLevel, description=$alertDescription " +
"(${AlertDescription.getName(alertDescription)})"
}
}
}

170
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/dtls/TlsClientImpl.kt Normal file
View File

@@ -0,0 +1,170 @@
/*
* Copyright @ 2018 - Present, 8x8 Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj.dtls
import org.bouncycastle.crypto.util.PrivateKeyFactory
import org.bouncycastle.tls.Certificate
import org.bouncycastle.tls.CertificateRequest
import org.bouncycastle.tls.DefaultTlsClient
import org.bouncycastle.tls.ExporterLabel
import org.bouncycastle.tls.ExtensionType
import org.bouncycastle.tls.HashAlgorithm
import org.bouncycastle.tls.ProtocolVersion
import org.bouncycastle.tls.SignatureAlgorithm
import org.bouncycastle.tls.SignatureAndHashAlgorithm
import org.bouncycastle.tls.TlsAuthentication
import org.bouncycastle.tls.TlsCredentials
import org.bouncycastle.tls.TlsSRTPUtils
import org.bouncycastle.tls.TlsServerCertificate
import org.bouncycastle.tls.TlsSession
import org.bouncycastle.tls.TlsUtils
import org.bouncycastle.tls.UseSRTPData
import org.bouncycastle.tls.crypto.TlsCryptoParameters
import org.bouncycastle.tls.crypto.impl.bc.BcDefaultTlsCredentialedSigner
import org.bouncycastle.tls.crypto.impl.bc.BcTlsCrypto
import org.jitsi.nlj.srtp.SrtpConfig
import org.jitsi.nlj.srtp.SrtpUtil
import org.jitsi.rtp.extensions.toHex
import org.jitsi.utils.logging2.Logger
import org.jitsi.utils.logging2.cdebug
import org.jitsi.utils.logging2.cinfo
import org.jitsi.utils.logging2.createChildLogger
import java.nio.ByteBuffer
import java.util.Hashtable
/**
* Implementation of [DefaultTlsClient].
*/
class TlsClientImpl(
private val certificateInfo: CertificateInfo,
/**
* The function to call when the server certificateInfo is available.
*/
private val notifyServerCertificate: (Certificate?) -> Unit,
parentLogger: Logger
) : DefaultTlsClient(BC_TLS_CRYPTO) {
private val logger = createChildLogger(parentLogger)
private var session: TlsSession? = null
private var clientCredentials: TlsCredentials? = null
/**
* Only set after a handshake has completed
*/
lateinit var srtpKeyingMaterial: ByteArray
private set
var chosenSrtpProtectionProfile: Int = 0
override fun getSessionToResume(): TlsSession? = session
override fun getAuthentication(): TlsAuthentication {
return object : TlsAuthentication {
override fun getClientCredentials(certificateRequest: CertificateRequest): TlsCredentials {
// NOTE: can't set clientCredentials when it is declared because 'context' won't be set yet
if (clientCredentials == null) {
clientCredentials = BcDefaultTlsCredentialedSigner(
TlsCryptoParameters(context),
(context.crypto as BcTlsCrypto),
PrivateKeyFactory.createKey(certificateInfo.keyPair.private.encoded),
certificateInfo.certificate,
if (TlsUtils.isSignatureAlgorithmsExtensionAllowed(context.serverVersion)) {
SignatureAndHashAlgorithm(
HashAlgorithm.sha256,
SignatureAlgorithm.ecdsa
)
} else {
null
}
)
}
return clientCredentials!!
}
override fun notifyServerCertificate(serverCertificate: TlsServerCertificate) {
this@TlsClientImpl.notifyServerCertificate(serverCertificate.certificate)
}
}
}
override fun getClientExtensions(): Hashtable<*, *> {
var clientExtensions = super.getClientExtensions()
if (TlsSRTPUtils.getUseSRTPExtension(clientExtensions) == null) {
if (clientExtensions == null) {
clientExtensions = Hashtable<Int, ByteArray>()
}
TlsSRTPUtils.addUseSRTPExtension(
clientExtensions,
UseSRTPData(SrtpConfig.protectionProfiles.toIntArray(), TlsUtils.EMPTY_BYTES)
)
}
clientExtensions.put(ExtensionType.renegotiation_info, byteArrayOf(0))
return clientExtensions
}
override fun processServerExtensions(serverExtensions: Hashtable<*, *>?) {
// TODO: a few cases we should be throwing alerts for in here. see old TlsClientImpl
val useSRTPData = TlsSRTPUtils.getUseSRTPExtension(serverExtensions)
val protectionProfiles = useSRTPData.protectionProfiles
chosenSrtpProtectionProfile =
DtlsUtils.chooseSrtpProtectionProfile(SrtpConfig.protectionProfiles, protectionProfiles.asIterable())
}
override fun getCipherSuites() = DtlsConfig.config.cipherSuites.toIntArray()
override fun getHandshakeTimeoutMillis(): Int = DtlsConfig.config.handshakeTimeout.toMillis().toInt()
override fun notifyHandshakeComplete() {
super.notifyHandshakeComplete()
logger.cinfo { "Negotiated DTLS version ${context.securityParameters.negotiatedVersion}" }
context.resumableSession?.let { newSession ->
session?.let { existingSession ->
if (existingSession.sessionID?.contentEquals(newSession.sessionID) == true) {
logger.cdebug {
val newSessionIdHex = ByteBuffer.wrap(newSession.sessionID).toHex()
"Resumed DTLS session $newSessionIdHex"
}
}
} ?: run {
logger.cdebug {
val newSessionIdHex = ByteBuffer.wrap(newSession.sessionID).toHex()
"Established DTLS session $newSessionIdHex"
}
this.session = newSession
}
}
val srtpProfileInformation =
SrtpUtil.getSrtpProfileInformationFromSrtpProtectionProfile(chosenSrtpProtectionProfile)
srtpKeyingMaterial = context.exportKeyingMaterial(
ExporterLabel.dtls_srtp,
null,
2 * (srtpProfileInformation.cipherKeyLength + srtpProfileInformation.cipherSaltLength)
)
}
override fun getSupportedVersions(): Array<ProtocolVersion> = arrayOf(ProtocolVersion.DTLSv12)
override fun notifyAlertRaised(alertLevel: Short, alertDescription: Short, message: String?, cause: Throwable?) =
logger.notifyAlertRaised(alertLevel, alertDescription, message, cause)
override fun notifyAlertReceived(alertLevel: Short, alertDescription: Short) =
logger.notifyAlertReceived(alertLevel, alertDescription)
}

168
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/dtls/TlsServerImpl.kt Normal file
View File

@@ -0,0 +1,168 @@
/*
* Copyright @ 2018 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj.dtls
import org.bouncycastle.crypto.util.PrivateKeyFactory
import org.bouncycastle.tls.Certificate
import org.bouncycastle.tls.CertificateRequest
import org.bouncycastle.tls.ClientCertificateType
import org.bouncycastle.tls.DefaultTlsServer
import org.bouncycastle.tls.ExporterLabel
import org.bouncycastle.tls.HashAlgorithm
import org.bouncycastle.tls.ProtocolVersion
import org.bouncycastle.tls.SignatureAlgorithm
import org.bouncycastle.tls.SignatureAndHashAlgorithm
import org.bouncycastle.tls.TlsCredentialedDecryptor
import org.bouncycastle.tls.TlsCredentialedSigner
import org.bouncycastle.tls.TlsSRTPUtils
import org.bouncycastle.tls.TlsSession
import org.bouncycastle.tls.TlsUtils
import org.bouncycastle.tls.UseSRTPData
import org.bouncycastle.tls.crypto.TlsCryptoParameters
import org.bouncycastle.tls.crypto.impl.bc.BcDefaultTlsCredentialedDecryptor
import org.bouncycastle.tls.crypto.impl.bc.BcDefaultTlsCredentialedSigner
import org.bouncycastle.tls.crypto.impl.bc.BcTlsCrypto
import org.jitsi.nlj.srtp.SrtpConfig
import org.jitsi.nlj.srtp.SrtpUtil
import org.jitsi.rtp.extensions.toHex
import org.jitsi.utils.logging2.Logger
import org.jitsi.utils.logging2.cinfo
import org.jitsi.utils.logging2.createChildLogger
import java.nio.ByteBuffer
import java.util.Hashtable
import java.util.Vector
class TlsServerImpl(
private val certificateInfo: CertificateInfo,
/**
* The function to call when the client certificateInfo is available.
*/
private val notifyClientCertificateReceived: (Certificate?) -> Unit,
parentLogger: Logger
) : DefaultTlsServer(BC_TLS_CRYPTO) {
private val logger = createChildLogger(parentLogger)
private var session: TlsSession? = null
/**
* Only set after a handshake has completed
*/
lateinit var srtpKeyingMaterial: ByteArray
private set
var chosenSrtpProtectionProfile: Int = 0
override fun getSessionToResume(sessionID: ByteArray?): TlsSession? {
return session
// TODO: do we need to map multiple sessions (per sessionID?)
// return super.getSessionToResume(sessionID)
}
override fun getServerExtensions(): Hashtable<*, *> {
val extensions = super.getServerExtensions()
?: Hashtable<Int, ByteArray>()
return extensions.also {
if (TlsSRTPUtils.getUseSRTPExtension(it) == null) {
TlsSRTPUtils.addUseSRTPExtension(
it,
UseSRTPData(intArrayOf(chosenSrtpProtectionProfile), TlsUtils.EMPTY_BYTES)
)
}
}
}
override fun processClientExtensions(clientExtensions: Hashtable<*, *>?) {
super.processClientExtensions(clientExtensions)
val useSRTPData = TlsSRTPUtils.getUseSRTPExtension(clientExtensions)
val protectionProfiles = useSRTPData.protectionProfiles
chosenSrtpProtectionProfile =
DtlsUtils.chooseSrtpProtectionProfile(SrtpConfig.protectionProfiles, protectionProfiles.asIterable())
}
override fun getCipherSuites() = DtlsConfig.config.cipherSuites.toIntArray()
override fun getRSAEncryptionCredentials(): TlsCredentialedDecryptor = BcDefaultTlsCredentialedDecryptor(
(context.crypto as BcTlsCrypto),
certificateInfo.certificate,
PrivateKeyFactory.createKey(certificateInfo.keyPair.private.encoded)
)
override fun getECDSASignerCredentials(): TlsCredentialedSigner = BcDefaultTlsCredentialedSigner(
TlsCryptoParameters(context),
(context.crypto as BcTlsCrypto),
PrivateKeyFactory.createKey(certificateInfo.keyPair.private.encoded),
certificateInfo.certificate,
SignatureAndHashAlgorithm(HashAlgorithm.sha256, SignatureAlgorithm.ecdsa)
)
override fun getCertificateRequest(): CertificateRequest {
val signatureAlgorithms = Vector<SignatureAndHashAlgorithm>(1)
signatureAlgorithms.add(SignatureAndHashAlgorithm(HashAlgorithm.sha256, SignatureAlgorithm.ecdsa))
return CertificateRequest(shortArrayOf(ClientCertificateType.ecdsa_sign), signatureAlgorithms, null)
}
override fun getHandshakeTimeoutMillis(): Int = DtlsConfig.config.handshakeTimeout.toMillis().toInt()
override fun notifyHandshakeComplete() {
super.notifyHandshakeComplete()
logger.cinfo { "Negotiated DTLS version ${context.securityParameters.negotiatedVersion}" }
context.resumableSession?.let { newSession ->
val newSessionIdHex = ByteBuffer.wrap(newSession.sessionID).toHex()
session?.let { existingSession ->
if (existingSession.sessionID?.contentEquals(newSession.sessionID) == true) {
logger.cinfo { "Resumed DTLS session $newSessionIdHex" }
}
} ?: run {
logger.cinfo { "Established DTLS session $newSessionIdHex" }
this.session = newSession
}
}
val srtpProfileInformation =
SrtpUtil.getSrtpProfileInformationFromSrtpProtectionProfile(chosenSrtpProtectionProfile)
if (!context.securityParameters.isExtendedMasterSecret) {
context.session?.exportSessionParameters()?.masterSecret?.let {
srtpKeyingMaterial = DtlsUtils.exportKeyingMaterial(
context,
ExporterLabel.dtls_srtp,
null,
2 * (srtpProfileInformation.cipherKeyLength + srtpProfileInformation.cipherSaltLength),
it
)
}
} else {
srtpKeyingMaterial = context.exportKeyingMaterial(
ExporterLabel.dtls_srtp,
null,
2 * (srtpProfileInformation.cipherKeyLength + srtpProfileInformation.cipherSaltLength)
)
}
}
override fun notifyClientCertificate(clientCertificate: Certificate?) {
notifyClientCertificateReceived(clientCertificate)
}
override fun notifyAlertRaised(alertLevel: Short, alertDescription: Short, message: String?, cause: Throwable?) =
logger.notifyAlertRaised(alertLevel, alertDescription, message, cause)
override fun notifyAlertReceived(alertLevel: Short, alertDescription: Short) =
logger.notifyAlertReceived(alertLevel, alertDescription)
override fun getSupportedVersions(): Array<ProtocolVersion> = arrayOf(ProtocolVersion.DTLSv12)
}

218
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/format/PayloadType.kt Normal file
View File

@@ -0,0 +1,218 @@
/*
* Copyright @ 2019-Present 8x8, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj.format
import org.jitsi.utils.MediaType
import java.util.concurrent.ConcurrentHashMap
import java.util.concurrent.CopyOnWriteArraySet
typealias PayloadTypeParams = Map<String, String>
typealias RtcpFeedbackSet = Set<String>
fun RtcpFeedbackSet.supportsPli(): Boolean = this.contains("nack pli")
fun RtcpFeedbackSet.supportsFir(): Boolean = this.contains("ccm fir")
fun RtcpFeedbackSet.supportsRemb(): Boolean = this.contains("goog-remb")
fun RtcpFeedbackSet.supportsTcc(): Boolean = this.contains("transport-cc")
/**
* Represents an RTP payload type.
*
* @author Boris Grozev
* @author Brian Baldino
*/
abstract class PayloadType(
/**
* The 7-bit RTP payload type number.
*/
val pt: Byte,
/**
* The encoding name.
*/
val encoding: PayloadTypeEncoding,
/**
* The media type (audio or video).
*/
val mediaType: MediaType,
/**
* The RTP clock rate.
*/
val clockRate: Int,
/**
* Additional parameters associated with the payload type (e.g. the "apt" used for RTX).
*/
val parameters: PayloadTypeParams = ConcurrentHashMap(),
/**
* The rtcp feedback messages associated with the payload type (e.g. nack, nack pli, transport-cc, goog-remb, ccm fir, etc).
*/
rtcpFeedbackSet: RtcpFeedbackSet = emptySet()
) {
val rtcpFeedbackSet = CopyOnWriteArraySet(rtcpFeedbackSet)
override fun toString(): String = with(StringBuffer()) {
append(pt).append(" -> ").append(encodingName())
.append(" (").append(clockRate)
.append(channelsString())
.append("): ").append(parameters)
toString()
}
open fun encodingName() = encoding.name.lowercase()
open fun channelsString() = ""
}
enum class PayloadTypeEncoding {
OTHER,
VP8,
VP9,
AV1,
H264,
RED,
RTX,
OPUS,
TELEPHONE_EVENT;
companion object {
private const val TEL_EVENT_TEXT = "telephone-event"
/**
* [valueOf] does not allow for case-insensitivity and can't be overridden, so this
* method should be used when creating an instance of this enum from a string
*/
fun createFrom(value: String): PayloadTypeEncoding {
return try {
if (value.lowercase() == TEL_EVENT_TEXT) {
return TELEPHONE_EVENT
}
valueOf(value.uppercase())
} catch (e: IllegalArgumentException) {
return OTHER
}
}
}
override fun toString(): String = with(StringBuffer()) {
if (super.equals(TELEPHONE_EVENT)) {
append(TEL_EVENT_TEXT)
} else {
append(super.toString())
}
toString()
}
}
abstract class VideoPayloadType(
pt: Byte,
encoding: PayloadTypeEncoding,
clockRate: Int = 90000,
parameters: PayloadTypeParams = ConcurrentHashMap(),
rtcpFeedbackSet: RtcpFeedbackSet = emptySet()
) : PayloadType(pt, encoding, MediaType.VIDEO, clockRate, parameters, rtcpFeedbackSet)
class Vp8PayloadType(
pt: Byte,
parameters: PayloadTypeParams = ConcurrentHashMap(),
rtcpFeedbackSet: RtcpFeedbackSet = emptySet()
) : VideoPayloadType(pt, PayloadTypeEncoding.VP8, parameters = parameters, rtcpFeedbackSet = rtcpFeedbackSet)
class Vp9PayloadType(
pt: Byte,
parameters: PayloadTypeParams = ConcurrentHashMap(),
rtcpFeedbackSet: RtcpFeedbackSet = emptySet()
) : VideoPayloadType(pt, PayloadTypeEncoding.VP9, parameters = parameters, rtcpFeedbackSet = rtcpFeedbackSet)
class Av1PayloadType(
pt: Byte,
parameters: PayloadTypeParams = ConcurrentHashMap(),
rtcpFeedbackSet: RtcpFeedbackSet = emptySet()
) : VideoPayloadType(pt, PayloadTypeEncoding.AV1, parameters = parameters, rtcpFeedbackSet = rtcpFeedbackSet)
class H264PayloadType(
pt: Byte,
parameters: PayloadTypeParams = ConcurrentHashMap(),
rtcpFeedbackSet: RtcpFeedbackSet = emptySet()
) : VideoPayloadType(pt, PayloadTypeEncoding.H264, parameters = parameters, rtcpFeedbackSet = rtcpFeedbackSet)
class RtxPayloadType(
pt: Byte,
parameters: PayloadTypeParams = ConcurrentHashMap()
) : VideoPayloadType(pt, PayloadTypeEncoding.RTX, parameters = parameters) {
val associatedPayloadType: Int =
parameters["apt"]?.toInt() ?: error("RtxPayloadType must contain 'apt'")
}
abstract class AudioPayloadType(
pt: Byte,
encoding: PayloadTypeEncoding,
clockRate: Int = 48000,
/**
* The number of channels
*/
val channels: Int = 1,
parameters: PayloadTypeParams = ConcurrentHashMap()
) : PayloadType(pt, encoding, MediaType.AUDIO, clockRate, parameters) {
override fun channelsString(): String = if (channels > 1) "/$channels" else ""
}
class OpusPayloadType(
pt: Byte,
clockRate: Int = 48000,
channels: Int = 2,
parameters: PayloadTypeParams = ConcurrentHashMap()
) : AudioPayloadType(pt, PayloadTypeEncoding.OPUS, clockRate, channels, parameters = parameters)
class TelephoneEventPayloadType(
pt: Byte,
clockRate: Int,
channels: Int = 1,
parameters: PayloadTypeParams = ConcurrentHashMap()
) : AudioPayloadType(pt, PayloadTypeEncoding.TELEPHONE_EVENT, clockRate, channels, parameters)
class AudioRedPayloadType(
pt: Byte,
clockRate: Int = 48000,
channels: Int = 1,
parameters: PayloadTypeParams = ConcurrentHashMap()
) : AudioPayloadType(pt, PayloadTypeEncoding.RED, clockRate, channels, parameters)
class VideoRedPayloadType(
pt: Byte,
clockRate: Int = 90000,
parameters: PayloadTypeParams = ConcurrentHashMap(),
rtcpFeedbackSet: RtcpFeedbackSet = emptySet()
) : VideoPayloadType(pt, PayloadTypeEncoding.RED, clockRate, parameters, rtcpFeedbackSet)
class OtherAudioPayloadType(
pt: Byte,
private val encodingName: String,
clockRate: Int,
channels: Int = 1,
parameters: PayloadTypeParams = ConcurrentHashMap()
) : AudioPayloadType(pt, PayloadTypeEncoding.OTHER, clockRate, channels, parameters) {
override fun encodingName() = encodingName.lowercase()
}
class OtherVideoPayloadType(
pt: Byte,
private val encodingName: String,
clockRate: Int,
parameters: PayloadTypeParams = ConcurrentHashMap()
) : VideoPayloadType(pt, PayloadTypeEncoding.OTHER, clockRate, parameters) {
override fun encodingName() = encodingName.lowercase()
}

310
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/rtcp/KeyframeRequester.kt Normal file
View File

@@ -0,0 +1,310 @@
/*
* Copyright @ 2018 - present 8x8, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj.rtcp
import org.jitsi.config.JitsiConfig
import org.jitsi.metaconfig.config
import org.jitsi.nlj.Event
import org.jitsi.nlj.PacketInfo
import org.jitsi.nlj.SetLocalSsrcEvent
import org.jitsi.nlj.stats.NodeStatsBlock
import org.jitsi.nlj.transform.node.TransformerNode
import org.jitsi.nlj.util.ReadOnlyStreamInformationStore
import org.jitsi.rtp.rtcp.rtcpfb.RtcpFbPacket
import org.jitsi.rtp.rtcp.rtcpfb.payload_specific_fb.RtcpFbFirPacket
import org.jitsi.rtp.rtcp.rtcpfb.payload_specific_fb.RtcpFbFirPacketBuilder
import org.jitsi.rtp.rtcp.rtcpfb.payload_specific_fb.RtcpFbPliPacket
import org.jitsi.rtp.rtcp.rtcpfb.payload_specific_fb.RtcpFbPliPacketBuilder
import org.jitsi.utils.MediaType
import org.jitsi.utils.RateLimit
import org.jitsi.utils.durationOfDoubleSeconds
import org.jitsi.utils.logging2.Logger
import org.jitsi.utils.logging2.cdebug
import org.jitsi.utils.logging2.createChildLogger
import org.jitsi.utils.min
import java.time.Clock
import java.time.Duration
import java.time.Instant
import java.util.concurrent.atomic.AtomicInteger
/**
* [KeyframeRequester] handles a few things around keyframes:
* 1) The bridge requesting a keyframe (e.g. in order to switch) via the [KeyframeRequester#requestKeyframe]
* method which will create a new keyframe request and forward it
* 2) PLI/FIR translation. If a PLI or FIR packet is forwarded through here, this class may translate it depending
* on what the client supports
* 3) Aggregation. This class will pace outgoing requests such that we don't spam the sender
*/
class KeyframeRequester @JvmOverloads constructor(
private val streamInformationStore: ReadOnlyStreamInformationStore,
parentLogger: Logger,
private val clock: Clock = Clock.systemDefaultZone()
) : TransformerNode("Keyframe Requester") {
private val logger = createChildLogger(parentLogger)
// Map the tuple of requester and SSRC to a rate limiter
private val perReceiverKeyframeLimiter = mutableMapOf<String, MutableMap<Long, RateLimit>>()
private val perSourceKeyframeLimiter = mutableMapOf<Long, RateLimit>()
private val keyframeLimiterSyncRoot = Any()
private val firCommandSequenceNumber: AtomicInteger = AtomicInteger(0)
private var localSsrc: Long? = null
private var waitInterval = minInterval
// Stats
// Number of PLI/FIRs received and forwarded to the endpoint.
private var numPlisForwarded: Int = 0
private var numFirsForwarded: Int = 0
// Number of PLI/FIRs received but dropped due to throttling.
private var numPlisDropped: Int = 0
private var numFirsDropped: Int = 0
// Number of PLI/FIRs generated as a result of an API request or due to translation between PLI/FIR.
private var numPlisGenerated: Int = 0
private var numFirsGenerated: Int = 0
// Number of calls to requestKeyframe
private var numApiRequests: Int = 0
// Number of calls to requestKeyframe ignored due to throttling
private var numApiRequestsDropped: Int = 0
// Number of requests dropped by each limiter, to show which one is binding.
private var numRequestsDroppedPerReceiverLimit: Int = 0
private var numRequestsDroppedSourceWideLimit: Int = 0
override fun transform(packetInfo: PacketInfo): PacketInfo? {
val pliOrFirPacket = packetInfo.getPliOrFirPacket() ?: return packetInfo
val now = clock.instant()
val sourceSsrc: Long
val canSend: Boolean
val forward: Boolean
when (pliOrFirPacket) {
is RtcpFbPliPacket -> {
sourceSsrc = pliOrFirPacket.mediaSourceSsrc
canSend = canSendKeyframeRequest(packetInfo.endpointId, sourceSsrc, now)
forward = canSend && streamInformationStore.supportsPli
if (forward) numPlisForwarded++
if (!canSend) numPlisDropped++
}
is RtcpFbFirPacket -> {
sourceSsrc = pliOrFirPacket.mediaSenderSsrc
canSend = canSendKeyframeRequest(packetInfo.endpointId, sourceSsrc, now)
// When both are supported, we favor generating a PLI rather than forwarding a FIR
forward = canSend && streamInformationStore.supportsFir && !streamInformationStore.supportsPli
if (forward) {
// When we forward a FIR we need to update the seq num.
pliOrFirPacket.seqNum = firCommandSequenceNumber.incrementAndGet()
// We manage the seq num space, so we should use the same SSRC
localSsrc?.let { pliOrFirPacket.mediaSenderSsrc = it }
numFirsForwarded++
}
if (!canSend) numFirsDropped++
}
// This is not possible, but the compiler doesn't know it.
else -> throw IllegalStateException("Packet is neither PLI nor FIR")
}
if (!forward && canSend) {
doRequestKeyframe(sourceSsrc)
}
return if (forward) packetInfo else null
}
/**
* Returns 'true' when at least one method is supported, AND this requester hasn't sent a request very recently.
*/
private fun canSendKeyframeRequest(requesterID: String?, mediaSsrc: Long, now: Instant): Boolean {
if (!streamInformationStore.supportsPli && !streamInformationStore.supportsFir) {
return false
}
synchronized(keyframeLimiterSyncRoot) {
/* A null requesterID is a dominant speaker switch, or a request relayed from another bridge (relayed
* RTCP carries no endpoint id). There is no receiver to attribute it to, so skip only the per-receiver
* limit; the source-wide limit still applies, since it is what protects the sender's encoder and this
* is the only bridge that sees every requester for the source. */
val perReceiverLimiter = requesterID?.let { requester ->
perReceiverKeyframeLimiter.computeIfAbsent(requester) { mutableMapOf() }
.computeIfAbsent(mediaSsrc) {
RateLimit(
defaultMinInterval = minInterval,
maxRequests = maxRequests,
interval = maxRequestInterval
)
}
}
if (perReceiverLimiter != null && !perReceiverLimiter.wouldAccept(now, waitInterval)) {
numRequestsDroppedPerReceiverLimit++
logger.cdebug {
"Ignoring keyframe request for $mediaSsrc from $requesterID, per-receiver rate limited"
}
return false
}
val perSourceLimiter = perSourceKeyframeLimiter.computeIfAbsent(mediaSsrc) {
RateLimit(
defaultMinInterval = sourceWideMinInterval,
maxRequests = sourceWideMaxRequests,
interval = sourceWideMaxRequestInterval
)
}
if (!perSourceLimiter.wouldAccept(now, sourceWideInterval())) {
numRequestsDroppedSourceWideLimit++
logger.cdebug { "Ignoring keyframe request for $mediaSsrc from $requesterID, per-source rate limited" }
return false
}
/* Both limits accept, so record the request with both only now. A receiver waiting for a keyframe
* re-requests on every packet, so if requests dropped by the source-wide limit counted against its
* per-receiver limit it would exhaust that limit while the source-wide one is closed, and then be unable
* to request again for max-request-interval after the source-wide limit reopens. */
perReceiverLimiter?.record(now)
perSourceLimiter.record(now)
logger.cdebug { "Keyframe requester requesting keyframe for $mediaSsrc, requested by $requesterID" }
return true
}
}
/**
* The minimum interval to enforce between keyframe requests for a source, from any receiver. [waitInterval] is
* derived from the per-receiver min-interval and the RTT, so it is never longer than the per-receiver interval;
* take the larger of the two so that source-wide-min-interval is actually enforced.
*/
private fun sourceWideInterval(): Duration = maxOf(waitInterval, sourceWideMinInterval)
fun requestKeyframe(requesterID: String?, mediaSsrc: Long? = null) {
val ssrc = mediaSsrc ?: streamInformationStore.primaryMediaSsrcs.firstOrNull() ?: run {
numApiRequestsDropped++
logger.cdebug { "No video SSRC found to request keyframe" }
return
}
numApiRequests++
if (!canSendKeyframeRequest(requesterID, ssrc, clock.instant())) {
numApiRequestsDropped++
return
}
doRequestKeyframe(ssrc)
}
private fun doRequestKeyframe(mediaSsrc: Long) {
val pkt = when {
streamInformationStore.supportsPli -> {
numPlisGenerated++
RtcpFbPliPacketBuilder(mediaSourceSsrc = mediaSsrc).build()
}
streamInformationStore.supportsFir -> {
numFirsGenerated++
RtcpFbFirPacketBuilder(
mediaSenderSsrc = mediaSsrc,
firCommandSeqNum = firCommandSequenceNumber.incrementAndGet()
).build()
}
else -> {
logger.warn("Can not send neither PLI nor FIR")
return
}
}
next(PacketInfo(pkt))
}
override fun handleEvent(event: Event) {
when (event) {
is SetLocalSsrcEvent -> {
if (event.mediaType == MediaType.VIDEO) {
localSsrc = event.ssrc
}
}
}
}
override fun trace(f: () -> Unit) = f.invoke()
override fun getNodeStats(): NodeStatsBlock = super.getNodeStats().apply {
addNumber("wait_interval_ms", waitInterval.toMillis())
addNumber("num_api_requests", numApiRequests)
addNumber("num_api_requests_dropped", numApiRequestsDropped)
addNumber("num_firs_dropped", numFirsDropped)
addNumber("num_firs_generated", numFirsGenerated)
addNumber("num_firs_forwarded", numFirsForwarded)
addNumber("num_plis_dropped", numPlisDropped)
addNumber("num_plis_generated", numPlisGenerated)
addNumber("num_plis_forwarded", numPlisForwarded)
addNumber("num_requests_dropped_per_receiver_limit", numRequestsDroppedPerReceiverLimit)
addNumber("num_requests_dropped_source_wide_limit", numRequestsDroppedSourceWideLimit)
}
override fun statsJson() = super.statsJson().apply {
put("num_api_requests", numApiRequests)
put("num_api_requests_dropped", numApiRequestsDropped)
put("num_firs_dropped", numFirsDropped)
put("num_firs_generated", numFirsGenerated)
put("num_firs_forwarded", numFirsForwarded)
put("num_plis_dropped", numPlisDropped)
put("num_plis_generated", numPlisGenerated)
put("num_plis_forwarded", numPlisForwarded)
put("num_requests_dropped_per_receiver_limit", numRequestsDroppedPerReceiverLimit)
put("num_requests_dropped_source_wide_limit", numRequestsDroppedSourceWideLimit)
}
fun onRttUpdate(newRtt: Double) {
// avg(rtt) + stddev(rtt) would be more accurate than rtt + 10.
waitInterval = min(minInterval, durationOfDoubleSeconds((newRtt + 10) / 1e3))
}
companion object {
private val minInterval: Duration by config {
"jmt.keyframe.min-interval".from(JitsiConfig.newConfig)
}
private val maxRequests: Int by config {
"jmt.keyframe.max-requests".from(JitsiConfig.newConfig)
}
private val maxRequestInterval: Duration by config {
"jmt.keyframe.max-request-interval".from(JitsiConfig.newConfig)
}
private val sourceWideMinInterval: Duration by config {
"jmt.keyframe.source-wide-min-interval".from(JitsiConfig.newConfig)
}
private val sourceWideMaxRequests: Int by config {
"jmt.keyframe.source-wide-max-requests".from(JitsiConfig.newConfig)
}
private val sourceWideMaxRequestInterval: Duration by config {
"jmt.keyframe.source-wide-max-request-interval".from(JitsiConfig.newConfig)
}
}
}
private fun PacketInfo.getPliOrFirPacket(): RtcpFbPacket? = when (val pkt = packet) {
// We intentionally ignore compound RTCP packets in order to avoid unnecessary parsing. We can do this because:
// 1. Compound packets coming from remote endpoint are terminated in RtcpTermination
// 2. Whenever a PLI or FIR is generated in our code, it is not part of a compound packet.
is RtcpFbFirPacket -> pkt
is RtcpFbPliPacket -> pkt
else -> null
}

104
jitsi-media-transform/src/main/kotlin/org/jitsi/nlj/rtcp/NackHandler.kt Normal file
View File

@@ -0,0 +1,104 @@
/*
* Copyright @ 2018 - Present, 8x8 Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.jitsi.nlj.rtcp
import org.jitsi.nlj.PacketHandler
import org.jitsi.nlj.PacketInfo
import org.jitsi.nlj.PacketOrigin
import org.jitsi.nlj.stats.EndpointConnectionStats
import org.jitsi.nlj.stats.NodeStatsBlock
import org.jitsi.nlj.transform.NodeStatsProducer
import org.jitsi.nlj.util.BufferPool
import org.jitsi.nlj.util.PacketCache
import org.jitsi.rtp.rtcp.RtcpPacket
import org.jitsi.rtp.rtcp.rtcpfb.transport_layer_fb.RtcpFbNackPacket
import org.jitsi.rtp.rtp.RtpPacket
import org.jitsi.utils.logging2.Logger
import org.jitsi.utils.logging2.cdebug
import org.jitsi.utils.logging2.createChildLogger
import java.time.Instant
/**
* When a nack packet is received, the [NackHandler] will try to retrieve the
* nacked packets from the cache and then send them to the RTX output pipeline.
*/
class NackHandler(
private val packetCache: PacketCache,
private val onNackedPacketsReady: PacketHandler,
parentLogger: Logger
) : NodeStatsProducer, RtcpListener, EndpointConnectionStats.EndpointConnectionStatsListener {
private var numNacksReceived = 0
private var numNackedPackets = 0
private var numRetransmittedPackets = 0
private var numPacketsNotResentDueToDelay = 0
private var numCacheMisses = 0
private val logger = createChildLogger(parentLogger)
private var currRtt: Double = -1.0
override fun rtcpPacketReceived(packet: RtcpPacket, receivedTime: Instant?) {
if (packet is RtcpFbNackPacket) {
onNackPacket(packet)
}
}
private fun onNackPacket(nackPacket: RtcpFbNackPacket) {
logger.cdebug { "Nack received for ${nackPacket.mediaSourceSsrc} ${nackPacket.missingSeqNums}" }
val now = System.currentTimeMillis()
numNacksReceived++
val nackedPackets = mutableListOf<RtpPacket>()
val ssrc = nackPacket.mediaSourceSsrc
numNackedPackets += nackPacket.missingSeqNums.size
nackPacket.missingSeqNums.forEach { missingSeqNum ->
packetCache.get(ssrc, missingSeqNum)?.let { container ->
val delay = now - container.timeAdded
val shouldResendPacket =
(currRtt == -1.0) ||
(delay >= Math.min(currRtt * .9, currRtt - 5))
if (shouldResendPacket) {
// The cache returns a null container on failure, never a container with a null packet. Maybe we
// can refactor to make it explicit.
nackedPackets.add(container.item!!)
packetCache.updateTimestamp(ssrc, missingSeqNum, now)
numRetransmittedPackets++
} else {
BufferPool.returnBuffer(container.item!!.buffer)
numPacketsNotResentDueToDelay++
}
} ?: run {
logger.cdebug { "Nack'd packet $ssrc $missingSeqNum wasn't in cache, unable to retransmit" }
numCacheMisses++
}
}
nackedPackets.forEach {
val packetInfo = PacketInfo(it)
packetInfo.packetOrigin = PacketOrigin.Retransmission
onNackedPacketsReady.processPacket(packetInfo)
}
}
override fun onRttUpdate(newRttMs: Double) {
currRtt = newRttMs
}
override fun getNodeStats(): NodeStatsBlock = NodeStatsBlock("Nack handler").apply {
addNumber("num_nack_packets_received", numNackedPackets)
addNumber("num_nacked_packets", numNackedPackets)
addNumber("num_retransmitted_packets", numRetransmittedPackets)
addNumber("num_packets_not_retransmitted", numPacketsNotResentDueToDelay)
addNumber("num_cache_misses", numCacheMisses)
}
}

Some files were not shown because too many files have changed in this diff Show More