# SSRC rewriting principles for VP8 simulcast/SVC WebRTC has no notion of [simulcast][simulcast] when receiving a video stream, and it is therefore critical that any switch made by the SFU must be completely transparent to the receiver [1]. 1) In the case of sending an RTP stream without using picture ids the SFU has to rewrite the RTP sequence numbers so that the stream looks continuous, and rewrite the RTP timestamps so that there is no significant jump compared to the last frame. The timestamp may not jump backward. 2) If picture ids are used, besides following the steps outlined in 1), the SFU also has to rewrite the picture ids to make them continuous. Note that it is not possible to suddenly drop picture ids from the stream, as the receiver only keeps state relevant to picture ids, if they are used and therefore won't fall back to RTP sequence numbers. 3) If picture ids, tl0 picture indexes and temporal indexes are used, besides following the steps outlined in 2), the SFU also has to rewrite the tl0 picture indexes. Note that it is possible to drop the tl0 picture indexes and the temporal indexes from the stream, but if they are used again the tl0 picture index has to continue from where it left off. Also note that the state of the tl0 picture indexes is only updated by the receiver when both the tl0 picture index and temporal index is received. # Analysis With simple simulcast, the SFU simply needs to apply a fixed delta (i.e. linear translation) to the RTP sequence numbers/timestamps [4] and VP8 picture IDs/tl0picidx [3] that it forwards. The specific deltas are determined when the stream switching occurs. This straightforward approach doesn't work well if the SFU wants to leverage VP8 temporal scalability. We start our analysis by exploring the implications of the first guideline and, as we will see, 2 and 3 follow naturally. With temporal scalability in VP8 (and other codecs)[2], where the SFU can drop frames to achieve lower frame rates, applying a fixed delta would leave gaps in the sequence numbers (because of the packets that the SFU drops). So the SFU needs to keep track of what it's sent and manage the sequence number space of the SFU-receiver leg (i.e. generate sequence numbers for every egress packet of a frame). This mode of operation has several implications. First, it means that if the SFU decides to skip a frame for whatever reason, it cannot go back and change that decision because there would be no space left in the sequence numbers. Furthermore, if packet re-ordering or loss has occurred, a decision must be made immediately as to how much of a gap to leave in the sequence numbers so delayed packets have a place to be transmitted. When a frame arrives and it is to be forwarded, the SFU needs to determine how to assign its projected sequence numbers. Decisions on how to assign sequence numbers are done by comparing the _new_ frame with an _existing_ (previously-projected) frame. (In most cases this existing frame will be the previous projected frame, in sequence number ordering to the new frame, though in some corner cases involving very old frames a different existing frame is chosen.) If there is a gap in receive sequence numbers between the existing and the new frame, the SFU can determine the worst-case gap size of projected sequence numbers to leave such that any not-yet-received packets can be forwarded correctly. However, the SFU can't (in general) know the temporal layer of frames it has not yet received. If it turns out that the not-yet-received frames were in fact not ones that the SFU wanted to forward, this will result in sequence number gaps in the projected packet sequence, which the receiver may interpret as packet loss. (Care must be taken in bandwidth estimation algorithms not to incorrectly reduce bandwidth based on this "false" packet loss; in particular, this means that senders operating on receiver-side REMB-based bandwidth estimation mustn't adjust the estimates based on received loss.) # Examples of projection ## Scenario A: Happy path (target TL0) Sender sends: TL0 (1, 2, 3), TL2 (4, 5), TL1 (6, 7), TL2 (8), TL0 (9, 10, 11) Bridge receives: what the sender sends Bridge sends: 101, 102, 103, (9 -> 104), (10 -> 105), (11 -> 106) When packet 9 arrives, the bridge can determine that sequence numbers 4 - 8 belong to non-routed packets and can be suppressed ## Scenario B: packet loss (target TL2) Sender sends: TL0 (1, 2, 3), TL2 (4, 5), TL1 (6, 7), TL2 (8), TL0 (9, 10, 11) Bridge receives: 1, 2, 3, 4, _(X)_, 6, 7, 8, 9, 10, 11, 5 Bridge sends: 101, 102, 103, 104, 106, 107, 108, 109, 110, 111, 105 When packet 6 arrives, the bridge computes the gap from the previously projected frame. Because it wants to route both the TL2 and TL1 frames, it leaves a gap in the outgoing sequence numbers, into which packet 5 can later be slotted. ## Scenario C: packet loss 1 (target TL0) Sender sends: TL0 (1, 2, 3), TL2 (4, 5), TL1 (6, 7), TL2 (8), TL0 (9, 10, 11) Bridge receives: 1, 2, 3, 4, _(X)_, 6, 7, 8, 9, 10, 11, 5 Bridge sends: 101, 102, 103, (9 -> 104), (10 -> 105), (11 -> 106) Similarly to Scenario A, when packet 9 arrives, the bridge can determine that sequence numbers 4-8 belong to non-routed packets. The status of the unreceived packet 5 can be determined because packet 4 will not have its "end of frame" flag set. ## Scenario D: packet loss 2 (target TL1) Sender sends: TL0 (1, 2, 3), TL2 (4, 5), TL1 (6, 7), TL2 (8), TL0 (9, 10, 11) Bridge receives: 1, 2, 3, 4, 5, 6, 7, _(X)_, 9, 10, 11, 8 Bridge sends: 101, 102, 103, (6 -> 104), (7 -> 105), (9 -> 107), (10 -> 108), (11 -> 109) In this case, when packet 9 arrives, the bridge cannot determine what type of frame packet 8 was -- it could have been a TL1. Therefore it must leave a gap in the outgoing sequence numbers. [simulcast]: https://ieeexplore.ieee.org/abstract/document/7992929 [1]: https://groups.google.com/d/topic/discuss-webrtc/gik2VH4hUjk/discussion [2]: https://webrtchacks.com/sfu-simulcast/ [3]: https://tools.ietf.org/html/rfc7741 [4]: https://tools.ietf.org/html/rfc3550