【WebRTC】视频发送链路中类的简单分析（上）
目录
- 1.视频流发送（VideoSendStream）
-
- 1.1 质量限制（QualityLimitationReason）
- 1.2 RTP配置项（RtpConfig）
- 1.3 视频流编码器配置项（VideoStreamEncoderSettings）
- 1.4 传输（Transport）
- 2.视频流发送实现（VideoSendStreamImpl）
-
- 2.1 码率分配监测器（BitrateAllocatorObserver）
- 2.2 编码接收器（VideoStreamEncoderInterface::EncoderSink）
- 3.RTP视频发送器（RtpVideoSender)
- 4.RTP发送视频（RTPSenderVideo）
-
- 4.1 RTPSenderVideo::SendEncodedImage()
- 4.2 RTPSenderVideo::SendVideo
- 4.3 RTPSenderVideo::LogAndSendToNetwork
- 5.RTP发送器（RTPSender）
视频流在采集编码之后，需要通过RTP协议进行发送，下面记录与视频流发送相关的一些类的声明，梳理视频流发送过程中，不同类的继承关系
1.视频流发送（VideoSendStream）
WebRTC中将已编码视频帧传送出去的类为VideoSendStream，其声明位于call/video_send_stream.h中，这个类声明的初衷为一个基类，其中声明了一系列视频流发送的纯虚函数，以及视频流发送状态和配置相关的变量。大致的内容包括：
（1）视频流的状态（类型，长宽，延时，帧率，帧数，码率等）
（2）发送器状态（编码实例名称，输入帧率，编码帧率，帧数，丢帧数，质量限制等）
（3）发送器配置项（RTP配置项，数据包传输，带宽测试等）
（4）发送器函数（发送，停止，添加资源，生成关键帧）
视频流的类型分为3类，kMedia是使用RTP协议传输的正常流，kRtx（Retransmission）是重传流，kFlexfec专门用于Flexfec。视频流状态当中的帧相关信息，与发送器状态当中的帧相关不完全一致，因为发送器还要考虑带宽，拥塞控制等问题，可能会进行丢包或质量限制，二者应该是在动态变化中形成一致。
视频发送的流程大致为：配置发送器，获取已编码帧信息，开启发送器，发送帧，关闭发送器
class VideoSendStream {
public:
// Multiple StreamStats objects are present if simulcast is used (multiple
// kMedia streams) or if RTX or FlexFEC is negotiated. Multiple SVC layers, on
// the other hand, does not cause additional StreamStats.
/*
如果使用了Simulcast（多个kMedia流），或者协商了RTX或FlexFEC，
就会出现多个StreamStats对象。另一方面，多个SVC层并不会导致额外的StreamStats
PS:
Simulcast表示可以对外传输多个视频流，每个视频流的分辨率和码率不同，以适应不同的用户群体或网络条件
其实现方式为，使用多个编码实例对同一个视频源，按照不同编码策略进行编码，每条流都有独特的SSRC
*/
struct StreamStats {
enum class StreamType {
// A media stream is an RTP stream for audio or video. Retransmissions and
// FEC is either sent over the same SSRC or negotiated to be sent over
// separate SSRCs, in which case separate StreamStats objects exist with
// references to this media stream's SSRC.
/*
一个媒体流是用于音频或视频的RTP流。重传和FEC（前向纠错）可以通过相同的SSRC发送，
或者协商后通过不同的SSRC发送，在后一种情况下，会存在引用这个媒体流SSRC的独立StreamStats对象
*/
kMedia,
// RTX streams are streams dedicated to retransmissions. They have a
// dependency on a single kMedia stream: `referenced_media_ssrc`.
/*
RTX流是专门用于重传的流。它们依赖于单个kMedia流：`referenced_media_ssrc`
*/
kRtx,
// FlexFEC streams are streams dedicated to FlexFEC. They have a
// dependency on a single kMedia stream: `referenced_media_ssrc`.
/*
FlexFEC流是专门用于FlexFEC的流。它们依赖于单个kMedia流：`referenced_media_ssrc`
*/
kFlexfec,
};
StreamStats();
~StreamStats();
std::string ToString() const;
StreamType type = StreamType::kMedia;
// If `type` is kRtx or kFlexfec this value is present. The referenced SSRC
// is the kMedia stream that this stream is performing retransmissions or
// FEC for. If `type` is kMedia, this value is null.
/*
如果`type`是kRtx或kFlexfec，这个值是存在的。被引用的SSRC是这个流正在执行重传
或FEC的kMedia流。如果`type`是kMedia，这个值是空的
*/
std::optional<uint32_t> referenced_media_ssrc;
// 帧数量
FrameCounts frame_counts;
// 宽度
int width = 0;
// 高度
int height = 0;
// TODO(holmer): Move bitrate_bps out to the webrtc::Call layer.
// 整体码率
int total_bitrate_bps = 0;
// 重传比特率
int retransmit_bitrate_bps = 0;
// `avg_delay_ms` and `max_delay_ms` are only used in tests. Consider
// deleting.
// 平均延时
int avg_delay_ms = 0;
// 最大延时
int max_delay_ms = 0;
// 用于RTP数据统计
StreamDataCounters rtp_stats;
// 用于RTCP包类型的数据统计
RtcpPacketTypeCounter rtcp_packet_type_counts;
// A snapshot of the most recent Report Block with additional data of
// interest to statistics. Used to implement RTCRemoteInboundRtpStreamStats.
// 一个包含对统计数据感兴趣的额外信息的最新报告块的快照。用于实现`RTCRemoteInboundRtpStreamStats`
std::optional<ReportBlockData> report_block_data;
// 编码帧率
double encode_frame_rate = 0.0;
// 已编码帧
int frames_encoded = 0;
std::optional<uint64_t> qp_sum;
// 整体编码耗时，以ms为单位
uint64_t total_encode_time_ms = 0;
// 总体已编码比特目标
uint64_t total_encoded_bytes_target = 0;
uint32_t huge_frames_sent = 0;
std::optional<ScalabilityMode> scalability_mode;
};
// 下面是一些发送视频流的状态信息
struct Stats {
Stats();
~Stats();
std::string ToString(int64_t time_ms) const;
// 编码实例名称
std::optional<std::string> encoder_implementation_name;
// 输入帧率
double input_frame_rate = 0;
// 编码帧率
int encode_frame_rate = 0;
// 平均编码耗时，以ms为单位
int avg_encode_time_ms = 0;
// 编码器使用率百分比
int encode_usage_percent = 0;
// 已编码帧数量
uint32_t frames_encoded = 0;
// https://w3c.github.io/webrtc-stats/#dom-rtcoutboundrtpstreamstats-totalencodetime
// 总体编码耗时，以ms为单位
uint64_t total_encode_time_ms = 0;
// https://w3c.github.io/webrtc-stats/#dom-rtcoutboundrtpstreamstats-totalencodedbytestarget
// 编码器目标总比特数
uint64_t total_encoded_bytes_target = 0;
// 帧数
uint32_t frames = 0;
// 由于采集问题而丢帧的数量
uint32_t frames_dropped_by_capturer = 0;
// 由于时间戳问题而丢帧的数量
uint32_t frames_dropped_by_bad_timestamp = 0;
// 由于编码队列太长而丢帧的数量
uint32_t frames_dropped_by_encoder_queue = 0;
// 由于码率限制而丢帧的数量
uint32_t frames_dropped_by_rate_limiter = 0;
// 由于拥塞窗口而丢帧的数量
uint32_t frames_dropped_by_congestion_window = 0;
// 由于编码器问题而丢帧的数量
uint32_t frames_dropped_by_encoder = 0;
// Bitrate the encoder is currently configured to use due to bandwidth
// limitations.
// 由于带宽限制，编码器当前配置使用的比特率
int target_media_bitrate_bps = 0;
// Bitrate the encoder is actually producing.
// 由编码器实际产生的码率
int media_bitrate_bps = 0;
// 流是否暂停
bool suspended = false;
// 是否由于bandwidth而降低分辨率
bool bw_limited_resolution = false;
// 是否由于CPU而降低分辨率
bool cpu_limited_resolution = false;
// 是否由于bandwidth而降低帧率
bool bw_limited_framerate = false;
// 是否由于CPU而减低帧率
bool cpu_limited_framerate = false;
// https://w3c.github.io/webrtc-stats/#dom-rtcoutboundrtpstreamstats-qualitylimitationreason
// 质量限制原因
QualityLimitationReason quality_limitation_reason =
QualityLimitationReason::kNone;
// https://w3c.github.io/webrtc-stats/#dom-rtcoutboundrtpstreamstats-qualitylimitationdurations
// 记录由于不同原因导致质量下降的持续时间
std::map<QualityLimitationReason, int64_t> quality_limitation_durations_ms;
// https://w3c.github.io/webrtc-stats/#dom-rtcoutboundrtpstreamstats-qualitylimitationresolutionchanges
// 记录由于质量限制而导致分辨率变化的次数
uint32_t quality_limitation_resolution_changes = 0;
// Total number of times resolution as been requested to be changed due to
// CPU/quality adaptation.
// 由于CPU/质量适应，请求改变分辨率的总次数
int number_of_cpu_adapt_changes = 0;
int number_of_quality_adapt_changes = 0;
// 是否进入了低分辨率状态
bool has_entered_low_resolution = false;
// 子流
std::map<uint32_t, StreamStats> substreams;
// 内容的类型
webrtc::VideoContentType content_type =
webrtc::VideoContentType::UNSPECIFIED;
// 发送出去的视频帧总数
uint32_t frames_sent = 0;
// 发送出去的“巨大”视频帧的数量
uint32_t huge_frames_sent = 0;
// 是否处于节能模式
std::optional<bool> power_efficient_encoder;
};
// 下面是一些配置项
struct Config {
public:
Config() = delete;
Config(Config&&);
explicit Config(Transport* send_transport);
Config& operator=(Config&&);
Config& operator=(const Config&) = delete;
~Config();
// Mostly used by tests. Avoid creating copies if you can.
Config Copy() const { return Config(*this); }
std::string ToString() const;
// RTP配置项
RtpConfig rtp;
// 编码器设置
VideoStreamEncoderSettings encoder_settings;
// Time interval between RTCP report for video
// 视频RTCP报告的时间间隔
int rtcp_report_interval_ms = 1000;
// Transport for outgoing packets.
// 出站数据包的传输
Transport* send_transport = nullptr;
// Expected delay needed by the renderer, i.e. the frame will be delivered
// this many milliseconds, if possible, earlier than expected render time.
// Only valid if `local_renderer` is set.
/*
渲染器所需的预期延迟，即如果可能的话，帧将比预期渲染时间提前这么多毫秒交付。
只有当设置了`local_renderer`时，此值才有效。
*/
int render_delay_ms = 0;
// Target delay in milliseconds. A positive value indicates this stream is
// used for streaming instead of a real-time call.
// 目标延迟时间，以毫秒为单位。正值表示此流用于流媒体传输，而不是实时通话
int target_delay_ms = 0;
// True if the stream should be suspended when the available bitrate fall
// below the minimum configured bitrate. If this variable is false, the
// stream may send at a rate higher than the estimated available bitrate.
// 如果可用比特率下降到最低配置的比特率以下时应该暂停流，则为真
// 如果这个变量为假，流可能会以高于估计的可用比特率的速度发送
bool suspend_below_min_bitrate = false;
// Enables periodic bandwidth probing in application-limited region.
// 启用在应用受限区域的周期性带宽探测
bool periodic_alr_bandwidth_probing = false;
// An optional custom frame encryptor that allows the entire frame to be
// encrypted in whatever way the caller chooses. This is not required by
// default.
// 一个可选的自定义帧加密器，允许整个帧按照调用者选择的任何方式进行加密。这不是默认必需的
rtc::scoped_refptr<webrtc::FrameEncryptorInterface> frame_encryptor;
// An optional encoder selector provided by the user.
// Overrides VideoEncoderFactory::GetEncoderSelector().
// Owned by RtpSenderBase.
/*
用户提供的一个可选编码器选择器。
覆盖了 VideoEncoderFactory::GetEncoderSelector()。
由 RtpSenderBase 拥有。
*/
VideoEncoderFactory::EncoderSelectorInterface* encoder_selector = nullptr;
// Per PeerConnection cryptography options.
// 每个PeerConnection的加密选项
CryptoOptions crypto_options;
// 帧转换器
rtc::scoped_refptr<webrtc::FrameTransformerInterface> frame_transformer;
private:
// Access to the copy constructor is private to force use of the Copy()
// method for those exceptional cases where we do use it.
Config(const Config&);
};
// Starts stream activity.
// When a stream is active, it can receive, process and deliver packets.
// 开启一个流，此时流可以进行接收，处理和传输数据包
virtual void Start() = 0;
// Stops stream activity.
// When a stream is stopped, it can't receive, process or deliver packets.
// 停止一个流，此时流不能接收，处理和传输数据包
virtual void Stop() = 0;
// Accessor for determining if the stream is active. This is an inexpensive
// call that must be made on the same thread as `Start()` and `Stop()` methods
// are called on and will return `true` iff activity has been started
// via `Start()`.
// 用于确定流是否处于活动状态的访问器。这是一个低成本的调用，必须在与`Start()`和
// `Stop()`方法调用相同的线程上进行，并且仅当通过`Start()`启动了活动时才会返回`true`
virtual bool started() = 0;
// If the resource is overusing, the VideoSendStream will try to reduce
// resolution or frame rate until no resource is overusing.
// TODO(https://crbug.com/webrtc/11565): When the ResourceAdaptationProcessor
// is moved to Call this method could be deleted altogether in favor of
// Call-level APIs only.
// 如果资源使用过量，VideoSendStream将尝试降低分辨率或帧率，直到没有资源使用过量
virtual void AddAdaptationResource(rtc::scoped_refptr<Resource> resource) = 0;
virtual std::vector<rtc::scoped_refptr<Resource>>
GetAdaptationResources() = 0;
virtual void SetSource(
rtc::VideoSourceInterface<webrtc::VideoFrame>* source,
const DegradationPreference& degradation_preference) = 0;
// Set which streams to send. Must have at least as many SSRCs as configured
// in the config. Encoder settings are passed on to the encoder instance along
// with the VideoStream settings.
// 设置要发送的流。必须至少有与配置中配置的一样多的SSRCs。编码器设置会与VideoStream设置一起传递给编码器实例
virtual void ReconfigureVideoEncoder(VideoEncoderConfig config) = 0;
virtual void ReconfigureVideoEncoder(VideoEncoderConfig config,
SetParametersCallback callback) = 0;
// 获取数据状态
virtual Stats GetStats() = 0;
// 生成关键帧
virtual void GenerateKeyFrame(const std::vector<std::string>& rids) = 0;
protected:
virtual ~VideoSendStream() {}
};
1.1 质量限制（QualityLimitationReason）
在VideoSendStream中，记录了由于质量限制的原因，QualityLimitationReason的声明如下，主要是因为CPU和带宽两个原因，可能会产生分辨率或帧率的调整。声明位于common_video/include/quality_limitation_reason.h中
enum class QualityLimitationReason {
kNone,
kCpu, // CPU原因
kBandwidth, // 带宽原因
kOther,
};
1.2 RTP配置项（RtpConfig）
RtpConfig的声明位于call/rtp_config.h中，其中声明了一些RTP的配置项，是一个综合类，其下还包括了很多的嵌套类
// RTP最大Size
static const size_t kDefaultMaxPacketSize = 1500 - 40; // TCP over IPv4.
struct RtpConfig {
RtpConfig();
RtpConfig(const RtpConfig&);
~RtpConfig();
std::string ToString() const;
// ssrc
std::vector<uint32_t> ssrcs;
// The Rtp Stream Ids (aka RIDs) to send in the RID RTP header extension
// if the extension is included in the list of extensions.
// If rids are specified, they should correspond to the `ssrcs` vector.
// This means that:
// 1. rids.size() == 0 || rids.size() == ssrcs.size().
// 2. If rids is not empty, then `rids[i]` should use `ssrcs[i]`.
// RTP流ID，放在RTP头扩展
std::vector<std::string> rids;
// The value to send in the MID RTP header extension if the extension is
// included in the list of extensions.
std::string mid;
// See RtcpMode for description.
// RTCP模式
RtcpMode rtcp_mode = RtcpMode::kCompound;
// Max RTP packet size delivered to send transport from VideoEngine.
// 最大packet大小
size_t max_packet_size = kDefaultMaxPacketSize;
// Corresponds to the SDP attribute extmap-allow-mixed.
bool extmap_allow_mixed = false;
// RTP header extensions to use for this send stream.
// 用于发送流的RTP头部扩展信息
std::vector<RtpExtension> extensions;
// TODO(nisse): For now, these are fixed, but we'd like to support
// changing codec without recreating the VideoSendStream. Then these
// fields must be removed, and association between payload type and codec
// must move above the per-stream level. Ownership could be with
// RtpTransportControllerSend, with a reference from RtpVideoSender, where
// the latter would be responsible for mapping the codec type of encoded
// images to the right payload type.
/*
目前，这些是固定的，但我们希望在不重新创建VideoSendStream的情况下支持更改编解码器。
然后这些字段必须被移除，并且载荷类型和编解码器之间的关联必须移到每个流的级别之上。
所有权可以归RtpTransportControllerSend所有，RtpVideoSender持有一个引用，后者
负责将编码图像的编解码器类型映射到正确的载荷类型。
*/
std::string payload_name;
// 负载类型
int payload_type = -1;
// Payload should be packetized using raw packetizer (payload header will
// not be added, additional meta data is expected to be present in generic
// frame descriptor RTP header extension).
bool raw_payload = false;
// Configurations for each RTP stream
// 每个RTP流的配置信息
std::vector<RtpStreamConfig> stream_configs;
// See LntfConfig for description.
LntfConfig lntf;
// See NackConfig for description.
NackConfig nack;
// See UlpfecConfig for description.
UlpfecConfig ulpfec;
// Flex前向纠错
struct Flexfec {
Flexfec();
Flexfec(const Flexfec&);
~Flexfec();
// Payload type of FlexFEC. Set to -1 to disable sending FlexFEC.
int payload_type = -1;
// SSRC of FlexFEC stream.
uint32_t ssrc = 0;
// Vector containing a single element, corresponding to the SSRC of the
// media stream being protected by this FlexFEC stream.
// The vector MUST have size 1.
//
// TODO(brandtr): Update comment above when we support
// multistream protection.
std::vector<uint32_t> protected_media_ssrcs;
} flexfec;
// Settings for RTP retransmission payload format, see RFC 4588 for
// details.
// 重传流
struct Rtx {
Rtx();
Rtx(const Rtx&);
~Rtx();
std::string ToString() const;
// SSRCs to use for the RTX streams.
std::vector<uint32_t> ssrcs;
// Payload type to use for the RTX stream.
int payload_type = -1;
} rtx;
// RTCP CNAME, see RFC 3550.
std::string c_name;
// Enables send packet batching from the egress RTP sender.
bool enable_send_packet_batching = false;
bool IsMediaSsrc(uint32_t ssrc) const;
bool IsRtxSsrc(uint32_t ssrc) const;
bool IsFlexfecSsrc(uint32_t ssrc) const;
std::optional<uint32_t> GetRtxSsrcAssociatedWithMediaSsrc(
uint32_t media_ssrc) const;
uint32_t GetMediaSsrcAssociatedWithRtxSsrc(uint32_t rtx_ssrc) const;
uint32_t GetMediaSsrcAssociatedWithFlexfecSsrc(uint32_t flexfec_ssrc) const;
std::optional<std::string> GetRidForSsrc(uint32_t ssrc) const;
};
1.3 视频流编码器配置项（VideoStreamEncoderSettings）
VideoStreamEncoderSettings的声明位于api/video/video_stream_encoder_settings.h中，声明了与编码器配置项有关的信息，如编码器切换，编码器能力等
struct VideoStreamEncoderSettings {
explicit VideoStreamEncoderSettings(
const VideoEncoder::Capabilities& capabilities)
: capabilities(capabilities) {}
// Enables the new method to estimate the cpu load from encoding, used for
// cpu adaptation.
// 从实验中估计CPU负载
bool experiment_cpu_load_estimator = false;
// Ownership stays with WebrtcVideoEngine (delegated from PeerConnection).
VideoEncoderFactory* encoder_factory = nullptr;
// Requests the WebRtcVideoChannel to perform a codec switch.
// 执行编码器切换的回调函数
EncoderSwitchRequestCallback* encoder_switch_request_callback = nullptr;
// Ownership stays with WebrtcVideoEngine (delegated from PeerConnection).
VideoBitrateAllocatorFactory* bitrate_allocator_factory = nullptr;
// Negotiated capabilities which the VideoEncoder may expect the other
// side to use.
// 编码能力
VideoEncoder::Capabilities capabilities;
// Enables the frame instrumentation generator that is required for automatic
// corruption detection.
bool enable_frame_instrumentation_generator = false;
};
1.4 传输（Transport）
Transport声明了RTP和RTCP的纯虚函数，其声明位于api/call/transport.h中
class Transport {
public:
virtual bool SendRtp(rtc::ArrayView<const uint8_t> packet,
const PacketOptions& options) = 0;
virtual bool SendRtcp(rtc::ArrayView<const uint8_t> packet) = 0;
protected:
virtual ~Transport() {}
};
2.视频流发送实现（VideoSendStreamImpl）
前面的VideoSendStream当中声明了一些视频流发送的纯虚函数，这些函数由VideoSendStreamImpl实现，VideoSendStreamImpl的声明位于video/video_send_stream_impl.h中，这个类在worker线程当中创建和释放。
这个函数实现的功能基本都是由其父类VideoSendStream声明的，但还添加了监视器和编码器接收端类（EncoderSink）
// VideoSendStreamImpl implements webrtc::VideoSendStream.
// It is created and destroyed on `worker queue`. The intent is to
// An encoder may deliver frames through the EncodedImageCallback on an
// arbitrary thread.
class VideoSendStreamImpl : public webrtc::VideoSendStream,
public webrtc::BitrateAllocatorObserver,
public VideoStreamEncoderInterface::EncoderSink {
public:
using RtpStateMap = std::map<uint32_t, RtpState>;
using RtpPayloadStateMap = std::map<uint32_t, RtpPayloadState>;
VideoSendStreamImpl(const Environment& env,
int num_cpu_cores,
RtcpRttStats* call_stats,
RtpTransportControllerSendInterface* transport,
Metronome* metronome,
BitrateAllocatorInterface* bitrate_allocator,
SendDelayStats* send_delay_stats,
VideoSendStream::Config config,
VideoEncoderConfig encoder_config,
const RtpStateMap& suspended_ssrcs,
const RtpPayloadStateMap& suspended_payload_states,
std::unique_ptr<FecController> fec_controller,
std::unique_ptr<VideoStreamEncoderInterface>
video_stream_encoder_for_test = nullptr);
~VideoSendStreamImpl() override;
// 处理RTCP包
void DeliverRtcp(const uint8_t* packet, size_t length);
// webrtc::VideoSendStream implementation.
// VideoSendStream的实现
void Start() override;
void Stop() override;
bool started() override;
// 添加自适应资源
void AddAdaptationResource(rtc::scoped_refptr<Resource> resource) override;
// 获取自适应资源
std::vector<rtc::scoped_refptr<Resource>> GetAdaptationResources() override;
// 设置资源
void SetSource(rtc::VideoSourceInterface<webrtc::VideoFrame>* source,
const DegradationPreference& degradation_preference) override;
// 重新配置视频编码器
void ReconfigureVideoEncoder(VideoEncoderConfig config) override;
void ReconfigureVideoEncoder(VideoEncoderConfig config,
SetParametersCallback callback) override;
// 获取发送器状态
Stats GetStats() override;
// 永久停止发送器并获取RTP状态
void StopPermanentlyAndGetRtpStates(RtpStateMap* rtp_state_map,
RtpPayloadStateMap* payload_state_map);
// 生成关键帧
void GenerateKeyFrame(const std::vector<std::string>& rids) override;
// TODO(holmer): Move these to RtpTransportControllerSend.
std::map<uint32_t, RtpState> GetRtpStates() const;
// 获取RTP负载状态
std::map<uint32_t, RtpPayloadState> GetRtpPayloadStates() const;
// 获取节奏因子参数
const std::optional<float>& configured_pacing_factor() const {
return configured_pacing_factor_;
}
private:
friend class test::VideoSendStreamPeer;
class OnSendPacketObserver : public SendPacketObserver {
public:
// 发送packet的监视器
OnSendPacketObserver(SendStatisticsProxy* stats_proxy,
SendDelayStats* send_delay_stats)
: stats_proxy_(*stats_proxy), send_delay_stats_(*send_delay_stats) {}
// 发送packet
void OnSendPacket(std::optional<uint16_t> packet_id,
Timestamp capture_time,
uint32_t ssrc) override {
stats_proxy_.OnSendPacket(ssrc, capture_time);
if (packet_id.has_value()) {
send_delay_stats_.OnSendPacket(*packet_id, capture_time, ssrc);
}
}
private:
SendStatisticsProxy& stats_proxy_;
SendDelayStats& send_delay_stats_;
};
std::optional<float> GetPacingFactorOverride() const;
// Implements BitrateAllocatorObserver.
// 更新Bitrate
uint32_t OnBitrateUpdated(BitrateAllocationUpdate update) override;
// 获取使用的速率
std::optional<DataRate> GetUsedRate() const override;
// Implements VideoStreamEncoderInterface::EncoderSink
// EncoderSink中的实现
// 编码器配置发生变化
void OnEncoderConfigurationChanged(
std::vector<VideoStream> streams,
bool is_svc,
VideoEncoderConfig::ContentType content_type,
int min_transmit_bitrate_bps) override;
// 码率分配变化
void OnBitrateAllocationUpdated(
const VideoBitrateAllocation& allocation) override;
// VideoLayers分配变化
void OnVideoLayersAllocationUpdated(
VideoLayersAllocation allocation) override;
// Implements EncodedImageCallback. The implementation routes encoded frames
// to the `payload_router_` and `config.pre_encode_callback` if set.
// Called on an arbitrary encoder callback thread.
// 编码结束后返回的回调函数，这个函数后续会调用数据包发送函数，将编码后的视频流发送出去
EncodedImageCallback::Result OnEncodedImage(
const EncodedImage& encoded_image,
const CodecSpecificInfo* codec_specific_info) override;
// Implements EncodedImageCallback.
// 丢帧回调
void OnDroppedFrame(EncodedImageCallback::DropReason reason) override;
// Starts monitoring and sends a keyframe.
// 开始发送视频流，发送一个关键帧
void StartupVideoSendStream();
// Removes the bitrate observer, stops monitoring and notifies the video
// encoder of the bitrate update.
// 停止视频流发送
void StopVideoSendStream() RTC_RUN_ON(thread_checker_);
// 配置视频流的保护策略，比如前向纠错（FEC）和重传（NACK）等
// 会根据当前的网络条件和编码器配置来决定是否需要启用这些保护机制，以及如何配置它们以优化视频传输的质量和可靠性
void ConfigureProtection();
// 配置视频流使用的同步源（SSRC）标识符
void ConfigureSsrcs();
// 通知编码器发生了超时，如果发生超时，可能需要重新配置编码器或调整参数
void SignalEncoderTimedOut();
// 用于标记编码器为活跃状态，这对于监控编码器性能和调整网络带宽分配非常重要
void SignalEncoderActive();
// A video send stream is running if VideoSendStream::Start has been invoked
// and there is an active encoding.
// 用于检查视频发送流是否正在运行
bool IsRunning() const;
// 获取当前视频流的媒体流分配配置
MediaStreamAllocationConfig GetAllocationConfig() const
RTC_RUN_ON(thread_checker_);
// 环境变量，能够通过这个变量访问其他全局的变量
const Environment env_;
// 检查某个方法是否在预期的线程上被调用
RTC_NO_UNIQUE_ADDRESS SequenceChecker thread_checker_;
// 管理RTP包的传输
RtpTransportControllerSendInterface* const transport_;
// 收集和报告发送统计信息
SendStatisticsProxy stats_proxy_;
// 观察和响应发送RTP包的事件
OnSendPacketObserver send_packet_observer_;
// 存储了视频发送流的配置，包括编码器设置、SSRC、CNAME等
const VideoSendStream::Config config_;
// 定义了视频内容的类型，比如是否是屏幕共享内容
const VideoEncoderConfig::ContentType content_type_;
// 负责视频流的编码工作
std::unique_ptr<VideoStreamEncoderInterface> video_stream_encoder_;
// 提供编码器的RTCP反馈信息
EncoderRtcpFeedback encoder_feedback_;
// 负责发送RTP视频包
RtpVideoSenderInterface* const rtp_video_sender_;
// 用于指示视频发送流是否正在运行
bool running_ RTC_GUARDED_BY(thread_checker_) = false;
/*
指示是否启用了“应用受限（Application Limited）”区域的探测（ALR Probing）。
ALR用于在网络带宽似乎不足以支持当前发送速率时，通过发送额外的数据包来探测网络的实际容量
*/
const bool has_alr_probing_;
// pacing参数
// Pacing是一种流量整形技术，用于平滑数据包的发送，避免网络拥塞，并确保数据包按照预定的速率发送
const PacingConfig pacing_config_;
// worker队列，常用于执行视频编码、数据包发送等后台任务
TaskQueueBase* const worker_queue_;
// 重复任务句柄，用于定期检查编码器的活动状态
RepeatingTaskHandle check_encoder_activity_task_
RTC_GUARDED_BY(thread_checker_);
// 用于线程安全地指示是否有活动正在进行
std::atomic_bool activity_;
// 用于指示编码器是否发生了超时
bool timed_out_ RTC_GUARDED_BY(thread_checker_);
// 码率分配接口
BitrateAllocatorInterface* const bitrate_allocator_;
// 是否有活跃的编码线程
bool has_active_encodings_ RTC_GUARDED_BY(thread_checker_);
// 是否禁用padding，Padding是网络传输中用于保持数据传输连续性的技术，通过发送无用数据来保持数据流的稳定性
bool disable_padding_ RTC_GUARDED_BY(thread_checker_);
// 最大padding码率
int max_padding_bitrate_ RTC_GUARDED_BY(thread_checker_);
// 编码器最小码率
int encoder_min_bitrate_bps_ RTC_GUARDED_BY(thread_checker_);
// 编码器最大码率
uint32_t encoder_max_bitrate_bps_ RTC_GUARDED_BY(thread_checker_);
// 编码器目标码率
uint32_t encoder_target_rate_bps_ RTC_GUARDED_BY(thread_checker_);
// 编码器优先码率
double encoder_bitrate_priority_ RTC_GUARDED_BY(thread_checker_);
// AV1的优先码率
const int encoder_av1_priority_bitrate_override_bps_
RTC_GUARDED_BY(thread_checker_);
ScopedTaskSafety worker_queue_safety_;
// Context for the most recent and last sent video bitrate allocation. Used to
// throttle sending of similar bitrate allocations.
// Video Bitrate Allocation的上下文
struct VbaSendContext {
VideoBitrateAllocation last_sent_allocation;
std::optional<VideoBitrateAllocation> throttled_allocation;
int64_t last_send_time_ms;
};
std::optional<VbaSendContext> video_bitrate_allocation_context_
RTC_GUARDED_BY(thread_checker_);
const std::optional<float> configured_pacing_factor_;
};
这个类实现视频流发送的接口由EncodedImageCallback这个回调接口实现，函数名为OnEncodedImage()，其实现方式如下，定义位于video/video_send_stream_impl.cc，当中主要的函数为rtp_video_sender_->OnEncodedImage()，进入到RTP层的发送逻辑当中。
EncodedImageCallback::Result VideoSendStreamImpl::OnEncodedImage(
const EncodedImage& encoded_image,
const CodecSpecificInfo* codec_specific_info) {
// Encoded is called on whatever thread the real encoder implementation run
// on. In the case of hardware encoders, there might be several encoders
// running in parallel on different threads.
// Indicate that there still is activity going on.
activity_ = true;
RTC_DCHECK(!worker_queue_->IsCurrent());
auto task_to_run_on_worker = [this]() {
RTC_DCHECK_RUN_ON(&thread_checker_);
if (disable_padding_) {
disable_padding_ = false;
// To ensure that padding bitrate is propagated to the bitrate allocator.
SignalEncoderActive();
}
// Check if there's a throttled VideoBitrateAllocation that we should try
// sending.
auto& context = video_bitrate_allocation_context_;
if (context && context->throttled_allocation) {
OnBitrateAllocationUpdated(*context->throttled_allocation);
}
};
worker_queue_->PostTask(
SafeTask(worker_queue_safety_.flag(), std::move(task_to_run_on_worker)));
// 调用RTPVideoSender的回调函数，进入底层RTP发送packet的逻辑当中
return rtp_video_sender_->OnEncodedImage(encoded_image, codec_specific_info);
}
2.1 码率分配监测器（BitrateAllocatorObserver）
BitrateAllocatorObserver的声明位于call/bitrate_allocator.h中，主要用于观察和响应比特率分配器（BitrateAllocator）的比特率更新。
// Used by all send streams with adaptive bitrate, to get the currently
// allocated bitrate for the send stream. The current network properties are
// given at the same time, to let the send stream decide about possible loss
// protection.
/*
被所有具有自适应比特率的发送流使用，以获取当前为发送流分配的比特率。
同时给出当前的网络属性，让发送流决定可能的丢包保护。
*/
class BitrateAllocatorObserver {
public:
// Returns the amount of protection used by the BitrateAllocatorObserver
// implementation, as bitrate in bps.
// 返回`BitrateAllocatorObserver`实现所使用的保护量，以比特率（bps）表示
virtual uint32_t OnBitrateUpdated(BitrateAllocationUpdate update) = 0;
// Returns the bitrate consumed (vs allocated) by BitrateAllocatorObserver
// 返回`BitrateAllocatorObserver`消耗的比特率（与分配的比特率相对）
virtual std::optional<DataRate> GetUsedRate() const = 0;
protected:
virtual ~BitrateAllocatorObserver() {}
};
2.2 编码接收器（VideoStreamEncoderInterface::EncoderSink）
编码接收器EncoderSink是VideoStreamEncoderInterface当中的一个嵌套类，主要声明接收编码后的视频配置变化，比特率变化等功能
// Interface for receiving encoded video frames and notifications about
class VideoStreamEncoderInterface {
public:
// Interface for receiving encoded video frames and notifications about
// configuration changes.
class EncoderSink : public EncodedImageCallback {
public:
virtual void OnEncoderConfigurationChanged(
std::vector<VideoStream> streams,
bool is_svc,
VideoEncoderConfig::ContentType content_type,
int min_transmit_bitrate_bps) = 0;
virtual void OnBitrateAllocationUpdated(
const VideoBitrateAllocation& allocation) = 0;
virtual void OnVideoLayersAllocationUpdated(
VideoLayersAllocation allocation) = 0;
};
// ...
}
3.RTP视频发送器（RtpVideoSender)
RTP视频发送器的声明位于call/rtp_video_sender.h中，如下所示。这个主要的功能是将发送数据传递到底层RTP模块，主要的内容包括：
（1）Rtp发送器状态（帧ID，目标比特率，RTP配置信息，编码器类型，RTP数据包控制器等）
（2）Rtp发送器函数（启用发送，检查激活状态，获取Rtp状态，获取Rtcp包，发送帧，状态更新回调函数）
RtpVideoSender是RTP层最上层的调度模块，其最核心的功能是将VideoSendStream传递过来的视频帧合理的传输到RTP底层传输模块，使用的函数为EncodedImageCallback::Result OnEncodedImage()
// RtpVideoSender routes outgoing data to the correct sending RTP module, based
// on the simulcast layer in RTPVideoHeader.
// RtpVideoSender根据RTPVideoHeader中的Simulcast层，将出站数据路由到正确的发送RTP模块
class RtpVideoSender : public RtpVideoSenderInterface,
public VCMProtectionCallback,
public StreamFeedbackObserver {
public:
// Rtp modules are assumed to be sorted in simulcast index order.
RtpVideoSender(
const Environment& env,
absl::Nonnull<TaskQueueBase*> transport_queue,
const std::map<uint32_t, RtpState>& suspended_ssrcs,
const std::map<uint32_t, RtpPayloadState>& states,
const RtpConfig& rtp_config,
int rtcp_report_interval_ms,
Transport* send_transport,
const RtpSenderObservers& observers,
RtpTransportControllerSendInterface* transport,
RateLimiter* retransmission_limiter, // move inside RtpTransport
std::unique_ptr<FecController> fec_controller,
FrameEncryptorInterface* frame_encryptor,
const CryptoOptions& crypto_options, // move inside RtpTransport
rtc::scoped_refptr<FrameTransformerInterface> frame_transformer);
~RtpVideoSender() override;
// 禁用拷贝构造和赋值构造
RtpVideoSender(const RtpVideoSender&) = delete;
RtpVideoSender& operator=(const RtpVideoSender&) = delete;
// 启用或禁用视频发送功能
void SetSending(bool enabled) RTC_LOCKS_EXCLUDED(mutex_) override;
// 检查视频发送流是否处于活跃状态
bool IsActive() RTC_LOCKS_EXCLUDED(mutex_) override;
// 在网络可用性发生变化时被调用
void OnNetworkAvailability(bool network_available)
RTC_LOCKS_EXCLUDED(mutex_) override;
// 获取Rtp的状态
std::map<uint32_t, RtpState> GetRtpStates() const
RTC_LOCKS_EXCLUDED(mutex_) override;
// 获取Rtp负载状态
std::map<uint32_t, RtpPayloadState> GetRtpPayloadStates() const
RTC_LOCKS_EXCLUDED(mutex_) override;
// 处理接收到的RTCP包
void DeliverRtcp(const uint8_t* packet, size_t length)
RTC_LOCKS_EXCLUDED(mutex_) override;
// Implements webrtc::VCMProtectionCallback.
// 用于请求保护措施，如FEC（前向纠错编码）和NACK（负向自动重传请求），以保护视频流免受网络丢包的影响
int ProtectionRequest(const FecProtectionParams* delta_params,
const FecProtectionParams* key_params,
uint32_t* sent_video_rate_bps,
uint32_t* sent_nack_rate_bps,
uint32_t* sent_fec_rate_bps)
RTC_LOCKS_EXCLUDED(mutex_) override;
// 'retransmission_mode' is either a value of enum RetransmissionMode, or
// computed with bitwise operators on values of enum RetransmissionMode.
// 设置重传流模式
void SetRetransmissionMode(int retransmission_mode)
RTC_LOCKS_EXCLUDED(mutex_) override;
// Implements FecControllerOverride.
// 设置是否允许使用FEC保护
void SetFecAllowed(bool fec_allowed) RTC_LOCKS_EXCLUDED(mutex_) override;
// Implements EncodedImageCallback.
// Returns 0 if the packet was routed / sent, -1 otherwise.
// 重要的回调函数，用于将上一层次获取的帧，送入到底层RTP链路传输
EncodedImageCallback::Result OnEncodedImage(
const EncodedImage& encoded_image,
const CodecSpecificInfo* codec_specific_info)
RTC_LOCKS_EXCLUDED(mutex_) override;
// 检测到码率分配更新
void OnBitrateAllocationUpdated(const VideoBitrateAllocation& bitrate)
RTC_LOCKS_EXCLUDED(mutex_) override;
// 检测到视频Layer更新
void OnVideoLayersAllocationUpdated(
const VideoLayersAllocation& layers) override;
// 检测到传输开销发生变化
void OnTransportOverheadChanged(size_t transport_overhead_bytes_per_packet)
RTC_LOCKS_EXCLUDED(mutex_) override;
// 检测到码率发生变化
void OnBitrateUpdated(BitrateAllocationUpdate update, int framerate)
RTC_LOCKS_EXCLUDED(mutex_) override;
// 有效载荷的码率
uint32_t GetPayloadBitrateBps() const RTC_LOCKS_EXCLUDED(mutex_) override;
// 获取保护码率，单位为bps，通常指的是用于FEC和NACK等保护措施的比特率
uint32_t GetProtectionBitrateBps() const RTC_LOCKS_EXCLUDED(mutex_) override;
// 设置编码数据，包括视频的宽度、高度和时间层数
void SetEncodingData(size_t width, size_t height, size_t num_temporal_layers)
RTC_LOCKS_EXCLUDED(mutex_) override;
// 获取已发送RTP数据包的信息
std::vector<RtpSequenceNumberMap::Info> GetSentRtpPacketInfos(
uint32_t ssrc,
rtc::ArrayView<const uint16_t> sequence_numbers) const
RTC_LOCKS_EXCLUDED(mutex_) override;
// From StreamFeedbackObserver.
// 处理数据包反馈向量
void OnPacketFeedbackVector(
std::vector<StreamPacketInfo> packet_feedback_vector)
RTC_LOCKS_EXCLUDED(mutex_) override;
private:
// 检查视频发送模块是否处于活跃状态
bool IsActiveLocked() RTC_EXCLUSIVE_LOCKS_REQUIRED(mutex_);
// 设置视频发送模块的活跃状态
void SetActiveModulesLocked(bool sending)
RTC_EXCLUSIVE_LOCKS_REQUIRED(mutex_);
// 更新视频发送模块的发送状态
void UpdateModuleSendingState() RTC_EXCLUSIVE_LOCKS_REQUIRED(mutex_);
// 配置视频流的保护策略，如FEC和NACK等
void ConfigureProtection();
// 配置SSRC
void ConfigureSsrcs(const std::map<uint32_t, RtpState>& suspended_ssrcs);
// 检查NACK（负向自动重传请求）是否被启用
bool NackEnabled() const;
// 获取编码后的数据开销，即除了实际媒体数据之外的额外数据量
DataRate GetPostEncodeOverhead() const;
// 计算数据包的开销率。它接受数据速率、数据包大小、每包开销和帧率作为参数，并返回开销率
DataRate CalculateOverheadRate(DataRate data_rate,
DataSize packet_size,
DataSize overhead_per_packet,
Frequency framerate) const;
// 在transport_checker_任务队列上设置RTP模块的活跃状态
void SetModuleIsActive(bool sending, RtpRtcpInterface& rtp_module)
RTC_RUN_ON(transport_checker_);
// 环境变量，包含了WebRTC环境的全局设置和功能
const Environment env_;
// 是否使用帧率来计算编码后的数据包开销
const bool use_frame_rate_for_overhead_;
// 是否有数据包反馈信息可用
const bool has_packet_feedback_;
// Semantically equivalent to checking for `transport_->GetWorkerQueue()`
// but some tests need to be updated to call from the correct context.
// 确保相关操作在同一个任务队列（transport_queue_）上执行
RTC_NO_UNIQUE_ADDRESS SequenceChecker transport_checker_;
TaskQueueBase& transport_queue_;
// TODO(bugs.webrtc.org/13517): Remove mutex_ once RtpVideoSender runs on the
// transport task queue.
mutable Mutex mutex_;
// 指示RtpVideoSender是否处于活跃状态
bool active_ RTC_GUARDED_BY(mutex_);
// 用来管理前向纠错（FEC）功能
const std::unique_ptr<FecController> fec_controller_;
// 是否允许使用FEC保护
bool fec_allowed_ RTC_GUARDED_BY(mutex_);
// Rtp modules are assumed to be sorted in simulcast index order.
// 多个RTP流
const std::vector<webrtc_internal_rtp_video_sender::RtpStreamSender>
rtp_streams_;
// RTP配置信息
const RtpConfig rtp_config_;
// 编码器类型
const std::optional<VideoCodecType> codec_type_;
// 控制RTP数据包的传输
RtpTransportControllerSendInterface* const transport_;
// When using the generic descriptor we want all simulcast streams to share
// one frame id space (so that the SFU can switch stream without having to
// rewrite the frame id), therefore `shared_frame_id` has to live in a place
// where we are aware of all the different streams.
/*
当使用通用描述符时，我们希望所有的Simulcast流共享一个帧ID空间
（这样SFU在切换流时就不需要重写帧ID），因此`shared_frame_id`必须存在于一个能够意识到所有不同流的地方
PS:
存储共享的帧ID。这个变量用于在多个Simulcast流之间共享帧ID空间，以便SFU
（Selective Forwarding Unit）可以在切换流时不需要重写帧ID
*/
int64_t shared_frame_id_ = 0;
// 是否为每个流独立使用帧ID。如果为true，则每个流有自己的帧ID空间
const bool independent_frame_ids_;
// RTP有效载荷参数
std::vector<RtpPayloadParams> params_ RTC_GUARDED_BY(mutex_);
// 每个数据包的传输开销字节数
size_t transport_overhead_bytes_per_packet_ RTC_GUARDED_BY(mutex_);
// 用于FEC和NACK等保护措施的比特率
uint32_t protection_bitrate_bps_;
// 编码器的目标比特率
uint32_t encoder_target_rate_bps_;
// 丢包掩码信息
std::vector<bool> loss_mask_vector_ RTC_GUARDED_BY(mutex_);
// 帧计数信息
std::vector<FrameCounts> frame_counts_ RTC_GUARDED_BY(mutex_);
// 观察帧计数的变化
FrameCountObserver* const frame_count_observer_;
// Effectively const map from SSRC to RtpRtcp, for all media SSRCs.
// This map is set at construction time and never changed, but it's
// non-trivial to make it properly const.
std::map<uint32_t, RtpRtcpInterface*> ssrc_to_rtp_module_;
ScopedTaskSafety safety_;
};
EncodedImageCallback::Result OnEncodedImage()的定义位于call/rtp_video_sender.c中，其中最核心的函数为sender_video->SendEncodedImage()，这个函数会将视频流送入到底层RTP模块去发送。rtp_streams_[].sender_video的类为RTPSenderVideo
EncodedImageCallback::Result RtpVideoSender::OnEncodedImage(
const EncodedImage& encoded_image,
const CodecSpecificInfo* codec_specific_info) {
// ...
bool send_result =
rtp_streams_[simulcast_index].sender_video->SendEncodedImage(
rtp_config_.payload_type, codec_type_, rtp_timestamp, encoded_image,
params_[simulcast_index].GetRtpVideoHeader(
encoded_image, codec_specific_info, frame_id),
expected_retransmission_time);
// ...
return Result(Result::OK, rtp_timestamp);
}
4.RTP发送视频（RTPSenderVideo）
RTPSenderVideo类的主要作用是将编码完成后的视频帧进行RTP封装，并将这些RTP包传递给PacedSender进行发送，其声明位于modules/rtp_rtcp/source/rtp_sender_video.h中，这个类将视频帧传输给网络层的步骤为：
-
SendEncodedImage()
检查当前帧是否进行帧变换，如果进行，则会将当前帧先进行变换，随后异步发送；如果不进行，则调用SendVideo()进行帧的发送 -
SendVideo()
对当前帧进行RTP格式的处理，后续调用LogAndSendToNetwork()将RTP格式的帧送入到网络模块中 -
LogAndSendToNetwork()
将RTP格式的帧，送入到网络模块中class RTPSenderVideo : public RTPVideoFrameSenderInterface {
public:
static constexpr TimeDelta kTLRateWindowSize = TimeDelta::Millis(2’500);struct Config {
Config() = default;
Config(const Config&) = delete;
Config(Config&&) = default;// All members of this struct, with the exception of `field_trials`, are // expected to outlive the RTPSenderVideo object they are passed to. // 时钟 Clock* clock = nullptr; // RTP发送器 RTPSender* rtp_sender = nullptr; // Some FEC data is duplicated here in preparation of moving FEC to // the egress stage. // FEC类型，指示视频流应该使用的FEC方案 std::optional<VideoFecGenerator::FecType> fec_type; // 表示每个RTP包的最大FEC开销字节数 size_t fec_overhead_bytes = 0; // Per packet max FEC overhead. // 用于加密视频帧 FrameEncryptorInterface* frame_encryptor = nullptr; // 是否需要对视频帧进行加密 bool require_frame_encryption = false; // 是否启用所有Simulcast层的重传 bool enable_retransmit_all_layers = false; // 指定RED（冗余编码）的载荷类型，RED为Redundant std::optional<int> red_payload_type; // 访问field trials（实验性特性） const FieldTrialsView* field_trials = nullptr; // 对视频帧进行变换 rtc::scoped_refptr<FrameTransformerInterface> frame_transformer; TaskQueueFactory* task_queue_factory = nullptr;};
explicit RTPSenderVideo(const Config& config);
virtual ~RTPSenderVideo();
//
capture_timeandclock::CurrentTimeshould be using the same epoch.
//expected_retransmission_time.IsFinite()-> retransmission allowed.
//encoder_output_sizeis the size of the video frame as it came out of the
// video encoder, excluding any additional overhead.
// Calls to this method are assumed to be externally serialized.
// 发送视频帧
bool SendVideo(int payload_type,
std::optional codec_type,
uint32_t rtp_timestamp,
Timestamp capture_time,
rtc::ArrayView payload,
size_t encoder_output_size,
RTPVideoHeader video_header,
TimeDelta expected_retransmission_time,
std::vector<uint32_t> csrcs) override;
// 发送编码后图像，其中会先处理帧，随后调用SendVideo发送
bool SendEncodedImage(int payload_type,
std::optional codec_type,
uint32_t rtp_timestamp,
const EncodedImage& encoded_image,
RTPVideoHeader video_header,
TimeDelta expected_retransmission_time);// Configures video structures produced by encoder to send using the
// dependency descriptor rtp header extension. Next call to SendVideo should
// have video_header.frame_type == kVideoFrameKey.
// All calls to SendVideo after this call must use video_header compatible
// with the video_structure.
// 设置视频帧的结构
// 1.FrameDependencyStructure是一个结构体，描述了视频帧之间的依赖关系，
// 包括帧类型（如关键帧、Delta帧等）、帧的层级结构（如SVC或Simulcast层）以及帧的参考关系
void SetVideoStructure(const FrameDependencyStructure* video_structure);
// Should only be used by a RTPSenderVideoFrameTransformerDelegate and exists
// to ensure correct syncronization.
// 在帧变换后设置视频结构，确保视频帧的依赖关系在变换后仍然保持正确
void SetVideoStructureAfterTransformation(
const FrameDependencyStructure* video_structure) override;// Sets current active VideoLayersAllocation. The allocation will be sent
// using the rtp video layers allocation extension. The allocation will be
// sent in full on every key frame. The allocation will be sent once on a
// none discardable delta frame per call to this method and will not contain
// resolution and frame rate.
// 设置当前活跃的视频层分配（VideoLayersAllocation）。这个分配将通过RTP视频层分配扩展
// （rtp video layers allocation extension）发送。在每个关键帧上，分配将完整发送。
// 在每个非可丢弃的Delta帧上，分配将被发送一次，并且此调用不会包含分辨率和帧率信息
void SetVideoLayersAllocation(VideoLayersAllocation allocation);
// Should only be used by a RTPSenderVideoFrameTransformerDelegate and exists
// to ensure correct syncronization.
// 在帧变换后设置视频层分配
// 在帧变换和视频层分配更新过程中，需要保持线程安全和同步，以避免数据竞争和不一致
void SetVideoLayersAllocationAfterTransformation(
VideoLayersAllocation allocation) override;// Returns the current post encode overhead rate, in bps. Note that this is
// the payload overhead, eg the VP8 payload headers and any other added
// metadata added by transforms. It does not include the RTP headers or
// extensions.
// TODO(sprang): Consider moving this to RtpSenderEgress so it’s in the same
// place as the other rate stats.
DataRate PostEncodeOverhead() const;// ‘retransmission_mode’ is either a value of enum RetransmissionMode, or
// computed with bitwise operators on values of enum RetransmissionMode.
void SetRetransmissionSetting(int32_t retransmission_settings);protected:
// 获取temporal ID，在SVC场景下常用
static uint8_t GetTemporalId(const RTPVideoHeader& header);
// 允许重传
bool AllowRetransmission(uint8_t temporal_id,
int32_t retransmission_settings,
TimeDelta expected_retransmission_time);private:
struct TemporalLayerStats {
FrequencyTracker frame_rate{kTLRateWindowSize};
Timestamp last_frame_time = Timestamp::Zero();
};enum class SendVideoLayersAllocation {
kSendWithResolution,
kSendWithoutResolution,
kDontSend
};
// 设置视频结构（内部函数）
void SetVideoStructureInternal(
const FrameDependencyStructure* video_structure);
// 设置视频Layer分配（内部函数）
void SetVideoLayersAllocationInternal(VideoLayersAllocation allocation);
// 添加Rtp头部信息的扩展
void AddRtpHeaderExtensions(const RTPVideoHeader& video_header,
bool first_packet,
bool last_packet,
RtpPacketToSend* packet) const
RTC_EXCLUSIVE_LOCKS_REQUIRED(send_checker_);size_t FecPacketOverhead() const RTC_EXCLUSIVE_LOCKS_REQUIRED(send_checker_);
// 记录并且将视频帧送入到网络模块中
void LogAndSendToNetwork(
std::vector<std::unique_ptr> packets,
size_t encoder_output_size);
// 是否允许冗余
bool red_enabled() const { return red_payload_type_.has_value(); }
// 更新条件重传设置
bool UpdateConditionalRetransmit(uint8_t temporal_id,
TimeDelta expected_retransmission_time)
RTC_EXCLUSIVE_LOCKS_REQUIRED(stats_mutex_);
// 可能更新当前的播放延迟
void MaybeUpdateCurrentPlayoutDelay(const RTPVideoHeader& header)
RTC_EXCLUSIVE_LOCKS_REQUIRED(send_checker_);
// 用于发送RTP包
RTPSender* const rtp_sender_;
Clock* const clock_;// These members should only be accessed from within SendVideo() to avoid
// potential race conditions.
// 检测潜在的竞争条件，确保SendVideo()函数中访问的成员变量是线程安全的
rtc::RaceChecker send_checker_;
// 重传配置
int32_t retransmission_settings_ RTC_GUARDED_BY(send_checker_);
// 上一次视频翻转
VideoRotation last_rotation_ RTC_GUARDED_BY(send_checker_);
// 上一次color space
std::optional last_color_space_ RTC_GUARDED_BY(send_checker_);
// 是否在下一帧传输颜色空间信息
bool transmit_color_space_next_frame_ RTC_GUARDED_BY(send_checker_);
// 视频帧的依赖结构信息
std::unique_ptr video_structure_
RTC_GUARDED_BY(send_checker_);
// 视频层分配信息
std::optional allocation_
RTC_GUARDED_BY(send_checker_);
// Flag indicating if we should sendallocation_.
// 是否应该发送allocation_
SendVideoLayersAllocation send_allocation_ RTC_GUARDED_BY(send_checker_);
// 上次完整发送的视频层分配信息
std::optional last_full_sent_allocation_
RTC_GUARDED_BY(send_checker_);// Current target playout delay.
// 当前目标播放延迟
std::optional current_playout_delay_
RTC_GUARDED_BY(send_checker_);
// Flag indicating if we need to sendcurrent_playout_delay_in order
// to guarantee it gets delivered.
// 是否需要发送current_playout_delay_以确保它被送达
bool playout_delay_pending_;
// Set by the field trial WebRTC-ForceSendPlayoutDelay to override the playout
// delay of outgoing video frames.
// 覆盖outgoing video frames的播放延迟
const std::optional forced_playout_delay_;// Should never be held when calling out of this class.
Mutex mutex_;
// red负载类型
const std::optional red_payload_type_;
// fec类型
std::optionalVideoFecGenerator::FecType fec_type_;
// 表示每个RTP包的最大前向纠错（FEC）开销字节数
const size_t fec_overhead_bytes_; // Per packet max FEC overhead.mutable Mutex stats_mutex_;
// 跟踪编码后的数据包开销比特率
BitrateTracker post_encode_overhead_bitrate_ RTC_GUARDED_BY(stats_mutex_);
// 存储每个时域层的帧统计信息
std::map<int, TemporalLayerStats> frame_stats_by_temporal_layer_
RTC_GUARDED_BY(stats_mutex_);
// 标记是否已经发送了第一帧视频
OneTimeEvent first_frame_sent_;// E2EE Custom Video Frame Encryptor (optional)
FrameEncryptorInterface* const frame_encryptor_ = nullptr;
// If set to true will require all outgoing frames to pass through an
// initialized frame_encryptor_ before being sent out of the network.
// Otherwise these payloads will be dropped.
const bool require_frame_encryption_;
// Set to true if the generic descriptor should be authenticated.
const bool generic_descriptor_auth_experiment_;
// 发送绝对捕获时间信息
AbsoluteCaptureTimeSender absolute_capture_time_sender_
RTC_GUARDED_BY(send_checker_);
// Tracks updates to the active decode targets and decides when active decode
// targets bitmask should be attached to the dependency descriptor.
ActiveDecodeTargetsHelper active_decode_targets_tracker_;const rtc::scoped_refptr
frame_transformer_delegate_;// Whether to do two-pass packetization for AV1 which leads to a set of
// packets with more even size distribution.
const bool enable_av1_even_split_;
};
4.1 RTPSenderVideo::SendEncodedImage()
函数主要的作用是检查当前帧是否需要变换，如果不需要变换，则调用SendVideo()。定义位于modules/rtp_rtcp/source/rtp_sender_video.cc中
bool RTPSenderVideo::SendEncodedImage(int payload_type,
std::optional<VideoCodecType> codec_type,
uint32_t rtp_timestamp,
const EncodedImage& encoded_image,
RTPVideoHeader video_header,
TimeDelta expected_retransmission_time) {
if (frame_transformer_delegate_) {
// The frame will be sent async once transformed.
return frame_transformer_delegate_->TransformFrame(
payload_type, codec_type, rtp_timestamp, encoded_image, video_header,
expected_retransmission_time);
}
return SendVideo(payload_type, codec_type, rtp_timestamp,
encoded_image.CaptureTime(), encoded_image,
encoded_image.size(), video_header,
expected_retransmission_time, /*csrcs=*/{});
}
4.2 RTPSenderVideo::SendVideo
函数的主要作用是对获取到的视频进行RTP封装，随后调用LogAndSendToNetwork()发送到网络模块
bool RTPSenderVideo::SendVideo(int payload_type,
std::optional<VideoCodecType> codec_type,
uint32_t rtp_timestamp,
Timestamp capture_time,
rtc::ArrayView<const uint8_t> payload,
size_t encoder_output_size,
RTPVideoHeader video_header,
TimeDelta expected_retransmission_time,
std::vector<uint32_t> csrcs) {
// ...
// 一些RTP的封装操作
LogAndSendToNetwork(std::move(rtp_packets), encoder_output_size);
// ...
}
4.3 RTPSenderVideo::LogAndSendToNetwork
函数的主要作用是将已经封装好的RTP数据包送入到发送队列中
void RTPSenderVideo::LogAndSendToNetwork(
std::vector<std::unique_ptr<RtpPacketToSend>> packets,
size_t encoder_output_size) {
{
MutexLock lock(&stats_mutex_);
size_t packetized_payload_size = 0;
for (const auto& packet : packets) {
// 检查是否是视频帧
if (*packet->packet_type() == RtpPacketMediaType::kVideo) {
packetized_payload_size += packet->payload_size();
}
}
// AV1 and H264 packetizers may produce less packetized bytes than
// unpacketized.
if (packetized_payload_size >= encoder_output_size) {
post_encode_overhead_bitrate_.Update(
packetized_payload_size - encoder_output_size, clock_->CurrentTime());
}
}
// 送入到发送队列当中
rtp_sender_->EnqueuePackets(std::move(packets));
}
5.RTP发送器（RTPSender）
RTPSender的声明位于modules/rtp_rtcp/source/rtp_sender.h中，这里开始就进入网络模块，从名称中也可以看出，前面的类当中都带有"Video"字样，这里只有RTP层级。这个类当中实现的主要功能为RTP数据的处理，包括检查RTP发送状态（RTX和Fec），检测到数据包的获取，检查RTP当中信息等，其中最核心的是将RTP数据包送入发送队列当中，使用的函数为EnqueuePackets()
class RTPSender {
public:
RTPSender(const Environment& env,
const RtpRtcpInterface::Configuration& config,
RtpPacketHistory* packet_history,
RtpPacketSender* packet_sender);
RTPSender(const RTPSender&) = delete;
RTPSender& operator=(const RTPSender&) = delete;
~RTPSender();
// 设置媒体流的发送状态
void SetSendingMediaStatus(bool enabled) RTC_LOCKS_EXCLUDED(send_mutex_);
// 查询当前是否正在发送媒体流
bool SendingMedia() const RTC_LOCKS_EXCLUDED(send_mutex_);
// 检查音频是否已经被配置
bool IsAudioConfigured() const RTC_LOCKS_EXCLUDED(send_mutex_);
// 获取RTP时间戳偏移量
uint32_t TimestampOffset() const RTC_LOCKS_EXCLUDED(send_mutex_);
// 设置RTP时间戳偏移量
void SetTimestampOffset(uint32_t timestamp) RTC_LOCKS_EXCLUDED(send_mutex_);
// 设置MID（Media ID）值，MID用于在SDP（Session Description Protocol）中标识媒体流
void SetMid(absl::string_view mid) RTC_LOCKS_EXCLUDED(send_mutex_);
// 获取当前的RTP序列号
uint16_t SequenceNumber() const RTC_LOCKS_EXCLUDED(send_mutex_);
// 设置RTP序列号
void SetSequenceNumber(uint16_t seq) RTC_LOCKS_EXCLUDED(send_mutex_);
// 设置RTP包的最大值
void SetMaxRtpPacketSize(size_t max_packet_size)
RTC_LOCKS_EXCLUDED(send_mutex_);
void SetExtmapAllowMixed(bool extmap_allow_mixed)
RTC_LOCKS_EXCLUDED(send_mutex_);
// RTP header extension
// 注册一个RTP头部扩展
bool RegisterRtpHeaderExtension(absl::string_view uri, int id)
RTC_LOCKS_EXCLUDED(send_mutex_);
// 检查是否已经注册了指定类型的RTP头部扩展
bool IsRtpHeaderExtensionRegistered(RTPExtensionType type) const
RTC_LOCKS_EXCLUDED(send_mutex_);
// 注销一个RTP头部扩展
void DeregisterRtpHeaderExtension(absl::string_view uri)
RTC_LOCKS_EXCLUDED(send_mutex_);
// 查询是否支持padding
bool SupportsPadding() const RTC_LOCKS_EXCLUDED(send_mutex_);
// 查询是否支持RTX（重传）载荷填充
bool SupportsRtxPayloadPadding() const RTC_LOCKS_EXCLUDED(send_mutex_);
// 生成padding
std::vector<std::unique_ptr<RtpPacketToSend>> GeneratePadding(
size_t target_size_bytes,
bool media_has_been_sent,
bool can_send_padding_on_media_ssrc) RTC_LOCKS_EXCLUDED(send_mutex_);
// NACK.
// 检测收到Nack数据包
void OnReceivedNack(const std::vector<uint16_t>& nack_sequence_numbers,
int64_t avg_rtt) RTC_LOCKS_EXCLUDED(send_mutex_);
// 重传指定ID的数据包
int32_t ReSendPacket(uint16_t packet_id) RTC_LOCKS_EXCLUDED(send_mutex_);
// ACK.
void OnReceivedAckOnSsrc(int64_t extended_highest_sequence_number)
RTC_LOCKS_EXCLUDED(send_mutex_);
void OnReceivedAckOnRtxSsrc(int64_t extended_highest_sequence_number)
RTC_LOCKS_EXCLUDED(send_mutex_);
// RTX.
// 设置Rtx状态
void SetRtxStatus(int mode) RTC_LOCKS_EXCLUDED(send_mutex_);
// 获取Rtx状态
int RtxStatus() const RTC_LOCKS_EXCLUDED(send_mutex_);
// 获取RTX流的SSRC
std::optional<uint32_t> RtxSsrc() const RTC_LOCKS_EXCLUDED(send_mutex_) {
return rtx_ssrc_;
}
// Returns expected size difference between an RTX packet and media packet
// that RTX packet is created from. Returns 0 if RTX is disabled.
// 计算RTX包与其原始媒体包之间的预期大小差异
size_t RtxPacketOverhead() const;
// 设置RTX（Retransmission RTP Stream）的载荷类型
void SetRtxPayloadType(int payload_type, int associated_payload_type)
RTC_LOCKS_EXCLUDED(send_mutex_);
// Size info for header extensions used by FEC packets.
// 获取前向纠错（FEC）包使用的RTP头部扩展的大小信息
static rtc::ArrayView<const RtpExtensionSize> FecExtensionSizes()
RTC_LOCKS_EXCLUDED(send_mutex_);
// Size info for header extensions used by video packets.
// 获取视频包使用的RTP头部扩展的大小信息
static rtc::ArrayView<const RtpExtensionSize> VideoExtensionSizes()
RTC_LOCKS_EXCLUDED(send_mutex_);
// Size info for header extensions used by audio packets.
// 获取音频包使用的RTP头部扩展的大小信息
static rtc::ArrayView<const RtpExtensionSize> AudioExtensionSizes()
RTC_LOCKS_EXCLUDED(send_mutex_);
// Create empty packet, fills ssrc, csrcs and reserve place for header
// extensions RtpSender updates before sending.
// 创建一个空的RTP包，并填充SSRC（同步源标识符）和CSRC（贡献源标识符）
std::unique_ptr<RtpPacketToSend> AllocatePacket(
rtc::ArrayView<const uint32_t> csrcs = {})
RTC_LOCKS_EXCLUDED(send_mutex_);
// Maximum header overhead per fec/padding packet.
// 获取FEC（前向纠错）或填充包的最大RTP头部长度
size_t FecOrPaddingPacketMaxRtpHeaderLength() const
RTC_LOCKS_EXCLUDED(send_mutex_);
// Expected header overhead per media packet.
// 计算每个媒体包的预期头部开销
size_t ExpectedPerPacketOverhead() const RTC_LOCKS_EXCLUDED(send_mutex_);
// Including RTP headers.
// 获取最大的RTP包大小，包括RTP头部
size_t MaxRtpPacketSize() const RTC_LOCKS_EXCLUDED(send_mutex_);
// 获取SSRC
uint32_t SSRC() const RTC_LOCKS_EXCLUDED(send_mutex_) { return ssrc_; }
std::optional<uint32_t> FlexfecSsrc() const RTC_LOCKS_EXCLUDED(send_mutex_) {
return flexfec_ssrc_;
}
// Pass a set of packets to RtpPacketSender instance, for paced or immediate
// sending to the network.
// 将一系列packets送入到RtpPacketSender的实例中，这些包会被安排或者立即发送到网络当中
void EnqueuePackets(std::vector<std::unique_ptr<RtpPacketToSend>> packets)
RTC_LOCKS_EXCLUDED(send_mutex_);
// 设置RTP状态
void SetRtpState(const RtpState& rtp_state) RTC_LOCKS_EXCLUDED(send_mutex_);
// 获取RTP状态
RtpState GetRtpState() const RTC_LOCKS_EXCLUDED(send_mutex_);
// 设置RTX的RTP状态
void SetRtxRtpState(const RtpState& rtp_state)
RTC_LOCKS_EXCLUDED(send_mutex_);
// 获取RTX的RTP状态
RtpState GetRtxRtpState() const RTC_LOCKS_EXCLUDED(send_mutex_);
private:
// 创建Rtx数据包
std::unique_ptr<RtpPacketToSend> BuildRtxPacket(
const RtpPacketToSend& packet);
// 是否是Fec数据包
bool IsFecPacket(const RtpPacketToSend& packet) const;
// 更新RTP头部的大小信息
void UpdateHeaderSizes() RTC_EXCLUSIVE_LOCKS_REQUIRED(send_mutex_);
// 更新最后一个发送的RTP包的状态
void UpdateLastPacketState(const RtpPacketToSend& packet)
RTC_EXCLUSIVE_LOCKS_REQUIRED(send_mutex_);
Clock* const clock_;
Random random_ RTC_GUARDED_BY(send_mutex_);
const bool audio_configured_;
const uint32_t ssrc_;
const std::optional<uint32_t> rtx_ssrc_;
const std::optional<uint32_t> flexfec_ssrc_;
RtpPacketHistory* const packet_history_;
RtpPacketSender* const paced_sender_;
mutable Mutex send_mutex_;
// 是否正在发送media
bool sending_media_ RTC_GUARDED_BY(send_mutex_);
// 最大packet尺寸
size_t max_packet_size_;
// 存储RTP头部扩展信息
RtpHeaderExtensionMap rtp_header_extension_map_ RTC_GUARDED_BY(send_mutex_);
// 存储媒体RTP包的最大头部大小
size_t max_media_packet_header_ RTC_GUARDED_BY(send_mutex_);
// 存储填充和FEC（前向纠错）RTP包的最大头部大小
size_t max_padding_fec_packet_header_ RTC_GUARDED_BY(send_mutex_);
// RTP variables
// 时间戳偏移量
uint32_t timestamp_offset_ RTC_GUARDED_BY(send_mutex_);
// RID value to send in the RID or RepairedRID header extension.
const std::string rid_;
// MID value to send in the MID header extension.
std::string mid_ RTC_GUARDED_BY(send_mutex_);
// Should we send MID/RID even when ACKed? (see below).
const bool always_send_mid_and_rid_;
// Track if any ACK has been received on the SSRC and RTX SSRC to indicate
// when to stop sending the MID and RID header extensions.
bool ssrc_has_acked_ RTC_GUARDED_BY(send_mutex_);
bool rtx_ssrc_has_acked_ RTC_GUARDED_BY(send_mutex_);
// Maximum number of csrcs this sender is used with.
size_t max_num_csrcs_ RTC_GUARDED_BY(send_mutex_) = 0;
int rtx_ RTC_GUARDED_BY(send_mutex_);
// Mapping rtx_payload_type_map_[associated] = rtx.
std::map<int8_t, int8_t> rtx_payload_type_map_ RTC_GUARDED_BY(send_mutex_);
bool supports_bwe_extension_ RTC_GUARDED_BY(send_mutex_);
RateLimiter* const retransmission_rate_limiter_;
};
EnqueuePackets()函数的实现位于modules/rtp_rtcp/source/rtp_sender.cc中，从代码中看出，调用了paced_sender_->EnqueuePackets()将发送packet
void RTPSender::EnqueuePackets(
std::vector<std::unique_ptr<RtpPacketToSend>> packets) {
RTC_DCHECK(!packets.empty());
Timestamp now = clock_->CurrentTime();
for (auto& packet : packets) {
RTC_DCHECK(packet);
RTC_CHECK(packet->packet_type().has_value())
<< "Packet type must be set before sending.";
if (packet->capture_time() <= Timestamp::Zero()) {
packet->set_capture_time(now);
}
}
paced_sender_->EnqueuePackets(std::move(packets));
}
在这里，使用的paced_sender_数据类型为RtpPacketSender，这是一个父类，其中并没有实现EnqueuePackets()这个函数，声明位于api/rtp_packet_sender.h
class RtpPacketSender {
public:
virtual ~RtpPacketSender() = default;
// Insert a set of packets into queue, for eventual transmission. Based on the
// type of packets, they will be prioritized and scheduled relative to other
// packets and the current target send rate.
virtual void EnqueuePackets(
std::vector<std::unique_ptr<RtpPacketToSend>> packets) = 0;
// Clear any pending packets with the given SSRC from the queue.
// TODO(crbug.com/1395081): Make pure virtual when downstream code has been
// updated.
virtual void RemovePacketsForSsrc(uint32_t /* ssrc */) {}
};
EnqueuePackets()具体的实现由TaskQueuePacedSender给出，由于篇幅问题，下一部分再记录