Additional encoding settings available with 'Nvidia encoder'

Hi all,

I'm trying to understand why enabling the 'Nivida encoder' setting suddenly gives me access to settings like Rate control, tuning, lookahead, and AQ strength. Why are these not settings available in the Native encoder?

Likewise, What on earth are 'rate control', Tuning, Lookahead, and AQ strength? I'm familiar with two-pass encoding but baffled as these aren't terms I've heard of when using FFMPEG encoders or Avid Media Composer...

Likewise, What exactly is the difference between the native encoder and the Nvidia encoder? When using the Native encoder, my GPU usage on export is at about 100%. My guess is (and it is a guess) that with Native encoding, the GPU is used to process effects like NR and colour grades, and the CPU is used to encode the file, whereas, with Nividia encoding, the GPU is used to both process the effects and encode the file. Could someone clarify this for me?

Thanks in advance!

Screenshot 2024-02-29 221640.png (59.41 KiB) Viewed 9586 times

Screenshot 2024-02-29 221654.png (41.38 KiB) Viewed 9586 times

You can read in very depth about all those Nvidia encoder parameters here:

https://docs.nvidia.com/video-technolog ... index.html

Rate control
NVENC supports several rate control modes and provides control over various parameters related to the rate control algorithm via structure NV_ENC_INITIALIZE_PARAMS::encodeConfig::rcParams. The rate control algorithm is implemented in NVENC firmware.

NVENC supports the following rate control modes:

Constant bitrate (CBR): In this mode, only averageBitRate is required and used as the target output bitrate by the rate control algorithm.

Variable bitrate (VBR): The encoder tries to conform to average bitrate of averageBitRate over the long term while not exceeding maxBitRate any time during the encoding. In this mode, averageBitRate must be specified. If maxBitRate isn’t specified, NVENC will set it to an internally determined default value. It is recommended that the client specify both parameters maxBitRate and averageBitRate for better control.

Constant QP: In this mode, the entire frame is encoded using QP

Target quality: This mode is specified by setting rateControlMode to VBR and desired target quality in targetQuality. The range of this target quality is 0 to 51 (fractional values are also supported in Video Codec SDK 8.0 and above). In this mode, the encoder tries to maintain constant quality for each frame, by allowing the bitrate to vary subject to the bitrate parameter specified in maxBitRate. The resulting average bitrate can, therefore, vary significantly depending on the video content being encoded. In this mode, if maxBitRate is set, it will form an upper bound on the actual bitrate. Therefore, if maxBitRate is set too low, the bitrate may become constrained, resulting in the desired target quality possibly not being achieved.

Selecting encoder preset configuration
As mentioned above, the client can use the presetGUID for configuring the encode session directly. This will automatically set the hardware encoder with appropriate parameters for the use-case implied by the tuning info/preset combination. If required, the client has the option to fine-tune the encoder configuration parameters in the preset and override the preset defaults. This approach is often-times more convenient from programming point of view as the programmer only needs to change the configuration parameters which he/she is interested in, leaving everything else pre-configured as per the preset definition.

Encoder TUNING INFO AND Preset Configurations
The NVIDIA Encoder Interface exposes four different tuning info enums (high quality, low latency, ultra-low latency and lossless) to cater to different video encoding use-cases. Table 1 shows the recommended tuning info applicable to some popular use-cases.

High quality for
High-quality latency-tolerant transcoding
Video archiving
Encoding for OTT streaming

Lossless for
Preserving original video footage for later editing
General lossless data archiving (video or non-video)

Multi pass frame encoding
When determining the QP to use for encoding a frame, it is beneficial if NVENC knows the overall complexity of the frame to distribute the available bit budget in the most optimal manner. In some situations, multi-pass encoding may also help catch larger motion between frames. For this purpose, NVENC supports the following types of multi-pass frame encoding modes:

1-pass per frame encoding
2-passes per frame, with first pass in quarter resolution and second pass in full resolution
2-passes per frame, with both passes in full resolution.

In 1-pass rate control modes, NVENC estimates the required QP for the macroblock and immediately encodes the macroblock. In 2-pass rate control modes, NVENC estimates the complexity of the frame to be encoded and determines bit distribution across the frame in the first pass. In the second pass, NVENC encodes macroblocks in the frame using the distribution determined in the first pass. As a result, with 2-pass rate control modes, NVENC can distribute the bits more optimally within the frame and can reach closer to the target bitrate, especially for CBR encoding. Note, however, that everything else being the same, performance of 2-pass rate control mode is lower than that of 1-pass rate control mode. The client application should choose an appropriate multi-pass rate control mode after evaluating various modes, as each of the modes has its own advantages and disadvantages. NV_ENC_TWO_PASS_FULL_RESOLUION generates better statistics for the second pass, whereas NV_ENC_TWO_PASS_QUARTER_RESOLUTION results in larger motion vectors being caught and fed as hints to second pass.

Look-ahead
Look-ahead improves the video encoder’s rate control accuracy by enabling the encoder to buffer the specified number of frames, estimate their complexity and allocate the bits appropriately among these frames proportional to their complexity. This also dynamically allocates B and P frames.

Adaptive Quantization (AQ)
This feature improves visual quality by adjusting encoding QP (on top of QP evaluated by the Rate Control Algorithm) based on spatial and temporal characteristics of the sequence. The current SDK support two flavors of AQ which are explained as follows:

Spatial AQ
Spatial AQ mode adjusts the QP values based on spatial characteristics of the frame. Since the low complexity flat regions are visually more perceptible to quality differences than high complexity detailed regions, extra bits are allocated to flat regions of the frame at the cost of the regions having high spatial detail. Although spatial AQ improves the perceptible visual quality of the encoded video, the required bit redistribution results in PSNR drop in most of the cases. Therefore, during PSNR-based evaluation, this feature should be turned off.

The intensity of QP adjustment ranges from 1 (least aggressive) to 15 (most aggressive).

Temporal AQ
Temporal AQ tries to adjust encoding QP (on top of QP evaluated by the rate control algorithm) based on temporal characteristics of the sequence. Temporal AQ improves the quality of encoded frames by adjusting QP for regions which are constant or have low motion across frames but have high spatial detail, such that they become better reference for future frames. Allocating extra bits to such regions in reference frames is better than allocating them to the residuals in referred frames because it helps improve the overall encoded video quality. If majority of the region within a frame has little or no motion, but has high spatial details (e.g. high-detail non-moving background) enabling temporal AQ will benefit the most.

One of the potential disadvantages of temporal AQ is that enabling temporal AQ may result in high fluctuation of bits consumed per frame within a GOP. I/P-frames will consume more bits than average P-frame size and B-frames will consume lesser bits. Although target bitrate will be maintained at the GOP level, the frame size will fluctuate from one frame to next within a GOP more than it would without temporal AQ. If a strict CBR profile is required for every frame size within a GOP, it is not recommended to enable temporal AQ. Additionally, since some of the complexity estimation is performed in CUDA, there may be some performance impact when temporal AQ is enabled.

Weighted Prediction
Weighted prediction involves calculation of a multiplicative weighting factor and an additive offset to the motion compensated prediction. Weighted prediction provides significant quality gain for contents having illumination changes. NVENCODE API supports weighed prediction for HEVC and H.264 starting from Pascal generation GPUs.

Weighted prediction is not supported if the encode session is configured with B frames.

Weighted prediction is not supported if DirectX 12 device is used.

Weighted prediction uses CUDA pre-processing and hence requires CUDA processing power, depending upon resolution and content.

Enabling weighted prediction may also result in very minor degradation in encoder performance.


Recommended NVENC settings for various use-cases
Recording/Archiving

High quality Tuning Info
Rate control mode = VBR
Very large VBV buffer size (4 seconds)
B Frames*
Look-ahead
B frame as reference
Finite GOP length (2 seconds)
Adaptive quantization (AQ) enabled**

*: Recommended for low motion games and natural video.
**: Recommended on second generation Maxwell GPUs and above.

Thank you for posting that. I'll study it in a bit.

I'm just wondering why this detail isn't available with the Native encoder?

There is also additional and more helpful info in the FFmpeg NVENC doc:
https://github.com/Xaymar/obs-StreamFX/ ... b89429f3c6

I don't know why there aren't more settings available with the native encoder.

You can read in very depth about all those Nvidia encoder parameters here:

https://docs.nvidia.com/video-technolog ... index.html

Thanks for posting this info - never seen it/read it.
Now my head hurts -

You can read in very depth about all those Nvidia encoder parameters here:

https://docs.nvidia.com/video-technolog ... index.html

Thanks for posting this info - never seen it/read it.
Now my head hurts -

I find most of these extra settings make no or very little different to output quality when decent bitrates are involved, and they only come into play when needing to seriously compress down to lower bitrates/file sizes.

For 4K UHD I tend to output at 60Mbps using H265 for the final thing, and so those extra settings really aren't going to make a visible difference no matter how you set them, if I was needing to get down to around 15Mbps, then perhaps they become important.

So providing you aren't needing to optimise every last 'bit' due to very low bit rates, leaving them at their defaults is usually optimal and perfectly okay.

Thank you for all your responses.

DR19 has added lookahead levels 0 to 3.

What would be best for 4K 24fps HDR?

Nobody knows?

I've found Lookahead 3 best.

I've found Lookahead 3 best.

I wrongly assumed it's something to do about compression efficiency, I raised it to 3 but it increased my file size by 70% at which point it shouldn't be surprising it improves quality.

Anyone know how to set up the encoding for blu-ray compliant h.264 file renders? My BD authoring tool is complaining of no VBV, wrong profile and level and bitrate too high (even though I set the max limit to 35mb/s). So it recodes the whole file and decimates picture quality.