OBS hardware encoder is not available

Help
1

I installed the obs using home manager:

  programs.obs-studio = {
    enable = true;
    plugins = with pkgs.obs-studio-plugins; [
      obs-backgroundremoval
    ];
  };

And tried to enable opengl and vaapi using this config:

hardware.opengl = {
    enable = true;
    extraPackages = with pkgs; [
      intel-media-driver # LIBVA_DRIVER_NAME=iHD
      intel-vaapi-driver # LIBVA_DRIVER_NAME=i965 (older but works better for Firefox/Chromium)
      vaapiVdpau
      libvdpau-va-gl
    ];
  };
  environment.sessionVariables.LIBVA_DRIVER_NAME = "iHD";

But OBS doesn’t show the hardware option in the video encoder config.

When I run obs --verbose, I see these logs related to the opengl:

info: Initializing OpenGL...
debug: Created EGLDisplay 0x121a0c0
info: Loading up OpenGL on adapter Intel Mesa Intel(R) HD Graphics 530 (SKL GT2)
info: OpenGL loaded successfully, version 4.6 (Core Profile) Mesa 23.1.9, shading language 4.60

debug: Loading module: obs-ffmpeg.so
error: os_dlopen(libnvidia-encode.so.1->libnvidia-encode.so.1): libnvidia-encode.so.1: cannot open shared object file: No such file or directory

debug: VAAPI: Initializing display in vaapi_device_h264_supported
debug: Libva: VA-API version 1.20.0
debug: Libva: Trying to open /run/opengl-driver/lib/dri/nouveau_drv_video.so
debug: Libva: Found init function __vaDriverInit_1_20
debug: Libva: va_openDriver() returns 0
debug: VAAPI: Display initialized
info: VAAPI: API version 1.20
debug: VAAPI: 'Mesa Gallium driver 23.1.9 for NV117' in use for device '/dev/dri/renderD128'
debug: VAAPI: Initializing display in vaapi_device_h264_supported
debug: Libva: VA-API version 1.20.0
debug: Libva: Trying to open /run/opengl-driver/lib/dri/iHD_drv_video.so
debug: Libva: Found init function __vaDriverInit_1_20
debug: Libva: va_openDriver() returns 0
debug: VAAPI: Display initialized
debug: VAAPI: 'Intel iHD driver for Intel(R) Gen Graphics - 23.3.5 ()' in use for device '/dev/dri/renderD129'
debug: '/dev/dri/renderD129' support encoding with VAProfileH264ConstrainedBaseline
debug: '/dev/dri/renderD129' support encoding with VAProfileH264Main
debug: '/dev/dri/renderD129' support encoding with VAProfileH264High
info: FFmpeg VAAPI H264 encoding supported
debug: VAAPI: Initializing display in vaapi_device_hevc_supported
debug: Libva: VA-API version 1.20.0
debug: Libva: Trying to open /run/opengl-driver/lib/dri/nouveau_drv_video.so
debug: Libva: Found init function __vaDriverInit_1_20
debug: Libva: va_openDriver() returns 0
debug: VAAPI: Display initialized
debug: VAAPI: 'Mesa Gallium driver 23.1.9 for NV117' in use for device '/dev/dri/renderD128'
debug: VAAPI: Initializing display in vaapi_device_hevc_supported
debug: Libva: VA-API version 1.20.0
debug: Libva: Trying to open /run/opengl-driver/lib/dri/iHD_drv_video.so
debug: Libva: Found init function __vaDriverInit_1_20
debug: Libva: va_openDriver() returns 0
debug: VAAPI: Display initialized
debug: VAAPI: 'Intel iHD driver for Intel(R) Gen Graphics - 23.3.5 ()' in use for device '/dev/dri/renderD129'
debug: '/dev/dri/renderD129' support encoding with VAProfileHEVCMain
debug: VAAPI: VAProfileHEVCMain10 is not supported by the device '/dev/dri/renderD129'
info: FFmpeg VAAPI HEVC encoding supported
1 Like
2

hey did you find a fix? (im having the same issue , neither nvenc nor quicksync show up)

3

Still not having the NVENC encoders. But others are showing.

  hardware.graphics = {
    enable = true;
    extraPackages = with pkgs; [
      intel-media-driver
      intel-vaapi-driver
      vaapiVdpau
      libvdpau-va-gl
      nvidia-vaapi-driver
    ];
  };
4

Might be relevant: Reddit - The heart of the internet

I enabled it via nixpkgs.config.cudaSupport = true; globally, and the rebuild is pretty heavy that it did not complete during my lunch break for me to verify it.

5

Yeah, you probably should not attempt enable this globally; that will result in rebuilding anything and everything with the option of cuda support. If you just need this for obs, use:

{ pkgs, ... }: {
  programs.obs-studio.package = pkgs.obs-studio.override {
    cudaSupport = true;
  };
}

The build will likely still take a while.

Don’t do that. Not only do none of those have anything to do with NVENC, they’re also largely incompatible with one another (literally only libvdpau-va-gl can be used alongside the others, and it should practically never be used, it’s essentially a hack for 15 year old GPUs).

If you have a modern intel GPU with integrated graphics, add intel-media-driver. If you have an older intel GPU that uses the i965 driver, use intel-vaapi-driver. If you have an AMD GPU, you don’t need to do anything, the mesa driver supports it out of the box.

If you have only an nvidia GPU - and no iGPU, neither intel nor AMD - use nvidia-vaapi-driver. That one is more of a last resort; it’s third party, very limited, requires careful configuration, and often just breaks. The intel/AMD drivers are simply better in every way.

For real encoding you probably want to use NVENC directly if you have an NVIDIA GPU, so the VAAPI driver will only be used for watching videos - a use case for which an iGPU is much better suited anyway.

vaapiVdpau is an alias of libva-vdpau-driver, which in turn is a hack that basically forwards VAAPI to VDPAU. In 2025, you basically don’t want this, just use your GPU’s VAAPI support.

You can also have your system provide VDPAU, but it’s an older standard that has been largely superseded by VAAPI, and mostly doesn’t work on wayland. You probabaly don’t want to use libvdpau-va-gl for this, though, test if your GPU drivers just support VDPAU out of the box (and make sure you use the iGPU for it by switching to it with the VDPAU_DRIVER variable, since again, this will almost only be used for watching videos, where your iGPU should comfortably keep up and burn less power doing so than the dGPU). If they don’t, I just wouldn’t bother with VDPAU, rather than using a hack for it.

6

It’s actually much less that I though. Only around 150 packages for my system. The heavy builds are about opencv and onnxruntime, which I do think can benefit greatly with GPU support.

1 Like
7

This kind of the config is likely the result of this article: Jellyfin - NixOS Wiki

I had the same snippet as well.

1 Like
8

Thank you for clarifying this.

Honestly i spent just 15min on this day when trying to enable it.

9

My problem was just enabling NVIDIA uvm. I saw in the Arch Wiki.

But TBF, i’m back using my 5600g AMD GPU.

I use Hyprland and my NVIDIA card is in my hybrid intel laptop.

I’ve searched this topic not only seeking for the encoder, but how to normalize laggy cursor and animations on external monitor and got no success.

Just NVIDIA, btw.

10

For wlroots compositors on an nvidia GPU you need these variables: dotfiles/nixos-modules/nvidia/de-compat.nix at 5e23c056a9af2d23681bcf7338d7df61371e754b · TLATER/dotfiles · GitHub

Look around my module in general for nvidia advice; Probably about time I break it into a new project: dotfiles/nixos-modules/nvidia at master · TLATER/dotfiles · GitHub

Personally I would recommend using prime offload, it works well.