Serve nix store over SSH and use as substituter

Help
1

I have a private github repo that builds the configuration of all my machines on my server which is configured as self-hosted github runner. Since the configurations are built on the server, ideally I would want to use the server as substituter so I can simply pull the build results on my other machines. I have set up the ssh store as described here, it looks like this:

  nix.sshServe = {
    enable = true;
    keys = [ tim desktop1 laptop1 ];
  };
  nix.settings.trusted-users = [ "root" "tim" ];

where tim is my user ssh public key, laptop1 and desktop1 are the keys in /etc/ssh/ssh_host_ed25519_key.pub on the machines respectively. This results in the following entry in /etc/ssh/sshd_conf on the server:

Match User nix-ssh
  AllowAgentForwarding no
  AllowTcpForwarding no
  PermitTTY no
  PermitTunnel no
  X11Forwarding no
  ForceCommand /nix/store/xb66g3x4iv7m95mja9zzi1ghhraxmpws-nix-2.11.1/bin/nix-store --serve
Match All

When I run nix-store -r /nix/store/azpl8z8fd1ryi67xjzhf3if0dfm14p3b-diffutils-3.8 --substituters ssh://tim@server (the ip of server is defined in /etc/hosts) I get error: cannot connect to 'tim@server'. Same for ssh://nix-ssh. On the server I see the following in the output of journalctl -fu sshd.service:

Jan 08 19:03:50 server sshd[3006035]: Connection closed by authenticating
 user tim IP port 43250 [preauth]
Jan 08 19:05:14 server sshd[3006045]: Connection closed by authenticating
 user nix-ssh IP port 39082 [preauth]

and nothing in the output of journalctl -fu nix-daemon.service. When I run the same command prefixed with sudo, everything works (nix realises the store path, downloading from the server, I see successful logins in sshd.service and nix-daemon.service). How do I make the store available for my regular user?

Another issue I face is for sudo nix shell --substituters ssh://nix-ssh@server nixpkgs#anything which results in:

warning: ignoring substitute for '/nix/store/hash-package' from 'ssh://nix-ssh@server', as it's not signed by any of the keys in 'trusted-public-keys'

Whats the key I should put there? Are the packages even signed automatically? Thanks for helping me out!

2

You need to set this option to sign your builds
https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-secret-key-files

I think your normal user isn’t allowed to specify substituters which aren’t in
https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-trusted-substituters

If it where allowed every user on a multi user system could introduce malicious binaries to the nix store with an freely chosen hash

3

there is also a issue on GitHub with the same problem Cannot connect to Nix store via SSH · Issue #76860 · NixOS/nixpkgs · GitHub

4

@StillerHarpo got it working in the end by setting nix.envVars.SSH_AUTH_SOCK = "/run/user/1000/gnupg/S.gpg-agent.ssh";. I have no idea though why it is required or not set automatically…

5

The issue here is twofold:

  • when you invoke a build as a normal user, the nix_daemon starts the build and it uses / inherits the root user’s ssh environment
  • when you specified the allowed keys, you specified the ssh-server host keys; it will work if you use the root user’s ssh pubkey instead.

The workaround giving it access to your agent is… also effective, but I’d be less comfortable with that personally.

1 Like
6

FYI I had the same issue and solved it by directly referencing my user ssh key as documented here:

https://nix.dev/manual/nix/2.29/store/types/#store-url-format

Store URLs can specify store settings using URL query strings, i.e. by appending ?name1=value1&name2=value2&... to the URL. For instance,

--store ssh://machine.example.org?ssh-key=/path/to/my/key

tells Nix to access the store on a remote machine via the SSH protocol, using /path/to/my/key as the SSH private key. The supported settings for each store type are documented below.

https://nix.dev/manual/nix/2.29/store/types/ssh-store

1 Like