### Author:  [@RaitoBezarius](https://discourse.nixosstag.fcio.net/u/RaitoBezarius)
#### Post date: [April 20, 2024, 11:26am UTC](https://discourse.nixosstag.fcio.net/t/pre-rfc-implement-dependency-retrieval-primitive/43418/6 "2024-04-20T11:26:46Z")
> [@physics-enthusiast](#):
>
> Not sure if this is a bug that has been fixed since then, but the link seems to indicate that the `toFile` builtin discards string context (I’m assuming this is because it converts said string into a store path with no corresponding `.drv`).
```nix
nix-repl> :p builtins.getContext "${(builtins.toFile "abc" "def")}"
{
"/nix/store/043113974zgxifcy4j8s1sx8fa8h9d7f-abc" = { path = true; };
}
```
I do see a source context here.
```nix
nix-repl> :p builtins.getContext "${(builtins.toFile "abc" "${pkgs.systemd}")}"
error:
… while calling the 'getContext' builtin
at «string»:1:1:
1| builtins.getContext "${(builtins.toFile "abc" "${pkgs.systemd}")}"
| ^
… while evaluating the argument passed to builtins.getContext
… while calling the 'toFile' builtin
at «string»:1:25:
1| builtins.getContext "${(builtins.toFile "abc" "${pkgs.systemd}")}"
| ^
error: files created by builtins.toFile may not reference derivations, but abc references !out!mmxfq68bslpjqki0f7svg31cjv2pnr36-systemd-255.2.drv
```
Though, interestingly… !
```nix
nix-repl> :p builtins.getContext "${(builtins.toFile "abc" "${builtins.toFile "def" "ghi"}")}"
{
"/nix/store/kc46v2cfdvilicx6g97qqg49a2q1hi8j-abc" = { path = true; };
}
```
or
```nix
nix-repl> :p builtins.getContext "${(builtins.toFile "abc" "${/tmp/nvim.raito}")}"
{
"/nix/store/wjsm4z6rk51q846h2xdrshkn8pss99wf-abc" = { path = true; };
}
```
Both of them look like bugs to me.
> [@physics-enthusiast](#):
>
> If I’m understanding you correctly, this would require recompiling a custom Nix?
Yep, I don’t find this _that_ shocking, theoretically speaking, we should be able to swap implementations of primops more dynamically for such usecases, e.g. Tvix enable (you can disable IO, etc.).
> [@physics-enthusiast](#):
>
> I don’t think it’s a bug, but it **is** a blocker for what I’m trying to do.
Right, it is just that I’m trying to distinguish statusquo, intended things and what is the new thing that you want to achieve in this pre-RFC.
> [@physics-enthusiast](#):
>
> Maybe I missed something, but this seems quite achievable by adding a little bit of context data to generic Nix objects. Just add an extra data field to [these types](https://github.com/NixOS/nix/blob/74e4bc9b1dedc6a3228b2cde0175f7e1c89bc14d/src/libexpr/primops.cc#L411) and propagate the union of that field whenever a primop is [called](https://github.com/NixOS/nix/blob/74e4bc9b1dedc6a3228b2cde0175f7e1c89bc14d/src/libexpr/eval-inline.hh#L85). Even if you have to address each one independently, there’s only a handful of fundamental Nix object types so it should be quite doable.
So, to the `Value` structure, right? This is the most performance critical structure of the whole interpreter and the one you will have the most trouble touching IMHO without decreasing performance.
For me, I think it’s a bit hard to properly add the extra data in the current interpreter because of the work to do before doing anything new, hence the suggestions to work “around” it.
> [@physics-enthusiast](#):
>
> Implementing tracking for dynamic derivations is quite easy. All you need is somewhere to store (transiently in this case) the new type of reference (the derivation that produced it, which we already know). See the first point of the tentative `getInstantiationDeps` rules.
I know this is conceptually easy, I’m talking about the implementation in this case.
> [@physics-enthusiast](#):
>
> I was going to say that `nix show derivation`, `nix-store --dump-db` and analogues don’t work in `recursive-nix`, but I went back and tested it and to my surprise, they do! I had considered it, but turns out the discussions I was reading were outdated and did not reflect the actual implementation, which admittedly caused me to assume it was much more limited than it was. To be fair, I haven’t exhaustively tested more complex dependency chains yet, and since there’s no way for inner nix to pass references to those store paths back to outer nix in pure eval (not even with IFDs) I’d have to implement the recursion in `bash`, but this _is_ more or less the `getRunDeps` and `getBuildDeps` functionality I was looking for. My only concern is that we can only rely on it continuing to work on the condition that:
It would be nice to collect a repository of examples. 🙂
(which could become test cases or something)
> [@physics-enthusiast](#):
>
> - `recursive-nix` ends up getting stablized (very likely)
I don’t think `recursive-nix` will be stabilized anytime soon.
> [@physics-enthusiast](#):
>
> - `nix show derivation` continues to work within inner nix (fairly likely)
Or that a replacement still work.
> [@physics-enthusiast](#):
>
> - `inputDrvs` and `inputSrcs` remain accessible via the `.drv` format or analogues ([somewhat likely](https://github.com/NixOS/nix/issues/7261#issuecomment-1448089881))
I think the contention here is .drv as an implementation detail, but I think you make a great case that a condition to hide the impl detail is to provide those knobs.
> [@physics-enthusiast](#):
>
> Now that you mention it, Flakes do effectively act like IFDs. They inject arbitrary Nix expressions (when used as the input of another Flake) while themselves being invisible external dependencies that cannot be discovered by their dependents. As they are now, Flakes seem to [cause all kinds of problems](https://github.com/NixOS/nix/issues/8953#issuecomment-1728592073) for offline (re)builds in general, and cause even rebuilds on the _same_ machine to require workarounds if done without Internet access. While I concede that IFDs aren’t exactly common, Flakes _are_ and we can use `getInstantiationDeps` to address them too (and more or less deal with IFDs for “free”).
Flakes causing a bunch of problems is unrelated to Flakes as a concept. It’s a consequence of their suboptimal implementation, IMHO.
The whole offline thingie is also a consequence of excessive sharp edges too IMHO.
> [@physics-enthusiast](#):
>
> Maybe it’ll be helpful if I outline my specific usecase to help clarify what I mean by terms like “native” and “maximal percentage of packages built from source”. I want:
>
> - A collection of files
> - (Ideally) generated at the same time as a `nix build` or `nixos-rebuild` of a chosen target (i.e. the information can be contained within a Nix expression)
> - Containing **only** store paths/derivation outputs that require internet access to build
> - Which can be portably transmitted to a machine with an effectively empty store
> - Such that it is able to build said target package
> - **Without** internet access
So you basically said that you want to know all FODs to rebuild a certain thing, which is [A function `rebuildDependencyClosure` that produces an efficient closure for offline rebuilds · Issue #180529 · NixOS/nixpkgs · GitHub](https://github.com/NixOS/nixpkgs/issues/180529) for me.
To produce it at the same time during `nix build` or `nixos-rebuild` seems a UX concern, I would put that aside.
Maybe what we want is something like `nix offlinize $store_path` while being online, etc.
To transmit what is needed to build a closure to another machine seems like that, you want to have the outPath of the rebuild dependency closure and just apply classical nix copy for it.
* * *
> [@physics-enthusiast](#):
>
> Right now the options for offline backups are:
>
> - `nix copy`ing the run-time closure
> - Compact (file/folder-size-wise)
> - **But** package configuration is fixed and it cannot be rebuilt
>
> - `nix copy`ing the build-time closure or outright cloning the entire store
> - Rebuilds are possible (unless you use Flakes or copy the whole store and use the workaround mentioned)
> - **But** every transitive dependency under the sun must be copied and saved into the binary cache. At that point if you’re on NixOS and the target package is `system.build.toplevel` you may as well just move `/home` elsewhere and image the partition.
Hm, you could also do so many other things? I feel like you are talking about on-the-shelf options, not _all the options_.
You could use something like marsnix and do a filter of what you want. You could do (a) and download all FODs too.
> [@physics-enthusiast](#):
>
> The ability to back up only downloaded sources combined with Nix’s reproducibility give you the best of both worlds. By building the intermediate derivations only when you need them (and then optionally GC-ing them afterwards), you get the rebuild flexibility of a build-time closure at the persistent disk space cost of a run-time closure.
Not sure if I agree that it gives the best of both worlds, but yes, it caters to the usecase “I’d like to modify my configuration offline and rebuild it and I have spare capacity”.
> [@physics-enthusiast](#):
>
> I guess, in a sense, this (pre-)RFC is as much a request for assurance that a certain capability (hermetic dependency retrieval) will always be supported as it is a request for the implementation of a specific means of achieving it. I still believe that a primop that directly returns references to all immediate dependencies of type `` (that can be used by other Nix expressions) is still the most ergonomic, composeable and flexible solution to the problem, but since `recursive-nix` gets us most of the way there for build-time (albeit with suboptimal ease-of-use and uncertain stability like I outlined above) and Tvix’s path-scanning for run-time, I think I can settle for just `getInstantiationDeps` (or some alternative that makes possible the same ends) if my justification for the other two is considered insufficient.
It makes sense to me, I agree that something like `getInstantiationDeps` would be interesting. I wished this was built-in as a result of doing Flakes but maybe that should be one of the component for fixing the Flakes layering violations.
> [@physics-enthusiast](#):
>
> Out of curiosity, would an optional derivation advanced attribute be less “risky” to introduce than a primop, in your opinion?
Well, take for example `disallowedReferences`, it’s currently an advanced derivation attribute which is broken, it adds references back when you disallow them. Nixpkgs has a workaround for them, which effectively split the set of packages in nixpkgs into the disjoint union of bootstrap packages (non-workarounded) and anything that uses `stdenv.mkDerivation` (workarounded), which is quite problematic if you wanted to implement `disallowedReferences` the right way because you would bust all the years of cache if so.
What is the solution here? We would need derivation special attributes versioning to be able to move the ecosystem forward.
Thus, it’s the same as primops, just slightly different and less visible. And I can cite many other special derivation attributes which are broken IMHO but unfixable.
> [@physics-enthusiast](#):
>
> I’ll give it a try. I suspect `getInstantiationDeps` might require recompiling a full fork though, since I need to modify the internal behavior of existing builtins like `import` to yield additional information rather than merely adding new ones (assuming I correctly understood what plugins are able to do. Can plugins hook function calls?).
It’s OK to recompile a full fork of the interpreter, take 2.18 as a snapshot, work your feature in there, and showcase it.
“Can plugins do […]?” Well, there’s two answers:
Theoretically, yes, it’s C++, you can make any trampoline you want 😛.
In practice, I’m not so sure you can do that idiomatically.
I would just take the shortest path if you value your time, so even if it’s ugly and bad, as long as you get an interesting result, there’s value to me.
---
### Author:  [@roberth](https://discourse.nixosstag.fcio.net/u/roberth)
#### Post date: [May 13, 2024, 2:56pm UTC](https://discourse.nixosstag.fcio.net/t/pre-rfc-implement-dependency-retrieval-primitive/43418/7 "2024-05-13T14:56:41Z")
> [@RaitoBezarius](#):
>
> Though, interestingly… !
>
> ```auto
> nix-repl> :p builtins.getContext "${(builtins.toFile "abc" "${builtins.toFile "def" "ghi"}")}"
> {
> "/nix/store/kc46v2cfdvilicx6g97qqg49a2q1hi8j-abc" = { path = true; };
> }
>
> ```
The context only needs to track direct references. The closure is fine:
```shellSession
$ nix-store -q -R /nix/store/kc46v2cfdvilicx6g97qqg49a2q1hi8j-abc
/nix/store/j5hzmq2wpf6rz7bq2w27lvxr4m1jl22s-def
/nix/store/kc46v2cfdvilicx6g97qqg49a2q1hi8j-abc
```
---
### Author:  [@RaitoBezarius](https://discourse.nixosstag.fcio.net/u/RaitoBezarius)
#### Post date: [May 13, 2024, 3:33pm UTC](https://discourse.nixosstag.fcio.net/t/pre-rfc-implement-dependency-retrieval-primitive/43418/8 "2024-05-13T15:33:56Z")
It makes sense. So this reveals indeed a gap in the language-level features to introspect the closure **without IFDs** as this makes it impossible to analyze the dependencies of any `toFile` as far as I understand it.
---
### Author:  [@roberth](https://discourse.nixosstag.fcio.net/u/roberth)
#### Post date: [May 13, 2024, 4:09pm UTC](https://discourse.nixosstag.fcio.net/t/pre-rfc-implement-dependency-retrieval-primitive/43418/9 "2024-05-13T16:09:53Z")
Not exclusively in the evaluator without IFD, correct.
Getting references (or their closure) _may_ require IFD, but not when it’s a constant path, or when first taking [`derivationOf` (TBD)](https://github.com/NixOS/nix/issues/10120) on a non-dynamic derivation output, to produce build dependencies instead of output references. Currently all derivations are non-dynamic, as the RFC 92 implementation is [not quite working](https://github.com/NixOS/nix/pull/9415) yet. (That’s a lot of dense, somewhat incomplete detail; hope it helps a bit though)
---
### Author:  [@physics-enthusiast](https://discourse.nixosstag.fcio.net/u/physics-enthusiast)
#### Post date: [May 28, 2024, 5:54am UTC](https://discourse.nixosstag.fcio.net/t/pre-rfc-implement-dependency-retrieval-primitive/43418/10 "2024-05-28T05:54:32Z")
@roberth Would something like a generic function inverse be on the table? Some `inverse` such that `inverse (primop arg1 arg2)` produces `{ function = primop; arguments = [arg1 arg2]; }` (or equivalent). That way we’d only need to propagate `import` contexts through the small subset of interactions that could `forceValue` an imported object without a function call, such as attrset attribute selections and imported lambdas which themselves call other imported lambdas, since we can recover import dependencies from the inputs of any other interaction by recovering the input themselves via `inverse`. Since most `Value`s are literals or the result of a function call (and hence have empty context), the performance impact of `import` contexts would be minimal, and we would simultaneously be able to avoid exposing internal lazy/strict behaviour to the user by leaving it to them to figure out which argument to include. For this particular usecase `inverse` would be very nice, but since it directly runs counter to the functional paradigm of pure functions I suspect the proposal wouldn’t get very far.
---
### Author:  [@roberth](https://discourse.nixosstag.fcio.net/u/roberth)
#### Post date: [May 28, 2024, 12:54pm UTC](https://discourse.nixosstag.fcio.net/t/pre-rfc-implement-dependency-retrieval-primitive/43418/11 "2024-05-28T12:54:21Z")
> [@physics-enthusiast](#):
>
> directly runs counter to the functional paradigm of pure functions
Yes, this would break so many rules; it must not be done.
It’s better to have a little helper function that achieves the same goals where it’s needed, but even then, you’re exposing a rather arbitrary and unstructured piece of information whose details nobody should have to depend on. Consider for example that `tests` and `updateScript` dependencies are also in there.
---
### Author:  [@physics-enthusiast](https://discourse.nixosstag.fcio.net/u/physics-enthusiast)
#### Post date: [May 28, 2024, 11:55pm UTC](https://discourse.nixosstag.fcio.net/t/pre-rfc-implement-dependency-retrieval-primitive/43418/12 "2024-05-28T23:55:21Z")
> [@roberth](#):
>
> Yes, this would break so many rules; it must not be done.
That’s rather unfortunate, but unsurprising.
Hmm, how about an extension of the flake interface? Right now the currently unsolved problem is something along the lines of:
1. Need to capture a strict superset of all `import` _calls_ (because even a single one missing is an eval failure).
2. Need to make the superset only as big as is necessary to achieve Point 1 so we don’t end up hitting `throw`s and `abort`s that were meant to be lazy. Basically, backing up imports should never itself act as a strict operation.
3. The set required by Point 2 is slowly shrinking as more aspects of the language become lazy.
4. Point 3 renders the output of whatever mechanism used for this purpose both unstable and unstabilisable without significantly constraining future increases in laziness.
5. Because of Point 4 and implicit [pure-nix return value stability guarantees](https://discourse.nixosstag.fcio.net/t/stability-of-function-return-values-in-nix/45124/2), I can’t just make this a primop.
Leaving it to the user was an attempt to get around this, but what if I instead shove the perpetually-unstable parts into the output of some flake subcommand (let’s call it `nix flake reify`) and declare that while using it ensures you can evaluate the flake in question offline, whatever actually ends up in the file is an implementation detail? Then, the stable, well-behaved remainder (`inputDrvs`, `inputSrcs`, and dynamic drivers) can be retrieved via primop.
Also,
> [@physics-enthusiast](#):
>
> Almost, but not quite. `inputSrcs` are instantiation-time dependencies. `inputDrvs` rely on string context for dependency resolution, and it is possible to [intentionally](https://discourse.nixosstag.fcio.net/t/what-am-i-doing-wrong-here/2517/5) or [unintentionally](https://discourse.nixosstag.fcio.net/t/how-to-compute-build-time-closure-without-nix/17434) lose it. I could be wrong on this, but since `.drv` creation and string interpolation is done at eval-time, if that “invisible” `inputDrv` itself depends on some kind of eval-time fetcher (which now cannot reach the internet), you can get an evaluation failure.
after some more thinking, I’m retracting this statement. Just came to the realization that even if you _could_ lose `inputDrvs`, those missing dependencies would not be mounted in the sandbox of their dependent either, so the online build would fail too (if it tried to access said path). So build dependencies are indeed just `inputDrvs`, and instantiation dependencies are `inputSrcs` + dynamic derivers + `import`s.
P.S. To expound on my concern about `import`s, consider the following toy example:
```auto
let
instanDep =
importedBool = import (instanDep/false.nix);
in
usedElsewhere = if importedBool then derivation {...} else derivation {...}
```
Now we have an invisible dependency on `instanDep` (evaluation fails without it present). Notably, this is an issue whether `instanDep` is produced by build-time IFD of a FOD or eval-time builtin fetcher (which is how flake inputs are obtained, last I checked).
---
### Author:  [@roberth](https://discourse.nixosstag.fcio.net/u/roberth)
#### Post date: [May 29, 2024, 9:28am UTC](https://discourse.nixosstag.fcio.net/t/pre-rfc-implement-dependency-retrieval-primitive/43418/13 "2024-05-29T09:28:40Z")
Tracking which values may depend on which IFDs would be costly in terms of performance or implementation complexity.
Current solutions are either to keep track of IFDs “by hand” in the expression language, or to log them, or evaluate with a separate store that will contain all IFDs by the end of it.
By the way it’s not just `import` that triggers the IFD phenomenon, but also `readFile` and other functions that get information from the file system.
---
### Author:  [@physics-enthusiast](https://discourse.nixosstag.fcio.net/u/physics-enthusiast)
#### Post date: [May 29, 2024, 10:04am UTC](https://discourse.nixosstag.fcio.net/t/pre-rfc-implement-dependency-retrieval-primitive/43418/14 "2024-05-29T10:04:24Z")
> [@roberth](#):
>
> Tracking which values may depend on which IFDs would be costly in terms of performance or implementation complexity.
>
> Current solutions are either to keep track of IFDs “by hand” in the expression language, or to log them, or evaluate with a separate store that will contain all IFDs by the end of it.
In case I wasn’t sufficiently clear before, my previous reply was an _alternative_ to the `getInstantiationDeps` I presented in my initial post. The idea was that to get around the function stability issues with the original proposal, we give up some granular control but retain airgappability at the installable level by just logging all the `import` calls (and equivalents that result in anything that isn’t a derivation, like `readFile`) made during a given `nix build`. In other words, we would no longer track `imports` per derivation or per Nix `Value`, but rather per final flake (i.e. the one not used as an input to other flakes). By appending to a global list, we no longer need to set up any complicated hermetic tracking structures. At the end of a build the store paths in the list are copied into an opaque file that allows you to `nix eval` a single flake/flake subattribute offline. Essentially, what I’m trying to achieve is the flake version of [Deterministic evaluation of Nix expressions (and tracking who produced a derivation) · Issue #553 · NixOS/nix · GitHub](https://github.com/NixOS/nix/issues/553).
> [@roberth](#):
>
> By the way it’s not just `import` that triggers the IFD phenomenon, but also `readFile` and other functions that get information from the file system.
Not just IFDs, but for airgappability we need any `import` or `import`-equivalent that depends on the existence of something outside the store to not throw. This includes, for example, your run of the mill import-from-fetchTarball.
---
### Author:  [@pwaller](https://discourse.nixosstag.fcio.net/u/pwaller)
#### Post date: [February 16, 2025, 1:05pm UTC](https://discourse.nixosstag.fcio.net/t/pre-rfc-implement-dependency-retrieval-primitive/43418/15 "2025-02-16T13:05:44Z")
Just wanted to weigh in that the problems discussed here are of interest/importance to me. It would be great to be able to efficiently write down a derivation which is the set of dependencies required for another derivation. Of course it is sort-of possible with the existing primitives but I’ve found those to be a bit janky in practice so far.
It’s a slight tangent but I also hit a problem efficiently populating a private cache (using ephemeral machines) in [Determining which cache a cached derivation is in · Issue #354 · nix-community/nix-eval-jobs · GitHub](https://github.com/nix-community/nix-eval-jobs/issues/354). I’m aware others are happily doing this kind of thing (from Ratio’s comments above) so maybe there is already a solution to this?
Have there been any interesting developments since this discussion?