The future of `writers`, `writeTextFile`-based build helpers, and and their `checkPhase`

Development Nixpkgs Architecture
1

Summary

We currently have three groups of build helpers that create a text file or a compiled binary from a given set of Nix strings. This topic aims to describe the current situation and the problem around the checkPhase and to build consensus about the future use/changes to the file-generating build helpers.

Trivial build helpers and writers

Among the trivial build helpers (pkgs/build-support/trivial-builders), a group of build helpers based on writeTextFile (which itself is a runCommand-based build helper) is used to create a text file, which includes writeText[Dir] and write[Shell]Script[Bin] and writeShellApplication; while a group of build helpers named write<Lang>Bin produce compiled executable binaries based on runCommand.

On the other hand, we have writers, which is a set of build helpers based on writers.makeScriptWriter and writers.makeBinWriter, and use the same interface to write to $out and $out/bin/<name>.

The checkPhase

The writeTextFile-based build helpers have a checkPhase executed after the output text file is generated. It checks the generated files at/inside $out in build helpers such as writeShellScript[Bin] and writeShellApplication, which is roughly equivalent to the installCheckPhase in stdenv.mkDerivation semantics. The writers don’t seem to provide a similar interface.

Here comes the tricky part: how should one modify/compile a text script they generated using the writeTextFile-based build helpers? The discussion started in ldub’s introduction of writeArgbashShellApplication, a new build helper based on writeShellApplication with Argbash’s compilation, but modify the generated Bash script with Argbash to make the argument specification in the script comment take effect. The Nixpkgs Reference Manual documents the checkPhase of writeTextFile as “Commands to run after generating the file,” implying that the modification should be placed inside doCheck. Still, it’s surprising that something like a checkPhase can modify the output files.

A straightforward solution to this problem is to place the modification to its specific phase, which was what NixOS/nixpkgs#243023 proposed initially. However, whether to name the new phase postBuild or postIntsall soon became the PR’s blocker. As mentioned, the checkPhase of writeTextFile was equivalent to the installCheckPhase of mkDerivation. If we name the new phase postInstall, it might be surprising to see postInstall come before checkPhase, but postBuild doesn’t reflect the “writing to output” nature of writeTextFile. Trying to solve the dispute, the PR evolves to become a checkPhase → installCheckPhase renaming for writeTextFile, making it much more bloated than initially planned.

Even after we make it to rename checkPhase to installCheckPhase (<pre/post>Check to <pre/post>InstallCheck) and introduce postInstall, it’s natural for people to expect writeTextFile to take preInstall, which makes me wonder if it would be easier to base writeTextFile on mkDerivation instead of runCommand.

Are writers replacing writeTextFile?

As complex build helpers like writeShellApplication are built on top of writeTextFile, writers focus on unifying the interface between build helpers that write to $out or to a file under the $out directory. There are already many writers for different scripting languages and compiled languages. Will they eventually replace the writeTextFile-based build helpers? If so, would something like checkPhase be implemented there?

2 Likes
2

I can’t speak to the ~big question here, but I do have some in-the-small thoughts…

  • I agree that the semantics of using checkPhase for this aren’t good, but I’ll note that I’ve abused it in exactly the same way to implement resholve.writeScript[Bin].

  • I’m tempted to see the proliferation of write[A][B][C] forms as a smell that something’s broken or poorly abstracted or has too much friction or that we need to promote better patterns?

    (I can’t quite put my finger on what atm, but I suspect the fact that we’re here having this conversation when it feels straightforward to figure out where this argbash invocation would fit into mkDerivation is pointing to some kind of conflict between the flexibility that comes with mkDerivation, a demand for simpler/lower-boilerplate abstractions, and maybe a lack of modular reusability at the right level of granularity to enable us to compose stuff like this?)

  • At least if this does ultimately use the writeABC style interface, I do feel like it might ~help with the sense of combinatorial explosion if it’s just a passthru function attached to argbash as argbash.writeShellApplication.

3 Likes
3

I have no particular opinion about this. Just wanted to mention #124080, which I created a long time ago and offers an alternative method of generating such scripts. I don’t know whether it’s better or worse, though :slight_smile:

1 Like
4

The idea of dispatching the writers to their specific language implementations is great, given the amount of them in Nixpkgs. Even if writeShellScript[Bin] can’t be bootstrapped trivially, I think at least it is worthy to disassemble the makeScriptWriter-based writers. Now that they are not documented in the Manuel yet, (gradually) deprecating them should not lead to a chaos.

5

Irrespective of whether something is in the manual or not, people will end up using it. And what specifically isn’t documented here?

6

pkgs.writers. Actually there are documentation comments, but currently they don’t occur in the Manual, of stable or unstable channels.

7

Then I can’t agree with your previous comment; pkgs.writers are well-used.

8

What I am complaining about is not the interfaces these writers themselves expose, but their location in the source tree and their attributes in the pkgs hierarchy. Once we move them to what they are implemented in, it should not be very hard to replace the existing uses with the new locations.