I am not sure about the cost of rebuilding CPython itself, but if we are looking at one package only during development, Python has also the advantage of being a scripting language. In nixos-rebuild-ng the package builds in less than 5 seconds including the tests in my pretty already kind old AMD Ryzen 7 4750U Pro laptop, around 15 seconds if we talk about linting (because of mypy).
And you can also run nix-shell -A nixos-rebuild-ng.devShellwith the most recent PR. This will put in a development shell where you can run python -m nixos_rebuild to test the program without compiling the package first and quickly debug an issue (breakpoint() helps a lot here). Bash in particular is also another language that is good at this point since in many cases you can simply run the script to see how it works (but debugging is another history, the available shell debugging tools are bad).
I am not sure how is the development experience in other languages (especially Rust here), but I think this is also something that matters because not everyone will have a full IDE setup to collaborate with the language of choice.
Small rust binaries again arenât orders of magnitude slower to build, especially if you take into account linting (and auto-formatting, and type checking, and style linting, âŚ). Itâll be a few seconds to a minute at most. Rust builds mostly become slow with heavy use of macros and generics, which will be less common in smaller codebases.
IME debugging programs this simple is rarely challenging enough to need a debugger, and if all you need is simple stuff like breakpoints and know your gdb as well as someone using pdb knows their python tooling you really donât need an IDE, you can absolutely use ed if thatâs your heartâs desire - itâd probably be less frustrating than using ed for Python since youâll catch more semantic errors without waiting for the code to run. And since LSP exists, if you do want an IDE setup Rust and Python are roughly the same amount of effort to set up.
Ultimately I think this is just one more example of how fruitless these discussions are; It mostly comes down to preference, with minor benefits and downsides to any language you could choose.
Familiarity on the scriptsâ core maintainersâ side is probably the most important metric (though strong type systems do have major benefits for anything but the tiniest codebases, and IMO Pythonâs type hints are so weak and cumbersome that they are just short of useless - not all of the Python community is fond of them either, with the argument that Python isnât really intended for writing âhardcoreâ software, which seriously hamstrings attempts at making them useful; but see, here we are at opinion again).
I do believe that anything small enough to be nixpkgs native will see similar code complexity, dev and runtime performance from pretty much all general-purpose languages anyone could reasonably think to use (save for JVM ecosystem stuff maybe, on account of JVM spinup times and the insanity of gradle).
I want to be clear that am not trying to downplay the Rust advantages here, it is definitely a good language and I think for bigger things (like e.g.: a Nix rewrite like Tvix) it is definitely the way to go. But at the same time I donât think it is the best language for everything and trying to downplay the disadvantages of the language is not productive either.
My main point here is that Python being a scripting language can have 0 build times in the way I said, just running python -m .... This is invaluable for debugging, especially when you donât have a proper IDE setup (also the fact that a minute is considered fast makes me worry about the mean Rust program, but this is nitpicking your argument).
Edit: auto-formatting and linters are pretty much instantaneous to run since we have ruff (written in Rust BTW) now, even in big codebases (see the benchmarks from their side). Type checking is the only thing that is still kind slow, especially since we canât reuse the mypy cache inside derivations (the same issue that Rust have with incremental builds). I am really hopeful that ruff eventually reimplements mypy, but this is for the future.
pdb doesnât really need any setup besides adding a few breakpoint() calls inside the code. I put this most as a discussion to know if Rust has something similar (e.g.: an accessible way to call the debugger). For myself I barely know how to setup gdb (I used it in college but this was 10 years ago), I probably could go back to it if I wanted but it is definitely more heavy weighted.
I donât know, I am learning a lot at least. I think the discussion has its fruits, even if there is no hard conclusion.
Heavily disagree on this one. While definitely type hints is not for every one, some of the biggest Python codebases are typed. And yes, it has its faults, but it is definitely not useless.
Almost everything that I write in Python nowadays is typed and not only it help me to find bugs but it also is an invaluable tool during debugging.
Edit: also the argument that âpart of community isnât fond of themâ is not a strong one. Python has much more programmers than Rust (according to ChatGPT between 3~4 times), it is possible that the subset of Python programmers that cares about types is bigger than all Rust programmers combined. ChatGPT says around 20~25% of all Python programmers cares about types, that would be at the same ballpark of all Rust programmers, but of course take this with a grain of salt.
Yeah, thatâs fair. I donât intend to discredit Python either, I mean, I still use it all the time, itâs great for more explorational stuff. It can be very nice to imperatively see the data (and errors) instead of having to grok compiler messages. I much prefer it e.g. for figuring out REST APIs.
My entire argument is just that the differences are ultimately very small, and well, you know best what works for you, so we should not prescribe e.g. Rust for everything, counter the post topic.
Even if it may seem like Iâd be a proponent for prescribing Rust, I just like the language a lot
Iâm also venting my gripes about the Python type system and saying why I donât think Python is great for even slightly more complex projects though, to which pointâŚ
Ranting about Python
Iâm with you, I use type hints in every Python codebase I write, itâs nice that they⌠exist.
But IME more often than not attempting to use them is an exercise in finding types in the first place, only to realize that there are no good types because everything is a dict and Python only pretends to be object oriented. You can then attempt to specify your dicts in excruciating detail, only to realize that mypy still doesnât spot half the errors, and makes up errors that donât actually exist, so you give up caring and just mark stuff as Any anyway, at which point you end up feeling like you might as well not have bothered.
Not to mention that most of the stdlib (which is half the reason people use python, batteries included and whatnot) does not have type hints, so it doesnât even help you assert your basic stdlib calls are correct.
I absolutely loved the addition of type hints, and I tried really, really hard to write typed python code, but 9 years later it still just feels like a toy.
My point about a large vocal part of the upstream community (indeed a tiny subset of the userbase) actively not wanting type hints to be part of the language isnât that theyâre not winning the popularity contest, it just makes all upstream initiatives to improve the situation have such heavy opposition that I donât see this ever changing much.
Iâve not yet used ruff in anger since stable nixpkgs didnât have a version of it with lsp support last time I did serious Python. Iâm looking forward to my next attempt, the pain of python-lsp-server (and indeed mypy) probably contributed to my diminishing opinions of Python.
All that said⌠I think weâre well off-topic. Should we move this to a âgripes and hypes of Pythonâ thread?
I think in the end this is most my conclusion too, however there is still a cost of adding new programming languages inside nixpkgs.
For example, letâs say I am writing a new tool in babashka for example. I think babashka is great, with its fast startup times and batteries included, and I used to love Clojure (probably still do), but just the binary has ~100MB and it depends in GraalVM Native Image, that at least in nixpkgs is a re-packaged binary from Oracle.
The babashka example is an obviously bad choice for anything that is considered core in nixpkgs because of it. But maybe we need to define what would make other languages a bad choice.
That said, I think it is clear that both Python and Rust are ok-ish in those aspects (since both languages are already used a lot in nixpkgs). But Go for me it is less clear, even if it looks acceptable from the answers I got in this topic.
It doesnât feel like your rants match my experience. Let me put it here:
Python counter-rant
I am writing modern Python in nixos-rebuild-ng (and by modern I mean I am actively using new features, e.g.: TypedDict from 3.9, Unpack and assert_never from 3.11, etc.), and except for tests I am not using Any anywhere.
Also my impression is that the community is largely pushing for improvements in the type system, all those things that I described (e.g.: Unpack) were created so we can express more complex types.
Maybe your experience is simply outdated? Or maybe not, I canât really speak for you, but I can give my point of view.
I donât want to discuss gripes about Python, I mostly want to expose where Python may make sense in the context of nixpkgs, that I concur is not everywhere.
But yes, this conversation is going way out of topic.
Maybe we shouldnât attempt to prescribe a single Nixpkgs programming language. Clearly, there is no obvious answer, and trying to pick among the favored ones is just going to lead to (thankfully friendly) debate about the merits of different programming languages.
However, we could at least have some set of âpreferredâ programming languages. Not to disallow programming languages from being used when appropriate. e.g. for PHP-specific tooling, using PHP would be fine if desired, but using PHP elsewhere would be undesired. Not because PHP is bad or anything, but because I think we can agree there would be value in generally limiting the set of programming languages in Nixpkgs, even if itâs only informally enforced.
I think Python and Rust make the cut easily. The one place I will probably get in trouble here is I would suggest Perl not be in the list: I have grown to respect Perl, but I think if weâre all being honest itâs not among the best programming languages for writing easy to maintain code. Other compiled languages like Go and Zig should probably be left out: theyâre great (I am somewhat of a Go zealot personally) but I think Rust is enough for that category. The most interesting other category to me would be more advanced shell interpreters like Nushell and Oil, but I know nothing about them to debate their merits.
Of course, maybe instead of simplifying the discussion Iâve just come up with a way for it to be even worse. But if we could come up with a small set of programming languages that cover a few niches and pick languages that we feel will lead to more maintainable code, that seems like it would be useful, even if only as a documented guideline and not a hard-and-fast rule.
I used to picked a language only in perspective of its inherent features like others in this thread. Now I nevertheless change my mind. It may be the libraries rather than the language itself that matter for Nixpkgs. Bash is awkward in every sense except for its mere existence as a system shell. However, with tons of CLI utilities the experience with it becomes less bad. In contrast, a âvery soundlyâ designed or implemented language (e.g. Scheme) may fail in scripting fluency for lack of fistful libraries. If a language is to get its place in Nixpkgs, the libraries of it, either shipped within or officially selected by Nixpkgs, must at least replicate the success of shells and their friends. That is:
easy yet powerful string and path manipulation, at least as easy & powerful as regex;
codec and pretty-print for popular data exchange formats, at least for JSON;
immediate file and process IO;
immediate HTTP GET;
and so on.
The standard libraries of Python and Ruby satisfy the first two clauses, but not the others. Smaller languages/implementations like Lua and Chez Scheme typically only support one of these features. Of course we can provide the libraries needed to nix-shell just like normal applications, but the choices made by different contributors would soon diverge like the languages themselves. So in addition to, and as a part of the selection of a preferred language, weâd better pick a set of preferred libraries as well.
By âimmediateâ I mean external commands / Web resources as accessible as normal functions and objects. Probably this needs a DSL (yes, a shell) to attain.
Actually, I proposed them just because of my poor experiences with the standard libraries you mentioned of Python.
subprocess.run is actually a really good API. Yes, it is kind verbose, but the nice thing about it is that it is also really clear what is happening (e.g.: check=True means that the return code is being checked, capture_output=True is self describing, etc.).
The part that the API I think it is bad is the equivalent of | (pipelines) in shell (there is subprocess.Popen, but it is really unpleasant to use and verbose), but they rarely matter since Python you can replace most of the text manipulation with what Python offers you.
nixos-rebuild-ng is actually a good show because nixos-rebuild uses lots of pipes while nixos-rebuild-ng uses none of it. And this reflects my experience with converting other shell scripts to Python.
I concur that urllib is lower level than what most people want. requests is much better though for just doing basic requests though, and you can easily pull it as dependency if you need in nixpkgs.
I used to find the verbosity annoying, but Iâve grown to really enjoy the subprocess.run API. Iâve been using it more lately since I found shell=True to call with a simple string instead of dealing with an array of args.
Using shell=True is generally discouraged since this mean youâre the mercy of the shell and all its pitfalls, for example shell injection is really easy to do if you donât take care to escape user input.
With shell=False this is much more difficult to happen because the result is not interpreted by the shell, however you still need to take care for some commands like I recently discovered with ssh.
Of course, this doesnât matter for user scripts, but for âseriousâ code it should be avoided.
Good to know and call out. Thanks! Iâm only using it for personal scripts, without user input, but definitely want to avoid shell injection in serious apps.
I want to make an observation that Python3 is by no means a small contribution to the closure size. python3 is 108MiB just by itself, and pulls in a number of dependencies that are of a non-trivial size (sqlite, a number of (de-)compression libraries, openssl, mail stuff, etc.) for an overall closure size at an approximately 175MiB. I have no doubt this size will keep on increasing at a relatively quick pace given the (C)Pythonâs development. But I digress.
For regular desktop systems the system closure size is probably not a huge deal, but when Nix is used to build containers, VMs (especially the micro variety), SD images for *Pis and other such artifacts, python3 slipping in can have quite an adverse affect on the image size. And python becoming an official choice will introduce it into system images in no time (if only as an implementation of stage1.)
Rust binaries are also known to be of somewhat non-trivial size but you can still fit a good amount useful logic across a good number of binaries into a 108MiB of budget and for that reason alone Rust suggests a much better value proposition. Not to mention that building a busybox-style binary is an effective way to mitigate the per-binary duplication. Fitting into 1.6MiB that fits a bash is perhaps a stretch for an average Rust binary as well, but beating pythonâs 108MiB would require quite some effort (and mistakes.)
Python3Minimal size is 29mb, closure size 70mb (glibc is 31mb which is given) while surprisingly nutshell is 40mb (closure size is 85mb)
I worked with Python professionally and as a hobby, and all I can say is that I donât like it BUT I donât see any issue with using it for non-builder scripts, like nixos-rebuild and other things required for NixOS system since somewhere down the chain something will require it anyways
That being said, I wish that minimal NixOS could be built much like BottleRocket OS (TL;DR: no interpreters)
For builders and anything else I like mRuby due to its size, power of ruby, and ease of bootstrap. Tho I realize that Iâm probably the only one who thinks thatâŚ
Also building more of a âruntimeâ out of quickjs is an option, there already are several options but yeah⌠It does not sound like the brightest idea around
I think this is a somewhat unfair comparison because you canât do much in Rust without importing third party packages, while in Python you can do a lot with just standard library offers.
I am almost sure that you canât reimplement nixos-rebuild in Rust without third-party libraries for example, while the current version of nixos-rebuild-ng has none, but of course I may be wrong here.
And as @gytis-ivaskevicius said there is python3Minimal that is much smaller.
I still think in general Rust still has the advantage in the closure size, but this may not be as big of as a gap that most people think once you factor the 2 things above.
In fairness, this depends heavily on your choice of library and what features you enable. Rust is ultimately a compiled language, it should handily beat python on size if configured properly. You could easily link against C++ libs to achieve small full-system closures if you wanted to prove the point (though thatâd make all the nice things about Rust go away).
Sharing dependencies between multiple installed projects is of course a big deal, if this really is a problem we could probably achieve dynamic linking with Rust in nixpkgs.
Hm, to me, on the technical merits, ignoring social aspects, thereâs an absolutely clear winner, by a wide margin, Go:
has simple type system, which helps to avoid stupid typos while doing âsystem adminâ tasks, while keeping the code easily understandable even for someone who looks at it for the first time,
produces statically-linked binaries, trivially cross-compilable, which makes it easy to admin via scping the binaries around,
has a reasonable dependency story, if more complex functionality is necessary,
is fast to compile, to the point that go run is a reasonable scripting workflow,
has home-grown codegen, so should be much faster to bootstrap from C than something that pulls llvm.
In particular, comparing with Rust, in the context of admin-like tasks, the benefits of Rust are not important at all, but the drawbacks feel like they can be significant. Nixpkgs doesnât need absence of gc, statically-checked thread safety, ability to write really fast code. In contrast, the requirement to deal with borrowing, the ability to go really smart with types, slow compile times, and complicated bootstrapping feel like they are relatively important.
That being said, Rust I think is fine for this sort of stuff, itâs just that it feels like nixpkgs is really well suited for Go (saying as a Rust&Zig programmer, lol).
