Commit Graph
100 Commits
Author SHA1 Message Date
Erik Arvstedt 2c8e29b35b lnd: extract option certPath
Improves service encapsulation.
2021-09-11 15:07:24 +02:00
Erik Arvstedt be12a49933 lightning-pool/loop: extract lnd variable 2021-09-11 15:07:24 +02:00
Erik Arvstedt 955b44404c delete helper/fetch-channel
This script was obsoleted by switching to flakes.
2021-09-11 15:07:23 +02:00
Erik Arvstedt 5087ce245f minor cleanups
- btcpayserver: remove unneeded trailing semicolons

- krops/get-sha256:
  `tail` is unneeded because `nix-prefetch-url` just outputs a single
  line containing the hash.
2021-09-11 15:07:23 +02:00
Erik Arvstedt 0d2db4e79f backups: add option postgresqlDatabases
This simplifies defining postgresql backups.
This change is covered by tests.py.
2021-09-11 15:07:23 +02:00
Erik Arvstedt c35e96a553 joinmarket: update patch hash
The patch hash has changed due to an update of the PR branch.
The PR has now been merged.
2021-09-05 22:33:17 +02:00
Erik Arvstedt 32ce9d0ff4 run-tests: fix 'eval' command for newer versions of nix
There's no common `nix` command argument syntax for eval'ing a nix
expression that supports both older and newer (flake support) versions of nix.
So fall back to nix-instantiate.
2021-09-04 08:17:38 +02:00
Erik Arvstedt 926f1febb7 make-container: update extra-container version
Keep this file in sync with the latest extra-container update.
2021-09-04 08:17:38 +02:00
Erik Arvstedt 9730be9282 joinmarket-yieldgenerator: simplify start script 2021-08-30 13:37:05 +02:00
Erik Arvstedt 179b86d19c joinmarket: allow recreating wallet from seed
This allows users to easily upgrade their wallets to use Fidelity Bonds.
2021-08-30 13:37:05 +02:00
Erik Arvstedt 7c5ef32b50 versioning: move list of changes to the top
Improves readability.
2021-08-30 13:37:05 +02:00
Erik Arvstedt b15d71605e joinmarket: fix leaking passwords
Previously, `bitcoin-rpcpassword-privileged` and `jm-wallet-password` were
passed as world readable arguments to sed and jm-genwallet subprocesses.
2021-08-30 13:37:04 +02:00
Erik Arvstedt d7f9e33e1c joinmarket-ob-watcher: move resource files to extra dir
Don't clutter joinmarket/bin with ob-watcher resource files.
2021-08-30 13:37:04 +02:00
Erik Arvstedt dde04f8cbe update nixpkgs-unstable
Includes:
btcpayserver: 1.1.2 -> 1.2.0
lightning-loop: 0.14.2-beta -> 0.15.0-beta
nbxplorer: 2.1.52 -> 2.1.58
2021-08-26 12:45:10 +02:00
Erik Arvstedt 87df809a88 add helper/update-flake.sh
This greatly simplifies updating nixpkgs.
See the comment at the top of update-flake.sh for a description.
2021-08-26 12:45:10 +02:00
Erik Arvstedt f7c2133250 add flake support
This change is fully backwards compatible.

We continue to use the standard non-flake evaluation mode in our
examples and internal tooling until the flakes design has stabilized.

'clightning-plugins = pkgs.recurseIntoAttrs' in pkgs/default.nix is
needed by flake-utils.lib.flattenTree in flake.nix.
It transforms the packages in `clightning-plugins` to top-level packages
named like `clightning-plugins/summary`. (The flake attr `packages`
must be a non-nested attrset of derivations.)
2021-08-26 12:45:10 +02:00
Erik Arvstedt de77281cba pkgs: import pinned nixpkgs in default.nix
pkgs/default.nix now explicitly specifies all its dependencies as arguments.
This is required for flake support.

Also simplify pinned.nix and python-packages by removing unused attrs.
2021-08-16 10:43:07 +02:00
Erik Arvstedt b0c66c41e1 tests: add container-minimal example 2021-08-15 22:42:03 +02:00
Erik Arvstedt a8a8b9ce4d backups: backup NixOS uid, gid mappings
Now that service uid, gid mappings are included in the backups, along
with the service data dirs, we can remove 'chown -R' for
clightning and liquidd data dirs.

Note that we used 'chown -R' only for these two services, while this
approach would have been relevant for all services with data dirs.
2021-08-15 22:40:35 +02:00
Erik Arvstedt ee8b83681b modules: document module dependencies 2021-08-15 22:40:35 +02:00
Erik Arvstedt 9f7d048769 modules: move assertion to lnd.nix
nix-bitcoin.nix is now no longer dependent on clightning.nix and lnd.nix.
Due to condition '!(config.services ? clightning)' lnd.nix still
doesn't depend on clightning.nix.

Also fix the assertion message by renaming clightning.bindPort to clightning.port.
2021-08-15 22:40:35 +02:00
Erik Arvstedt cce9a3f6b2 modules: move nix-bitcoin options to file 'nix-bitcoin.nix'
This allows modules.nix to consist only of a list of modules.
2021-08-15 22:40:35 +02:00
Erik Arvstedt fdc278a0b8 lib: fix comment 2021-08-15 11:29:36 +02:00
Erik Arvstedt 13b4650e84 versioning: add usage comment 2021-08-15 11:29:36 +02:00
Erik Arvstedt ca3c7a281b secrets: mark option 'secretsSetupMethod' as internal 2021-08-15 11:29:36 +02:00
Erik Arvstedt f9a0fd7a17 nodeinfo: fix indentation 2021-08-15 11:29:36 +02:00
Erik Arvstedt 4ece606e8b examples/minimal-configuration: improve comment 2021-08-15 11:29:35 +02:00
Erik Arvstedt 6de9aba854 run-tests: quote scriptDir
scriptDir may contain spaces.
2021-08-15 11:29:35 +02:00
Erik Arvstedt 1ef8cbb384 joinmarket: fix allowRunAsUsers setting
This option requires user names instead of groups.
2021-08-15 11:29:35 +02:00
Erik Arvstedt fb36f2abe5 joinmarket-ob-watcher: use consistent mode formatting
Remove redundant leading zero.
2021-08-15 11:29:34 +02:00
Erik Arvstedt f14af1fc48 treewide: use consistent echo message formatting
Quote the echo message.
2021-08-15 11:29:34 +02:00
Erik Arvstedt b8043d3db5 treewide: use consistent bash script indentation
Always use two spaces.
2021-08-15 11:29:34 +02:00
Erik Arvstedt c758d68ea4 lib: rename privileged -> rootScript
The naming is now analogous the related function `script`.
2021-08-15 11:29:34 +02:00
Erik Arvstedt 1c3735b600 examples/README: add nixbitcoin.org server repo 2021-08-15 11:29:33 +02:00
Erik Arvstedt c041079ae1 configuration.nix: reorder sections
Move backups and netns-isolation to the end.
2021-08-15 11:29:33 +02:00
Erik Arvstedt 3734ab38a6 configuration.nix: improve wording and formatting 2021-08-15 11:29:33 +02:00
Erik Arvstedt a2454975a5 doas: fix recursive calls to doas
Doas was broken for recursive calls like `doas -u operator lncli`
where `lncli` internally calls doas.
2021-08-14 10:46:42 +02:00
Erik Arvstedt 7c876664b1 netns test: update matching of 'capsh' output
The output now contains multiple lines.
2021-08-14 10:46:42 +02:00
Erik Arvstedt 308a11f22b tests: avoid postgresql timeout failures on CI nodes 2021-08-14 10:46:42 +02:00
Erik Arvstedt 01804e6dfb tests: improve test script formatting
Remove annyoing spacing constraints enforced by the previous 'black' linter.
2021-08-14 10:46:42 +02:00
Erik Arvstedt 1be924529d tests: adapt to new linter
The Python test driver now uses 'pyflakes'.
Remove hacks that were needed for the 'black' linter.
2021-08-14 10:46:42 +02:00
Erik Arvstedt c1c663d0a9 tests: fix formatting 2021-08-14 10:46:42 +02:00
Erik Arvstedt c4c2b03e19 extra-container: 0.6 -> 0.7
Version 0.7 adds support for NixOS 21.05.
2021-08-14 10:46:41 +02:00
Erik Arvstedt 161baa7e68 joinmarket-ob-watcher: allow required 'mbind' system call 2021-08-14 10:46:41 +02:00
Erik Arvstedt ca64a4a64f clightning-plugins.prometheus: use current nixpkgs version of prometheus-client 2021-08-14 10:46:41 +02:00
Erik Arvstedt 3aab1fc267 spark-wallet: update to new node-env 2021-08-14 10:46:41 +02:00
Erik Arvstedt a0e5894f1f backups: remove illegal option definition 2021-08-14 10:46:41 +02:00
Erik Arvstedt 35fe939cf8 security: update /proc restriction mechanism
NixOS option `security.hideProcessInformation` for globally restricting
access to /proc has been removed.
Use per-service restrictions via 'ProtectProc' instead.

Rename
`nix-bitcoin.security.hideProcessInformation` to
`nix-bitcoin.security.dbusHideProcessInformation`
because this option now only implements the dbus restriction.
2021-08-14 10:46:41 +02:00
Erik Arvstedt 178a0dcf8f services: use new 'tor' options 2021-08-14 10:46:41 +02:00
Erik Arvstedt e44f78ebb8 services: set isSystemUser for service users
'isSystemUser' has to be explicitly set in NixOS 21.05.
Previously, it was the implicit default.
2021-08-14 10:46:40 +02:00
Erik Arvstedt 0ef66c920b treewide: use services.getty option
services.mingetty is equivalent but deprecated.
2021-08-14 10:46:40 +02:00
Erik Arvstedt a25ceecca5 update to NixOS 21.05 2021-08-12 11:18:26 +02:00
Erik Arvstedt b758150c9e pinned: expose nixpkgsStable, nixpkgsUnstable
This allows accessing the pinned nixpkgs.
E.g., this is useful for comparing package versions between stable
and unstable.
2021-08-12 11:18:08 +02:00
Erik Arvstedt 881a3aa004 joinmarket: use preStart, postStart 2021-08-10 10:12:42 +00:00
Erik Arvstedt ed364f5932 tests: speed up stopping of clightning with clboss
This speeds up the 'backups' test by 90 s.
2021-07-29 20:27:34 +02:00
Erik Arvstedt 637a58d826 lnd: improve waiting for active RPC server
- Simplify
- Add comment
- Avoid the unneeded default call to sleep
2021-07-29 20:27:34 +02:00
Erik Arvstedt adeccce06e lnd: simplify use of curl 2021-07-29 20:27:33 +02:00
Erik Arvstedt 7ae0a38701 electrs: replace local version with nixpkgs 2021-05-04 11:31:35 +00:00
Erik Arvstedt c5f67629e6 joinmarket: add copy of cryptography 3.3.2 2021-04-07 19:05:40 +02:00
Erik Arvstedt 3cce822c64 run-tests: add 'vm' command 2021-03-26 23:23:22 +01:00
Erik Arvstedt 4cddf284e9 treewide: remove use of deprecated stdenv.lib 2021-03-22 14:39:32 +01:00
Erik Arvstedt 6ba5880b2c test.py: improve composability of test 'banlist-and-restart'
The test now works if not all services previously used in the
`systemctl restart` command are available.
2021-03-22 14:39:31 +01:00
Erik Arvstedt 44439e2a81 tests: optimize building multiple tests at once
The result of `import tests.nix {}` is now an attrset of tests.
This makes it easier and more efficient to evaluate or build multiple
tests in one call to `nix build`.

Simplify tests.nix by removing the large module args scope in favor of
self-contained scenario module definitions.

Add CPU core and memory size defaults to the test configuration to
enable building tests without `run-tests.sh`.

Add the following top-level args to tests.nix:
- `extraScenarios` to provide a nix-level way to define extra scenarios.
- `pkgs` to allow building tests with custom pkgs or systems.
2021-03-22 14:35:29 +01:00
Erik Arvstedt 9ca52af523 tests: improve make-test-vm.nix
1. fixedTest: Remove some unneeded layers of function calls.

2. Don't add a modified version of `black` to the global pkgs set.
   Tests should not affect the pkgs of the tested system modules.
   Fix the driver build script instead by adding an extra arg to the
   call to `black`.
2021-03-22 14:35:29 +01:00
Erik Arvstedt 08fe9ba84a services: add finer-grained address family restrictions
Due to a possible NixOS bug, this commit has no effect on NixOS 20.09
where `RestrictAddressFamilies` is a no-op.
It's only relevant for NixOS unstable with cgroups v2.

bitcoind+zmq: instead of allowing all address families, only add the required
AF_NETLINK family.

lnd: lnd only runs a zmq client, not a server, therefore it requires
no additional address families.

lightning-pool, clightning-plugin-zmq: add AF_NETLINK.
2021-03-22 14:35:29 +01:00
Erik Arvstedt 020433cec6 services: add helper fn setAllowedIPAddresses
Also use 'allowLocalIPAddresses' instead of 'allowTor' in bitcoind-import-banlist
which doesn't use Tor.
2021-03-22 13:20:45 +01:00
Erik Arvstedt cdf27d9d0c bitcoind: improve service timeouts
- Improve readability by using minutes
- set `TimeoutStopSec` like in bitcoin/contrib/init/bitcoind.service.
  Stopping bitcoind can exceed the default timeout during IBD.
2021-03-16 12:46:19 +01:00
Erik Arvstedt 09cd3ce5e4 lnd: show curl error messages 2021-03-16 12:46:19 +01:00
Erik Arvstedt d214605b32 spark-wallet: add flakes compatibility
Pure flakes can't use NIX_PATH.
2021-03-16 12:46:19 +01:00
Erik Arvstedt 81db927f66 spark-wallet/generate: remove supplement.json
This file is empty and has no effect.
2021-03-16 12:46:19 +01:00
Erik Arvstedt 84b3217c3d fetch-release: minor improvements
This script is potentially fetched from an untrusted source and should
be in good shape to be easily auditable.

- Create just one TMPDIR
- Improve comments
- Use `cut` to extract sha256
- Use camelCase var names like in other scripts
2021-03-16 12:46:18 +01:00
Erik Arvstedt 45d0964e27 examples/shell.nix: minor improvements
- Use idiomatic var name `pkgs` for the imported nixpkgs.
- Don't add `figlet` to PATH because it's only used internally.
- Only print figlet in interactive shells to avoid interfering with
  stdout when running `nix-shell --run <cmd>`.
- Define `fetch-release` as a function to enable running it via
  `nix-shell --run fetch-release`
2021-03-16 12:46:18 +01:00
Erik Arvstedt cc7149eb78 examples: improve robustness of deploy scripts
- The scripts now work within arbitrary nix-shells.
  Previously, they failed when run from nix shells other than
  `examples/shell.nix`.

- The scripts now work from arbitrary working dirs.
2021-03-16 12:46:18 +01:00
Erik Arvstedt e8b47f099c examples: add deploy-krops.sh 2021-03-15 19:02:58 +01:00
Erik Arvstedt 03db1a61b1 lnd, joinmarket: don't write to secrets dir
Keeping the secrets dir read-only is more simple and robust.

- lnd seed mnemonic creation and joinmarket wallet creation can be
  run as the regular service user instead of root.

- It is easier to switch to a third-party secrets deployment
  method in the future.

Don't create a seed mnemonic for lnd when a wallet exists.
This avoids creating unused mnemonics and helps simplifying
the migration command in `versioning.nix`.
2021-03-15 18:50:15 +01:00
Erik Arvstedt 55d87490ec secrets: make configuration more robust
- Fail at evaluation when secrets setup is not configured.
  Previously, bitcoind failed at runtime due to the missing secrets target.

- Fail at evaluation when conflicting secrets setup methods are used.
  This happens when `secretsSetupMethod` has more than one definition.
2021-03-15 12:42:52 +00:00
Erik Arvstedt b0c223e716 secrets: don't stop services when the secrets target stops
With krops or nixops the secrets target is always restarted during
the deployment process.
This previously caused unnecessary restarts of all nix-bitcoin services.
2021-03-15 12:42:52 +00:00
Erik Arvstedt b701cb5603 secrets: add option 'generateSecrets'
Move this feature from a module preset to a regular option, so that it's
easily discoverable and accessible.

Simplify the implementation of `generateSecrets` by adding it to the
existing `setup-secrets` service script.

Also rename option setup-secrets -> setupSecrets.
2021-03-15 12:42:52 +00:00
Erik Arvstedt 03515a8da6 examples/qemu: disable hardened preset
- Hardening is irrelevant in examples
- The decreased performance is especially noticeable in VMs
2021-03-15 12:42:52 +00:00
Erik Arvstedt 908af3bfb8 examples/deploy-qemu-vm: show progress when waiting
This improves the user experience on VM startup, which can take a few
seconds.
2021-03-15 12:42:51 +00:00
Erik Arvstedt ccba86a0f0 examples: extract common features for running QEMU VMs
Needed by commit: "examples: add deploy-krops.sh"
2021-03-15 12:42:51 +00:00
Erik Arvstedt cb3d5446c8 examples: use relative paths to example files
This ensures that the examples files from the current example dir are used.
Previously, the default examples from the nix-bitcoin source were used.
2021-03-15 12:42:51 +00:00
Erik Arvstedt 8e3feece67 run-tests: fix interrupt handling for --copy-src
Previously, `run-tests.sh --copy-src ...` exited with status 0 (success) when interrupted (SIGINT).
It now exits with an error status.
2021-02-12 21:39:46 +01:00
Erik Arvstedt ce2b445777 treewide: use runuser for dropping privileges
When running as root, use runuser instead of sudo.
As opposed to sudo or doas, runuser is a standalone
binary that needs no external configuration.
Also, it's a bit faster.
2021-02-09 12:44:01 +00:00
Erik Arvstedt 0a2c8e4864 run-tests: add option --copy-src 2021-02-08 12:20:20 +01:00
Erik Arvstedt 803584a288 backups: don't use hardcoded secrets dir 2021-02-07 22:45:38 +01:00
Erik Arvstedt c29d44b49a ci: use 'cachix watch-exec'
Simplifies the build script.
This feature appeared in a recent cachix update.
2021-02-07 22:45:37 +01:00
Erik Arvstedt 6a32812412 services: add names for systemd helper scripts
The systemd journal now shows a specific script name instead of
the generic name "script" before script output.
2021-02-07 22:45:36 +01:00
Erik Arvstedt 6982699613 services: use consistent layout
Use the following order of definitions for all services:
- assertions
- configuration of other services
- environment.systemPackages
- tmpfiles
- own service
- users
- secrets
2021-02-07 22:42:23 +01:00
Erik Arvstedt a43534dda0 services: improve config file setup
- btcpayserver, nbxplorer: Add quotes to the
  dataDir arg. (dataDir can contain spaces.)

- clightning, liquidd: use 'install'
2021-02-07 22:42:22 +01:00
Erik Arvstedt 18f2002cf0 joinmarket-yieldgenerator: improve systemd journal output
Journal entries now look like
`joinmarket-yieldgenerator[9795]: User data location: /var/lib/joinmarket`
instead of
`bash[9795]: User data location: /var/lib/joinmarket`
2021-02-07 22:41:46 +01:00
Erik Arvstedt 9d0b8c8f6f joinmarket-ob-watcher: use DynamicUser
DynamicUser simplifies services that don't need a persistent uid/gid,
like joinmarket-ob-watcher.

For existing installations the data dir migration to dynamic users
is automatically handled by systemd.
2021-02-07 22:41:44 +01:00
Erik Arvstedt e9c98f415c joinmarket: explain need for tor control socket 2021-02-07 22:41:31 +01:00
Erik Arvstedt d9c87b6a8f joinmarket: fix wallet creation
- Fix jm-wallet-seed being globally readable.

- Handle seed extraction failures.
  If seed extraction fails, remove the newly created wallet.
  This guarantees that wallets always have an accompanying seed.
2021-02-07 22:41:31 +01:00
Erik Arvstedt 7458350108 treewide: remove deprecated types.loaOf 2021-02-07 22:41:31 +01:00
Erik Arvstedt 9cf038939c treewide: use mkEnableOption 2021-02-07 22:41:31 +01:00
Erik Arvstedt 7a97304f13 treewide: remove unit descriptions
Systemd's `Description` option is a misnomer (as confessed by `man systemd.unit`):
Its value is used by user-facing tools in place of the unit file name, so this option
could have been more aptly named `label` or `name`.
`Description` should only be set if the unit file name is not sufficient for naming a unit.
This is not the case for our services, except for `systemd.services.nb-netns-bridge`
whose description has been kept.

As an example how this affects users, weird journal lines like
```
nb-test systemd[1]: Starting Run clightningd...
```
are now replaced by
```
nb-test systemd[1]: Starting clightning.service...
```
2021-02-07 22:41:31 +01:00
Erik Arvstedt a942177ecf treewide: remove user descriptions
User descriptions are stored in the `comment` field in /etc/passwd.
In our case, these are completely redundant and don't add any useful information.
2021-02-07 22:41:30 +01:00
Erik Arvstedt 4f6ff408ef treewide: remove unneeded string literals 2021-02-07 22:41:29 +01:00
Erik Arvstedt e6a6c721c1 treewide: streamline 'extraConfig' descriptions 2021-02-07 22:40:11 +01:00