Commit Graph
100 Commits
Author SHA1 Message Date
Erik Arvstedt 9662c19ab1 onionServices: use actual user name of services
Previously, onionAddresses definitions in onionServices were of the form
onionAddresses.access.<service> = [<service>];

This caused failures for configurations where a service user name was
overridden or for bitcoind whose default user is 'bitcoin' instead of 'bitcoind'.

Now set the equivalent of:
onionAddresses.access.<actualServiceUser> = [<service>];

Implement this via a new option `onionAddresses.services` to make things more
readable and to work around an infinite recursion error in onionServices.
2021-01-31 22:26:49 +01:00
Erik Arvstedt 5c09845e6f bitcoind: tag incoming connections as onion on enforceTor 2021-01-31 22:26:49 +01:00
Erik Arvstedt 8f9ea61d6e update nixpkgs-unstable
- bitcoind 0.20.1 -> 0.21.0
  Manually create a wallet in the backup test because bitcoind
  does not create a default wallet anymore

- disable the failing elementsd build on unstable
2021-01-31 22:26:30 +01:00
Erik Arvstedt 05e5ec99ec modules packages: build electrs, lightning-loop with nixpkgs stable
Building with nixBitcoinPkgsUnstable was only a temporary measure to
fix build errors on stable.
2021-01-30 11:38:47 +01:00
Erik Arvstedt 44546561fc run-tests: allow defining scenarios via cmdline args
This simplifies running self-contained scenarios for testing and debugging.
2021-01-30 11:38:47 +01:00
Erik Arvstedt fc40776689 improve backup test
Only check enabled services. This allows running the backup test with a custom subset of other
tests.

Also, show a meaningful error on test failure.
Previously, just an AssertionError without a message was shown.
Because the test code is evaluated from a string and not a file, there
was also no backtrace to the tests file.
2021-01-30 11:38:47 +01:00
Erik Arvstedt 9a67a32779 fix build-to-cachix
- Don't fail on `kill $cachixPid` when cachix has already exited.
  This fixes some failing CI builds.

- Stop the cachix background worker before the final `cachix push`.
  This can avoid unneeded reuploads.
  Use the coreutils version of tail on cirrus which supports
  argument --pid.
2021-01-30 11:38:47 +01:00
Erik Arvstedt 915df059f4 joinmarket: 0.8.0-bcfa7eb -> 0.8.0-a5e8879 2021-01-17 17:40:01 +00:00
Erik Arvstedt 254246cf39 joinmarket: use installPhase
This simplifies the build.
2021-01-17 14:17:14 +01:00
Erik Arvstedt e2922eb4ce move rpc thread count setting to lightning modules 2021-01-14 13:25:12 +01:00
Erik Arvstedt 352fc4e8fe liquid: remove insecure and redundant option 'rpcpassword' 2021-01-14 13:25:11 +01:00
Erik Arvstedt 757a66b9bd liquid: move rpcuser definition to module 2021-01-14 13:25:11 +01:00
Erik Arvstedt 0e00c39d47 secure-node: improve layout 2021-01-14 13:25:11 +01:00
Erik Arvstedt 5f7a7962f7 backups: remove redundant option 'program'
Not needed until we support other backup backends.
2021-01-14 13:25:11 +01:00
Erik Arvstedt 04d8560f86 secure-node: remove qrencode, tor from systemPackages
Keep jq which is useful for analyzing service cli output.
2021-01-14 13:25:10 +01:00
Erik Arvstedt 323a431aba improve nodeinfo
- enable usage outside of secure-node.nix
- use json as the output format
- show ports
- also show local addresses, which is particularly useful when
  netns-isolation is enabled
- only show enabled services
2021-01-14 13:25:10 +01:00
Erik Arvstedt f6b883a9ac remove webindex
This module is outdated and incomplete. We can readd an improved version in
the future.

Move nanopos nginx proxy tests to the nanopos test.
2021-01-14 13:25:10 +01:00
Erik Arvstedt 2a240d6f4a enable-tor: disable default onion services for clightning, lnd, btcpayserver
In case of btcpayserver the default onion service is a security risk
because any visitor can register an admin account on a freshly setup node.
2021-01-14 13:25:09 +01:00
Erik Arvstedt 18c7842e1a modules: show warnings for obsolete options 2021-01-14 13:25:09 +01:00
Erik Arvstedt 45c40c4eb9 versioning: simplify assertion evaluation 2021-01-14 13:25:09 +01:00
Erik Arvstedt bed00fe937 lnd: use onionServices for address announcing 2021-01-14 13:25:09 +01:00
Erik Arvstedt 3980cd5a41 clightning: use onionServices for address announcing 2021-01-14 13:25:08 +01:00
Erik Arvstedt bd2a46cb73 spark-wallet: use onionServices
Also remove the unneeded definition of ReadWritePaths because the
service doesn't need write access to onion files.
2021-01-14 13:25:08 +01:00
Erik Arvstedt 87fb9f246b add 'enable-tor' preset
Move 'enforceTor' and onion-service definitions from secure-node.nix.
Use the onionServices module to define onion services.

Onion services now automatically work for services that bind to an INADDR_ANY (`0.0.0.0`) address.
2021-01-14 13:25:08 +01:00
Erik Arvstedt 05b5402bb1 add nix-bitcoin.onionServices 2021-01-14 13:25:07 +01:00
Erik Arvstedt fffe988248 onionAddresses: add readonly option 'dataDir'
Used by 'onionServices' in a later commit for services that announce
their onion address.
2021-01-14 13:25:07 +01:00
Erik Arvstedt 5f34b094d3 onionAddresses: improve script
- use -e to check for existence of /var/lib/tor/state, use shorter
  polling interval
- clear existing dataDir contents to avoid accumulating obsolete data
- use concatMapStrings instead of foldl'
2021-01-14 13:25:07 +01:00
Erik Arvstedt b266f23251 onionAddresses: use service 'script' option
This also makes the script stop on errors.
2021-01-14 13:25:07 +01:00
Erik Arvstedt 6d13b26d0a onionAddresses: add more precise type for option 'access' 2021-01-14 13:25:06 +01:00
Erik Arvstedt 93562f76dd onionAddresses: remove redundant option 'enable'
The service can be disabled via `onion-addresses.access = mkForce {};`

Also remove redundant description.
2021-01-14 13:25:06 +01:00
Erik Arvstedt 43c247e3fe onionAddresses: use StateDirectory instead of tmpfiles
Simplifies the dataDir setup.
2021-01-14 13:25:06 +01:00
Erik Arvstedt 5c6977b006 rename onion-chef -> nix-bitcoin.onionAddresses
This clarifies its function.
2021-01-14 13:25:05 +01:00
Erik Arvstedt 55073eee70 remove nix-bitcoin.pkgs.lib
Type ipv4Address is not needed anymore because all services have
separate 'port' and 'address' options.
2021-01-14 13:25:05 +01:00
Erik Arvstedt 09e0042aa8 spark-wallet: add consistent address options 2021-01-14 13:25:05 +01:00
Erik Arvstedt 39f16c0b4a liquidd: add consistent address options 2021-01-14 13:25:05 +01:00
Erik Arvstedt b5d76ba1b3 electrs: add consistent address options 2021-01-14 13:25:04 +01:00
Erik Arvstedt 8fa32b7f91 btcpayserver: add consistent address options 2021-01-14 13:25:04 +01:00
Erik Arvstedt e78a609687 clightning: add consistent address options
Also remove option 'autolisten'. This option has no effect because
option 'bind-addr' is always set.
2021-01-14 13:25:04 +01:00
Erik Arvstedt b41a720c28 lnd: add consistent address options
Also fix btcpayserver by connecting to the lnd restAddress instead of the p2p address.
2021-01-14 13:25:03 +01:00
Erik Arvstedt dd4a0238f9 bitcoind: group rpc options under parent option 'rpc' 2021-01-14 13:25:03 +01:00
Erik Arvstedt 5b7e0d09b2 bitcoind: add consistent address options 2021-01-14 13:25:03 +01:00
Erik Arvstedt 2bfb4efbd8 make-container: fix usage comment 2020-12-19 13:18:50 +01:00
Erik Arvstedt 3403795c86 tests: add example scripts 2020-12-18 19:56:56 +01:00
Erik Arvstedt ff94985b8b tests: add test 'hardened' 2020-12-18 19:56:56 +01:00
Erik Arvstedt c8e73c959e fix 'hardened' profile for NixOS 20.09
The 'scudo' memory allocator set by the 'hardened' profile breaks some
services on 20.09.
The fix for NixOS unstable (https://github.com/NixOS/nixpkgs/pull/104052)
is ineffective on 20.09.

As a workaround, add a custom 'hardened' preset that uses the default allocator.
2020-12-18 19:56:56 +01:00
Erik Arvstedt 44b06aea5a extra-container: 0.5-pre -> 0.5 2020-12-18 19:56:56 +01:00
Erik Arvstedt a359cdfb66 generate-secrets: use pwgen
Password length and alphabet is unchanged, but the restriction to
include at least one numeric and one capital char has been removed.
This restriction is not needed by client applications,
adds code complexity, and even (insignificantly) reduces entropy.

Reason for switching to pwgen:
apg uses /dev/random instead of /dev/urandom which brings no security
benefits but can stall the generate-secrets script on low-entropy
devices due to blocking.

Since `security.rngd` has been disabled in NixOS 20.09, blocking
in generate-secrets can also appear on regular NixOS desktop systems.
2020-12-18 19:56:56 +01:00
Erik Arvstedt a5a2fc7274 make-container: fix renamed variable
The variable was only renamed in run-tests.sh, which broke containers.
2020-12-16 01:29:12 +01:00
Erik Arvstedt 9977fa69af ci: use run-tests.sh 2020-12-11 13:27:06 +01:00
Erik Arvstedt a82f0f5f48 add test 'pkgsUnstable'
Included in 'basic' tests.
Function 'doBuild' is needed by the following commit.
2020-12-11 13:27:05 +01:00
Erik Arvstedt 95bc1237e2 run-tests: rename testDir -> scriptDir 2020-12-11 13:27:05 +01:00
Erik Arvstedt 466d23deaa ci: extract build-to-cachix.sh
Needed by the following commits.
Also, don't use the cachix cache as a substituter for local, non-CI
builds. This obviates the need for the 'untrusted' warning in build.sh.
2020-12-11 13:27:05 +01:00
Erik Arvstedt a70c3bf210 make-test-vm: remove unneeded leftover arg attrs 2020-12-11 13:27:05 +01:00
Erik Arvstedt ed65e78a2b make-test: expose test config
This is useful for programmatically exploring a test config or for building a test system
on a custom platform.
2020-12-11 13:27:04 +01:00
Erik Arvstedt 7265742655 run-tests: add 'instantiate' command
Useful for diffing test derivations.
2020-12-11 13:27:04 +01:00
Erik Arvstedt 8cbdef8bf6 run-tests: fix CLI
Restore the original behavior that was accidentally changed:
When no args are given, run the basic test suite.
Otherwise, run the given command with default scenario 'default'.
Previously, `run-tests.sh build` ran the basic test suite instead of
building the default scenario.
2020-12-11 13:27:04 +01:00
Erik Arvstedt 1c0233c0a8 use Cirrus CI
- Make more economic use of the free CI resources by removing redundant build tasks:
  - Build unstable pkgs in a single separate task ("pkgs_unstable").
  - All stable pkgs are implicitly built by the modules tests.
- The build script (ci/build.sh) can now be executed locally for easier
  debugging.
- Use an explicit 'cachix push' command instead of helper/wait-for-network-idle.rb.
  This is simpler and more reliable.
2020-12-06 19:07:54 +01:00
Erik Arvstedt a6346c2561 electrs: 0.8.5 -> 0.8.6 2020-12-01 12:51:36 +01:00
Erik Arvstedt 4640821f96 make-test.nix: use writeText
Needed for the following commit which adds derivation outputs to `dataFile`.
2020-11-18 20:21:34 -06:00
Erik Arvstedt 5399f73b20 add txzmq python pkg 2020-11-18 20:21:34 -06:00
Erik Arvstedt e62e163177 add clightning python pkgs 2020-11-18 20:21:34 -06:00
Erik Arvstedt 1a16e55237 move python packages to pkgs/python-packages
Remove obsolete passthru from joinmarket because joinmarket packages are
now accessible via pkgs/python-packages.
2020-11-18 20:21:34 -06:00
Erik Arvstedt 7e3d2965a5 testing framework: re-add features removed in nixpkgs 20.09
Since nixpks 20.09, the test output is just an empty directory.
Restore saving the log output and linking to the driver.

Without linking to the driver, the driver is eligible for
garbage collection after running a test via `run-tests.sh --out-link-prefix`,
which implies lengthy driver rebuilds.
2020-11-11 22:27:46 +01:00
Erik Arvstedt f1681f5b45 add option nix-bitcoin.pkgs, remove overlay
This works around a nixpkgs bug where overlays are ignored in containers.
2020-11-09 22:10:07 +01:00
Erik Arvstedt b1a5f5e7b9 don't add service-only pkgs to systemPackages
These packages are not useful in shell environemnts.
2020-11-09 22:10:07 +01:00
Erik Arvstedt e39ad358a4 lnd: remove unused bitcoind from PATH 2020-11-09 22:10:07 +01:00
Erik Arvstedt 0972af55f1 netns: fix setup service restarts 2020-11-03 21:55:13 +01:00
Erik Arvstedt 63a464431b netns: fail when netns already exists
Under normal circumstances, service-specific netns should never exist
before the netns setup service starts.
An existing netns is a genuine error that should not be silently ignored.
2020-11-03 21:55:13 +01:00
Erik Arvstedt b4b607dfa5 netns: simplify firewall setup 2020-10-29 22:36:20 +01:00
Erik Arvstedt 25639cec42 netns: fix error msg when starting netns
Previously, the failing initial `netns delete` resulted in a
"Cannot remove namespace file ..." error visible in the journal
and status output.
2020-10-29 21:21:30 +01:00
Erik Arvstedt 67068afd6b netns: fix error when stopping netns
A short time after `netns delete` finishes, the peer link in the main
netns is automatically removed.
When `link del` is run before that, it fails with
`Cannot find device "nb-veth-br-*"` and the netns service enters a failed state.
2020-10-29 21:21:30 +01:00
Erik Arvstedt 4ff88efc50 netns: add address binding test
Proposed by Jonas Nick.
2020-10-29 21:21:30 +01:00
Erik Arvstedt 8da01fe8a6 lightning-loop: allow RPC access from main netns
Note that this also exposes the REST server, which is secured by
macaroon auth like the RPC server.
2020-10-29 21:21:29 +01:00
Erik Arvstedt d76b080b74 lightning-loop: add RPC and REST server options 2020-10-29 21:21:29 +01:00
Erik Arvstedt 9ddf7864a4 lightning-loop regtest: fix incorrectly succeeding test
When 'loop getparams' fails, jq gets no stdin and exits with code 0.
Because -o pipefail is not enabled in the testing shell, the whole test
command succeeds, although it should fail.

Just test "loop getparams" instead and ignore its output.
2020-10-29 21:21:29 +01:00
Erik Arvstedt e66636ef0e liquidd: use type str for rpcbind 2020-10-29 21:21:29 +01:00
Erik Arvstedt de23fdd377 lnd: use type str for rpclisten, restlisten 2020-10-29 21:21:28 +01:00
Erik Arvstedt 8b053326cc bitcoind: use type str for rpcbind
Extra RPC bind addresses can still be added via extraConfig.
2020-10-29 21:21:28 +01:00
Erik Arvstedt 6903e8afcc netns-liquidd: allow RPC access from main netns 2020-10-29 21:21:28 +01:00
Erik Arvstedt 82f4901880 netns-lnd: allow RPC access from main netns 2020-10-29 21:21:27 +01:00
Erik Arvstedt 58d24e735d netns-bitcoind: allow RPC access from main netns 2020-10-29 21:21:27 +01:00
Erik Arvstedt 0e2ff948d3 test: add scenario 'netnsRegtest'
The 'basic' test command now cover regtest mode and using nix-bitcoin without
the secure-node preset.
2020-10-29 21:21:27 +01:00
Erik Arvstedt e0675cb256 move enforceTor logic to service modules
This enables tor support for services without using secure-node.nix
2020-10-29 21:21:27 +01:00
Erik Arvstedt 0cc8caa737 lnd: only set tor.active on enforceTor
This also enables the test scenario 'netnsRegtest' introduced in a
later commit by fixing the following bug:
For unknown reasons, when tor.active=true and tor is not running, lnd
fails with a tor connection error on netns-isolation, but runs fine
without netns-isolation.
2020-10-29 21:21:26 +01:00
Erik Arvstedt 9a931483b9 netns test: remove strict dependency on clightning, electrs
This allows the netns test to be run with a reduced service set for debugging.
2020-10-29 21:21:26 +01:00
Erik Arvstedt bae1b7f413 netns test: improve ping test
- Use fping for pinging multiple hosts in parallel.
  Significantly improves test runtime:
  >13 s -> ~200 ms for the negative ping tests.
- Only test network namespaces that are enabled.
  This allows running the netns test with a reduced service set for debugging.
- Remove deprecated services, instead add btcpayserver, spark-wallet
2020-10-29 21:21:26 +01:00
Erik Arvstedt 2a9b918f72 generate-secrets: always run with Bash, stop on errors 2020-10-23 10:54:15 +02:00
Erik Arvstedt 5e0e16529c netns: fix default addressblock value type
Also remove redundant definition in secure-node.nix
2020-10-20 18:21:37 +02:00
Erik Arvstedt b574cb097f examples: add deploy-container-minimal.sh 2020-10-20 12:35:23 +02:00
Erik Arvstedt 67e49fe415 services: auto-enable dependencies 2020-10-19 14:55:59 +02:00
Erik Arvstedt d2dbad256f shell.nix: fix failing generate-secrets
generate-secrets failed due to the `PYTHONPATH` env var that was set
through the nixops19_09 buildInput.

Fixes:
- Don't pull in the build environments of binaries that should be
  available in PATH. Only extend PATH instead.
- Run generate-secrets in an empty environment
2020-10-19 11:35:32 +02:00
Erik Arvstedt e61d7b1d46 test: improve lightning-loop regtest 2020-10-19 08:59:26 +00:00
Erik Arvstedt e6340426c1 deploy-container.sh: fix sudo
The 'echo sudo' approach used previously failed when PATH or
NIX_PATH contains spaces. Exec the script with sudo instead.
2020-10-18 13:42:58 +02:00
Erik Arvstedt c19f7ebb01 examples: add option --interactive|-i 2020-10-18 13:42:58 +02:00
Erik Arvstedt 33ff8d82be examples: fix running outside of examples dir 2020-10-18 13:42:54 +02:00
Erik Arvstedt 9951f10e74 test: add scenario 'regtest' 2020-10-16 23:55:13 +02:00
Erik Arvstedt d3ece59919 add module 'versioning' 2020-10-16 23:23:00 +02:00
Erik Arvstedt 1f96ca67c5 electrs test: make service shutdown optional
Needed for regtest scenario.
2020-10-16 18:01:52 +02:00
Erik Arvstedt eb42fc8e06 test: extract test 'joinmarket-yieldgenerator'
Needed for regtest scenario.
2020-10-16 18:01:52 +02:00
Erik Arvstedt 06b2ec5b02 joinmarket: add regtest support 2020-10-16 18:01:52 +02:00