Changelog-None

It's still deprecated: we need the description since

1. This information is useful for any validation we want to do, such as
   the HSM, or runes.
2. We want this information in listpays so we can tell what we actually paid.
3. In general, we should never sign commitments to things we don't have!

I expect to have this information about payments *whatever the frontend* is,
which is why we deprecated (and then removed) this unintended use.  The spec
is pretty clear on this:

BOLT ElementsProject#11:
```
A reader:
...
  - MUST check that the SHA2 256-bit hash in the `h` field exactly matches the hashed
  description.
```

However, neither BTCPayServer nor lnbits updated despite the long deprecation
period, so revert 2afe7a1.

Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Rusty Russell <[email protected]>

Added clarification for sums signing, file ownership, and pyln publishing
as well as a reminder to update pyln version for the release.

Changelog-None

This reintroduce lnprototest after 2 releases,
there was a lot of breaking around it and this
will patch them (most of them)!

However, there are some issue related to channel opening and closing
that need some additional love and are disabled for now, but I think it
is good to introduce lnprototest now again in the CI, to be able to
stress the fix for now and see if there are other problem around.

I will take care of it!

Changelog-None
Signed-off-by: Vincenzo Palazzo <[email protected]>

Makes it easier when we remove support for a version.

Signed-off-by: Rusty Russell <[email protected]>

We promised two versions after v0.12, and here we are.

Signed-off-by: Rusty Russell <[email protected]>

…quietly.

Signed-off-by: Rusty Russell <[email protected]>
Changelog-Removed: JSON-RPC: `checkmessage` now always returns an error when the pubkey is not specified and it is unknown in the network graph (deprecated v0.12.0)

Changelog-Removed: JSON-RPC: `listpeers`.`local_msat` and `listpeers`.`remote_msat` (deprecated v0.12.0)
Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Rusty Russell <[email protected]>
Changelog-Removed: JSON-RPC: all the non-msat-named millisatoshi fields deprecated in v0.12.0.

Signed-off-by: Rusty Russell <[email protected]>
Changelog-Removed: JSON-RPC: the "msat" suffix on millisatoshi fields, as deprecated in v0.12.0.

Now there's no compat variant, we can rename this function.

Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Rusty Russell <[email protected]>
Changelog-Removed: JSON-RPC: require the `"jsonrpc": "2.0"` property (requests without this deprecated in v0.10.2).

It seems that bitcoind frequently dies on this test.  I assume running
the multiple nodes under valgrind with the extra 214 blocks is too
memory-hungry?

Signed-off-by: Rusty Russell <[email protected]>

When plugins receive a "shutdown" notification, then can call this
method which will shutdown `cln_plugin`.

Then they can await `plugin.join()` and do any remaining cleanup
there.

This helps avoid a pain-point where plugin authors need to handle
2 separate plugin shutdown mechanisms ElementsProject#6040

Signed-off-by: Vincenzo Palazzo <[email protected]>

While the user trying to fetch an invoice by specifing the quantity we do
not work as expected.

Running the command

```
lightning-cli fetchinvoice -k offer='lno1qgsqvgnwgcg35z6ee2h3yczraddm72xrfua9uve2rlrm9deu7xyfzrcgqffqszsk2p6hycmgv9ek2grpyphxjcm9ypmkjer8v46pyzmhd9jxwet5wvhxxmmdzsqs593pq0ylsvakdua5h976f4g3eautgjt3udvtyga47eaw7339sjrhpwpwz' quantity=2
```

and we answer back with

```json
{
   "code": -32602,
   "message": "quantity parameter required"
}
```

This is caused because we forget to bind the `quanity` field from the
RPC into the `invrequest`.

Reported-by: @aaronbarnardsound
Link: ElementsProject#6089
Signed-off-by: Vincenzo Palazzo <[email protected]>
Changelog-EXPERIMENTAL: fetchinvoice: fix: do not ignore the `quantity` field
into the invreq field.

Changelog-Added: JSON-RPC: `listfunds` now has a `channel_id` field.

This changes connectd to use `status_fail()` on TOR problems during statup
instead of `err()`. Using `err()` did not write to the logfile.

To find out TOR problems during startup, the user needed to stop the system
daemon and call `lightningd` manually in console to see the error.

`status_fail()` logs and exits, but also prints a whole stacktrace,
which is a bit too much imho on config errors. But currently there is
no `status_SOMETHING` method that logs, prints and exists on an error
without stacktrace.

Changelog-None

This seems to be getting stuck in CI, so make sure we time out if it happens.

Signed-off-by: Rusty Russell <[email protected]>

We would sleep for 10msec (default) and try again, spamming the logs
every second.  But we're waiting for revoke_and_ack, and that handler
already sets off the timer, so there's no need to spin at all!

Fixes: ElementsProject#6077
Changelog-Fixed: `channeld`: no longer spin and spam logs when waiting for revoke_and_ack.
Signed-off-by: Rusty Russell <[email protected]>

Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3.
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](pallets/werkzeug@2.2.2...2.2.3)

---
updated-dependencies:
- dependency-name: werkzeug
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>

We were handing 3 to hsmd (and Ken added that in 7b2c561,
so I guess he's OK with that being the minimum supported version!).

Signed-off-by: Rusty Russell <[email protected]>

Importantly, adds the version number at the *front* to help future
parsing.

Signed-off-by: Rusty Russell <[email protected]>


Header from folded patch 'fix-hsm-check-pubkey.patch':

fixup! hsmd: capability addition: ability to check pubkeys.

Signed-off-by: Rusty Russell <[email protected]>

It's needed as the db and wallet is being set up (db migrations), so
it's simpler this way to always use ld->bip32_base for the next patch.

Signed-off-by: Rusty Russell <[email protected]>

At the moment only lightingd needs it, and this avoids missing any
places where we do bip32 derivation.

This uses a hsm capability to mean we're backwards compatible with older
hsmds.

Signed-off-by: Rusty Russell <[email protected]>
Changelog-Added: Protocol: we now always double-check bitcoin addresses are correct (no memory errors!) before issuing them.

`struct lightningd` is not completely initialized, so we added a
"migration_context" which only had some of the fields.  But we ended
up handing in `struct lightningd` anyway, so I don't think this
complexity is worthwhile.

Signed-off-by: Rusty Russell <[email protected]>

If the fuzzer passes an empty data buffer, the fuzz target currently
attempts to read from it. We should short-circuit instead.

The issue is that common_setup() wasn't called by the fuzz target,
leaving secp256k1_ctx as NULL.

UBSan error:

$ UBSAN_OPTIONS="print_stacktrace=1:halt_on_error=1" \
    ./fuzz-channel_id crash-1575b41ef09e62e4c09c165e6dc037a110b113f2

INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1153355603
INFO: Loaded 1 modules   (25915 inline 8-bit counters): 25915 [0x563bae7ac3a8, 0x563bae7b28e3),
INFO: Loaded 1 PC tables (25915 PCs): 25915 [0x563bae7b28e8,0x563bae817c98),
./fuzz-channel_id: Running 1 inputs 1 time(s) each.
Running: crash-1575b41ef09e62e4c09c165e6dc037a110b113f2
bitcoin/pubkey.c:22:33: runtime error: null pointer passed as argument 1, which is declared to never be null
external/libwally-core/src/secp256k1/include/secp256k1.h:373:3: note: nonnull attribute specified here
    #0 0x563bae41e3db in pubkey_from_der bitcoin/pubkey.c:19:7
    ElementsProject#1 0x563bae4205e0 in fromwire_pubkey bitcoin/pubkey.c:111:7
    ElementsProject#2 0x563bae46437c in run tests/fuzz/fuzz-channel_id.c:42:3
    ElementsProject#3 0x563bae2f6016 in LLVMFuzzerTestOneInput tests/fuzz/libfuzz.c:23:2
    ElementsProject#4 0x563bae20a450 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long)
    ElementsProject#5 0x563bae1f4c3f in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long)
    ElementsProject#6 0x563bae1fa6e6 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long))
    ElementsProject#7 0x563bae223052 in main (tests/fuzz/fuzz-channel_id+0x181052) (BuildId: f7f56e14ffc06df54ab732d79ea922e773de1f25)
    ElementsProject#8 0x7fa7fa113082 in __libc_start_main
    ElementsProject#9 0x563bae1efbdd in _start

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior bitcoin/pubkey.c:22:33 in

Which gets libwally upset post-update to 0.8.8

Libwally update breaks compatibility, so
we do this in one large step.

Changelog-Changed: JSON-RPC: elements network PSET now only supports PSETv2.
Changelog-Added: JSON-RPC: PSBTv2 supported for fundchannel_complete, openchannel_update, reserveinputs, sendpsbt, signpsbt, withdraw and unreserveinputs parameter psbt, openchannel_init and openchannel_bump parameter initialpsbt, openchannel_signed parameter signed_psbt and utxopsbt parameter utxopsbt

PSBTv2 support is quite low in the ecosystem, so having a call to convert
log messages and the like should be useful since they'll often be in v2.

Changelog-Added: Added setpsbtversion RPC to aid debugging and compatibility

Also, fix the case where we didn't use --network with EXPOSE_TCP,
as reported by @theborakompanioni:

```
I get Wrong network! Our Bitcoin backend is running on 'regtest', but we expect 'main'. with LIGHTNINGD_NETWORK := regtest when param --network is not provided.
```

Signed-off-by: Rusty Russell <[email protected]>

While we are cleaning up the list forwards with the autoclean plugin we are
not taking into count the forward's payments with the status set to
`local_failed`. In this case, the forwards have no resolved
time because it was not resolved by us due to some local error.

So, this commit is fixing the auto clean plugin by allowing to delete
of the forwards with status set to local_failed by taking into count
the received_time, with the assumption that the received_time, in this case,
is equal to the resolved time (?)

Reported-by: @denis2342
Link: ElementsProject#6058
Signed-off-by: Vincenzo Palazzo <[email protected]>
Changelog-Fixed: plugin: autoclean: considerer the forwards with status
set to `local_failed`.

Fixes ElementsProject#6064
Reported-by: denis2342 <@denis2342>

Changelog-Changed: grpc: The mTLS private keys are no longer group-readable

We should rather hand the annotation through to the user code, and
warn there.

We actually have an assertion that there are no channels remaining when
we delete peers, so this is confusing!

Actually removing the constraint is db-specific and deeply non-trivial.

Signed-off-by: Rusty Russell <[email protected]>

This relaxes the assertion that it won't be used, and renames the
function to be clear.

Signed-off-by: Rusty Russell <[email protected]>

…se channel.

Signed-off-by: Rusty Russell <[email protected]>

We used to have a text version, so this was named 'scid'.  Fix it now.

Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Rusty Russell <[email protected]>

We don't cover three common patterns:
1. Optional integers (db_col_u64 has different form from structs)
2. Optional strings.
3. Optional array fields.

But it does neaten and reduce the scope for cut&paste errors in the
common "if not-NULL, tal and assign".

Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Rusty Russell <[email protected]>

This doesn't restore every bit of information we have, but it does
contain the important ones.

Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Rusty Russell <[email protected]>

i.e. recurse properly in SQL generation.  This is about to happen.

Signed-off-by: Rusty Russell <[email protected]>

We didn't handle the case of an array inside a subobject.  But that
happens when we add the next commit!

Signed-off-by: Rusty Russell <[email protected]>

Changelog-Added: JSON-RPC: `listpeerchannels` now has `channel_type` field.
Signed-off-by: Rusty Russell <[email protected]>

Changelog-Added: JSON-RPC: `listclosedchannels` to show old, dead channels we previously had with peers.
Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Rusty Russell <[email protected]>
Changelog-Added: JSON-RPC: `sql` now includes `listclosedchannels`.

We have 3 personas:

 - Users
 - Developers
 - Maintainers

The first one basically cover the installation documentation. The
latter two are sorted into the "Developer" category, and the reference
category serves as a quick lookup for facts on anything CLN related.

Changelog-None

Signed-off-by: Vincenzo Palazzo <[email protected]>

… won't load.

Signed-off-by: Rusty Russell <[email protected]>

Should do nothing to normal ones, but fix up old invalids ones.

Signed-off-by: Rusty Russell <[email protected]>

In the now-misnamed "last_tx" field.

Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Rusty Russell <[email protected]>

In preparation for the listrunes command.

Changelog-Added: Plugins: `commando-listrunes` command to show issued runes.

Does not yet persist the blacklist.
Changelog-Added: Plugins: `commando-blacklist` command to disable select runes.

Actually check them when we're going to use a rune.

pubkey_from_hexstr() was failing, which we didn't notice because we
weren't checking the return value. The problem was that we were passing
it a strlen that was half the actual length.

Relevant error:

  [libsecp256k1] illegal argument: !secp256k1_fe_is_zero(&ge->x)

  ==417723== ERROR: libFuzzer: deadly signal
    ElementsProject#7 0x7f5deaacc7fb in abort
    ElementsProject#8 0x51b0b0 in secp256k1_default_illegal_callback_fn secp256k1.c
    ElementsProject#9 0x51bd8e in secp256k1_ec_pubkey_serialize
    ElementsProject#10 0x4e235b in pubkey_to_der bitcoin/pubkey.c:29:7
    ElementsProject#11 0x4e2941 in pubkey_cmp bitcoin/pubkey.c:89:2
    ElementsProject#12 0x4e333d in bitcoin_redeem_2of2 bitcoin/script.c:144:6
    ElementsProject#13 0x4f1396 in run tests/fuzz/fuzz-close_tx.c:78:19

…sanitize=undefined

Most importantly, configurator used to use bitshifts on signed
integers which -fsanitize=undefined caught.

But also, tal played fast and loose with typing and aliases, which was
a signficant amount of rework.

Signed-off-by: Rusty Russell <[email protected]>

For example, if we use -fsanitize=undefined, we can't do unaligned
integer access, but since we didn't test with the sanitizer flags, we
didn't know this, and set `HAVE_UNALIGNED_ACCESS=1`.

Also, add -fno-sanitize-recover= in developer mode, so we actually
fail binaries if something is detected.

Signed-off-by: Rusty Russell <[email protected]>

Of course, NULL and length 0 are natural partners, but We Can't Have Nice Things.

Signed-off-by: Rusty Russell <[email protected]>

Stupid, stupid C committee.

Signed-off-by: Rusty Russell <[email protected]>

With the warning that we were trying to put "inf" into a u64, we can
see that this calculation was wrong to use integers!

Signed-off-by: Rusty Russell <[email protected]>

It's defined to be nonull:

```
channeld/channeld.c:2381:2: runtime error: null pointer passed as argument 1, which is declared to never be null
/usr/include/stdlib.h:856:3: note: nonnull attribute specified here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior channeld/channeld.c:2381:2 in 
```

Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Rusty Russell <[email protected]>

This method is no longer used in cln nor in the plugins repo.

Changelog-None

This can be adapted and used to create test gossip stores.
The test is just skipped by design as it would fail on intention.

 - moves offset into GossipHeader hdr which is passed to all constuctors
 - reads .flags as u16 instead of extracting it from the .length, see 0274d88
 - adds zombie and ratelimit flag to GossipHeader
 - bytes_read start at 0 instead of 1 which is more correct,
   the one byte is then corrected for when setting the offset of new header.
 - bytes_read is increased in pull_bytes as this is the only place where
   something is read
 - use new style for various format-strings

Also caches certain __hash__ and __str__ operations,
This way graph operations can be done quicker.