fees going to the moon when trying to pay an invoice

[robtex]

When increasing the allowed fee percentage, the fees increase as well.

$ cli/lightning-cli pay lnbc210n1pdfzmm0pp549h0putsurt0wsx76903h0f7ua3gjjjv3r6pyggm3r3cgzz6926sdr80v3xgg36yfqkgepqvysxxmmdd4jkuapz9s3xjg36ygurgvmyx4skvdedxfsnqv3dxsukgvfdvfnxzdpd8ycnvcm98yeryvr9vcejylgcqzysecu2ntuq7fanzh8w7f5x65ykqettx5kfn6a0krqrwlgyt8ssmp9pyk5t6c3u4vx248j984pyruf2h6usqxk2sdrafm0pdv8suhl7faqpc7j8uk
{ "code" : 206, "message" : "Fee 1001 is 4.766667% of payment 21000; max fee requested is 0.500000%", "data" : { "fee" : 1001, "feepercent" : 4.766667, "msatoshi" : 21000, "maxfeepercent" : 0.500000 } }

$ cli/lightning-cli pay lnbc210n1pdfzmm0pp549h0putsurt0wsx76903h0f7ua3gjjjv3r6pyggm3r3cgzz6926sdr80v3xgg36yfqkgepqvysxxmmdd4jkuapz9s3xjg36ygurgvmyx4skvdedxfsnqv3dxsukgvfdvfnxzdpd8ycnvcm98yeryvr9vcejylgcqzysecu2ntuq7fanzh8w7f5x65ykqettx5kfn6a0krqrwlgyt8ssmp9pyk5t6c3u4vx248j984pyruf2h6usqxk2sdrafm0pdv8suhl7faqpc7j8uk null null null 5
{ "code" : 206, "message" : "Fee 1501 is 7.147619% of payment 21000; max fee requested is 5.000000%", "data" : { "fee" : 1501, "feepercent" : 7.147619, "msatoshi" : 21000, "maxfeepercent" : 5.000000 } }

$ cli/lightning-cli pay lnbc210n1pdfzmm0pp549h0putsurt0wsx76903h0f7ua3gjjjv3r6pyggm3r3cgzz6926sdr80v3xgg36yfqkgepqvysxxmmdd4jkuapz9s3xjg36ygurgvmyx4skvdedxfsnqv3dxsukgvfdvfnxzdpd8ycnvcm98yeryvr9vcejylgcqzysecu2ntuq7fanzh8w7f5x65ykqettx5kfn6a0krqrwlgyt8ssmp9pyk5t6c3u4vx248j984pyruf2h6usqxk2sdrafm0pdv8suhl7faqpc7j8uk null null null 8
{ "code" : 206, "message" : "Fee 2000 is 9.523810% of payment 21000; max fee requested is 8.000000%", "data" : { "fee" : 2000, "feepercent" : 9.523810, "msatoshi" : 21000, "maxfeepercent" : 8.000000 } }

$ cli/lightning-cli pay lnbc210n1pdfzmm0pp549h0putsurt0wsx76903h0f7ua3gjjjv3r6pyggm3r3cgzz6926sdr80v3xgg36yfqkgepqvysxxmmdd4jkuapz9s3xjg36ygurgvmyx4skvdedxfsnqv3dxsukgvfdvfnxzdpd8ycnvcm98yeryvr9vcejylgcqzysecu2ntuq7fanzh8w7f5x65ykqettx5kfn6a0krqrwlgyt8ssmp9pyk5t6c3u4vx248j984pyruf2h6usqxk2sdrafm0pdv8suhl7faqpc7j8uk null null null 10
{ "preimage" : "ad9510bee09e882805a80bf4608f4ff427ac7ff062ccea05d73b290ec029a421", "tries" : 16 }

[ZmnSCPxj]

Given the high tries on the final succeeding payment, it seems that there are a large number of shorter routes that are failing. Since pay will give up immediately as soon as the returned route goes past the indicated maxfeepercent, we do not actually check if the route is viable for payment.

Note that we cannot check if the route is viable without actually executing the payment. So if you requested for example at 0.5% maxfeepercent, and we found a route at 4.7667%, we cannot assure that we actually succeed at that fee: it could be that this route is still nonviable (capacity is not sufficient at some point on the route, or node is offline along the route). Our only alternative is to attempt the payment immediately, but then if it succeeds we have used a route beyond the indicated maxfeepercent.

[robtex]

Thanks. Kind of makes perfect sense, even though i thought the fee sounded a bit high. still cheaper than on-chain :)
Can i debug which path it it actually takes on pay? other than getroute (which gives me a nice path with low fees.)

[ZmnSCPxj]

Well the payment being made is 21 satoshi, which is very low, so fees become much higher in proportion; more typical payments in the millibitcoin and microbitcoin range will have the fee negligible in proportion to the payment.

The route should be visible in the logs at the debug level, but do note that it takes quite a bit of time between routing attempts when it is actually performing the payment. If there is much other activity on your node you will have to dig through the logs. I should probably report more details during payment...

[fixone]

can confirm that I've also encountered this behavior, regardless of what fee I put in, the error says that it's less than the minimum required, e.g.

lnc pay -k bolt11=lntb1u1pdfyutepp5a37urzu62fj6wqrfg33cnuh53t70drga53wusuay2ntu8sycryksdyu0v3xgg36yffx2ctyypqhyarfvdkx2w3q23yy7428fp29xgz0fcsycj28fp2yuj2wgusyznjyyp2ys3fqf3yjytpzdy3r5gnx8p3xzvty8psj6vp5vvez6dpsx3nz6cnpvvmj6vejxa3xzenzv3jr2wf5yf7scqzysm3y5jvndfvl8wt9ytqgwj8s26pz6p93y476xqwg4rlpv4lrkha5jhkajnv32a8suvxzdqxev575ka9w9xxjgged3f0hs4qxuh07r6lgqjuk3hn maxfeepercent=1
{ "code" : 206, "message" : "Fee 1002 is 1.002000% of payment 100000; max fee requested is 1.000000%", "data" : { "fee" : 1002, "feepercent" : 1.002000, "msatoshi" : 100000, "maxfeepercent" : 1.000000 } }

lnc pay -k bolt11=lntb1u1pdfyutepp5a37urzu62fj6wqrfg33cnuh53t70drga53wusuay2ntu8sycryksdyu0v3xgg36yffx2ctyypqhyarfvdkx2w3q23yy7428fp29xgz0fcsycj28fp2yuj2wgusyznjyyp2ys3fqf3yjytpzdy3r5gnx8p3xzvty8psj6vp5vvez6dpsx3nz6cnpvvmj6vejxa3xzenzv3jr2wf5yf7scqzysm3y5jvndfvl8wt9ytqgwj8s26pz6p93y476xqwg4rlpv4lrkha5jhkajnv32a8suvxzdqxev575ka9w9xxjgged3f0hs4qxuh07r6lgqjuk3hn maxfeepercent=2
{ "code" : 206, "message" : "Fee 2002 is 2.002000% of payment 100000; max fee requested is 2.000000%", "data" : { "fee" : 2002, "feepercent" : 2.002000, "msatoshi" : 100000, "maxfeepercent" : 2.000000 } }

lnc pay -k bolt11=lntb1u1pdfyutepp5a37urzu62fj6wqrfg33cnuh53t70drga53wusuay2ntu8sycryksdyu0v3xgg36yffx2ctyypqhyarfvdkx2w3q23yy7428fp29xgz0fcsycj28fp2yuj2wgusyznjyyp2ys3fqf3yjytpzdy3r5gnx8p3xzvty8psj6vp5vvez6dpsx3nz6cnpvvmj6vejxa3xzenzv3jr2wf5yf7scqzysm3y5jvndfvl8wt9ytqgwj8s26pz6p93y476xqwg4rlpv4lrkha5jhkajnv32a8suvxzdqxev575ka9w9xxjgged3f0hs4qxuh07r6lgqjuk3hn maxfeepercent=3
{ "code" : 206, "message" : "Fee 10012 is 10.012000% of payment 100000; max fee requested is 3.000000%", "data" : { "fee" : 10012, "feepercent" : 10.012000, "msatoshi" : 100000, "maxfeepercent" : 3.000000 } }

lnc pay -k bolt11=lntb1u1pdfyutepp5a37urzu62fj6wqrfg33cnuh53t70drga53wusuay2ntu8sycryksdyu0v3xgg36yffx2ctyypqhyarfvdkx2w3q23yy7428fp29xgz0fcsycj28fp2yuj2wgusyznjyyp2ys3fqf3yjytpzdy3r5gnx8p3xzvty8psj6vp5vvez6dpsx3nz6cnpvvmj6vejxa3xzenzv3jr2wf5yf7scqzysm3y5jvndfvl8wt9ytqgwj8s26pz6p93y476xqwg4rlpv4lrkha5jhkajnv32a8suvxzdqxev575ka9w9xxjgged3f0hs4qxuh07r6lgqjuk3hn maxfeepercent=11
{ "code" : 206, "message" : "Fee 11010 is 11.010000% of payment 100000; max fee requested is 11.000000%", "data" : { "fee" : 11010, "feepercent" : 11.010000, "msatoshi" : 100000, "maxfeepercent" : 11.000000 } }

until finally

lnc pay -k bolt11=lntb1u1pdfyutepp5a37urzu62fj6wqrfg33cnuh53t70drga53wusuay2ntu8sycryksdyu0v3xgg36yffx2ctyypqhyarfvdkx2w3q23yy7428fp29xgz0fcsycj28fp2yuj2wgusyznjyyp2ys3fqf3yjytpzdy3r5gnx8p3xzvty8psj6vp5vvez6dpsx3nz6cnpvvmj6vejxa3xzenzv3jr2wf5yf7scqzysm3y5jvndfvl8wt9ytqgwj8s26pz6p93y476xqwg4rlpv4lrkha5jhkajnv32a8suvxzdqxev575ka9w9xxjgged3f0hs4qxuh07r6lgqjuk3hn maxfeepercent=12
{ "code" : 205, "message" : "Could not find a route" }

[ZmnSCPxj]

Yes, this can indeed happen: there are no routes that are viable (i.e. all routes to the destination are depleted in the direction you are going on). We cannot know this fact until we try all routes, and we cannot really try a route without potentially paying it, and if the feerate is higher than the user-given max feerate then that is probably worse behavior (there is no point in specifying a user-given max feerate if we do not respect it anyway).

I think I shall mark this as wontfix.

[robtex]

I agree it is awontfix since I can't see it can be fixed (other than perhaps trying via AMP, but that is a different issue.)
I would however appreciate more detailed output on both fail and success, but that might be more a separate feature request.

[ZmnSCPxj]

#1100 improves log output and is likely to be merged in "soon" for some value of soon.

#1088, I propose to provide "low-level" commands that make pay asynchronous, to provide feedback as it happens.