How to use TbSync with self signed or otherwise untrusted certificates?

It is indeed possible to use TbSync with self signed certificates. You have to add an exception for your server:

• in Mozilla Thunderbird, go to Tools > Options > Advanced > Certificates.
• click on "View Certificates" and go to tab "Servers".
• click on "Add Exception".
• enter the name of your server and click on "Get Certificate".
• save the exception.

Another option is to import the certificate of your signing authority (CA or root) into Thunderbird. Instead of going to the "Server" tab, go to the "Certification Authority" tab and import the certificate of your signing authority.