Trusted peer discovery and improved NAT puncturing

[synctext]

Currently all Dispersy communities have their own isolated walker. This is not very efficient. This issue aims to build upon the ongoing multichain work and create a trusted peer discovery mechnism with high-performance NAT puncturing.

Background reading (general):

• foundations of epidemic protocols from 1987
• Gossip tutorial
• Dispersy bundle synchronisation
• UDP NAT and Firewall Puncturing in the Wild
• Tribler simple overview, from 2006

Trusted peer discovery:

• multichain trust mechanism and cooperation currency
• Trust-based collection of information in distributed reputation networks
• A Network Science Perspective of a Distributed Reputation Mechanism

Technical docs:

• http://tribler.readthedocs.io/en/latest/
• http://dispersy.readthedocs.io/en/devel/

[synctext]

@YourDaddyIsHere your thesis issue...

[YourDaddyIsHere]

Well, thanks.

[synctext]

More reading, besides Dispersy.py reading..:

• P2PWNED: Modeling and Evaluating the Resilience of Peer-to-Peer Botnets

[synctext]

As a first warm-up experiment, please implement a Dispersy walker in Python using both Twisted and Protocol Buffers.

To prepare, please also dive into this code and understand the structure. This is a fully functional walker in Java, written by a bachelor student.. See how peer lists are stored and messages are defined there.
Something much more advanced has been build by @qstokkink using Protocol Buffers, see the code repo here.

The goal is to create a as-simple-as-possible Introduction-req and introduction-response parser and generator that can take steps on our live network. For this first prototype it is OK to have infinite memory growth and simplistic non-existent error handling.

Please read the protocol rules for peer discovery carefully and try to simply always walk to a fresh walk-Candidate.

[qstokkink]

* If you want to read about my serializer you should start here to get an impression: https://github.com/qstokkink/TriblerProtobufSerialization/tree/triblermessages#complete-example

[synctext]

This above task might take a bit of time.
A smaller 1-day starting task.. Please create a blocking cmdline tool like contact_bootstrap_peers.py which simply sends out a introduction-request message to our hard-coded bootstrap servers, blocks infinitely until it obtains a response, prints that, then exits.
Usage of a minimal amount of Python lines is appreciated, as we can use this within a deeper and broader tutorial.

[YourDaddyIsHere]

@synctext
Thanks, I have gone through the "fully functional walker in java by bachelor student", I find that they use their own bootstrap server rather than the tracker of Tribler.

Yes, implementing a dispersy walker needs a bit time, so I am now doing the smaller experiment.

[YourDaddyIsHere]

@qstokkink
thanks, that is helpful

[devos50]

@YourDaddyIsHere to make your experiment as self-contained as possible, I would suggest to run your own Dispersy bootstrap server. This is not hard to do and you can use our existing twistd plugin for this: https://github.com/Tribler/dispersy/blob/devel/twisted/plugins/tracker_plugin.py. After you've started this plugin, you can create a bootstraptribler.txt file in your Dispersy working directory where you specify the IP/port of the tracker you just started (see https://github.com/Tribler/dispersy/blob/12c050af3b15dd0b92fa10d52aad78d40379f665/discovery/community.py#L208).

[YourDaddyIsHere]

@devos50
OK

[YourDaddyIsHere]

@synctext Hello, happy new year.
I can now well play with twisted and protocol buffer. But:
What does "take steps on our live network" mean? The live network means the REAL dispersy network? But the real dispersy network uses conversion.py, (rather than protocol buffer) to encode and decode messages (conversion.py uses built-in Struct.pack to decode and encode), if I use protocol buffer to do serialization, it won't be compatible with REAL dispersy network.

So, maybe live network doesn't mean the REAL dispersy network, and I don't need to follow the all the fashion of REAL dispersy network (using conversion.py for serialization etc.) and simply implements a walker follow protocol rules for peer discovery carefully? If so, I almost finish that.

[synctext]

indeed. Quinten already indicated we can't make it binary compatible easily.
Due to a security fix we need to upgrade the tunnel community anyways. We can break compatibility in next release.
Then you can do walks in live network for testing and maturing your code. Any unit tests yet?

[YourDaddyIsHere]

@synctext
not yet.
Do you think I should still stick with protocol buffer (which may not be compatible with dispersy message serialization fashion) or just use conversion.py for serialization?
Since it is just a warm-up, maybe...it is wise to "borrow" something from existing dispersy?

[synctext]

clean implementation with protocol buffers please

[qstokkink]

@YourDaddyIsHere if you want the quick and dirty for sending through Dispersy: you can simply pack the Protocol Buffers serialization into a string and use this as a Payload for a Dispersy message (example where I used this to send JSON here). The tunnel community uses a different method to sign Dispersy messages anyway (which is partially the reason why the Dispersy message definitions are all the same).

That said, I have a PR open which provides dynamic Dispersy messages using Protocol Buffers serialization. I doubt you will need all of Dispersy's fancy message sharing options for your work though.

[synctext]

Please focus on MultiChain community (for trusted peer discovery):

• https://github.com/Tribler/tribler/blob/devel/Tribler/community/multichain/community.py#L29

Docs on tunnel community:

• http://repository.tudelft.nl/islandora/object/uuid%3A98728747-571c-4df8-ac88-f5b5cf19626c
• http://repository.tudelft.nl/islandora/object/uuid%3Ace3bd867-6540-426d-87d0-348bdf78279d
• http://repository.tudelft.nl/islandora/object/uuid%3A997890d1-4141-4597-92eb-3dbaa4dc44a1

[synctext]

Please publish for next meeting:
https://github.com/YourDaddyIsHere/walker_with_clean_design
Talk to real Dispersy tracker and walk on the search community.
Determine response rate.

Rough timeline deadline:

• Jan: Live network
• March: finish NAT testing environment with 25+ boxes in jenkins
• Apr: live with MultiChain
• June: trusted walker goes live

[synctext]

@remkonaber Please sync with this student for your Ansible work.
19 NAT boxes are responsive. power cycle from software via USB.
Server: natlab.ewi.tudelft.nl

[synctext]

More technical reading.. It is difficult to write a single NAT walker for all uses. This describes the requirements for a high-volume node. Especially the hamming/load balancing part is important to understand: http://blog.libtorrent.org/2016/09/dht-bootstrap-node/

ToDo: please understand, read their code, and evaluate the possibility of also implementing: "This function generates a transaction ID based on the destination IP address and a local secret value. This is a way to verify responses without storing state for each ping."

[YourDaddyIsHere]

OK,got it,reading now

[synctext]

Please remove existing Dispersy and Tribler code. Create a minimal walker with just 4 messages. Objective: simplicity, clean, understandable code, correctness, and code coverage.
Preference is just a single .py file with 250 lines of code, single hard-coded IPv4 to bootstrap, hard-coded multi-chain master member.

[synctext]

possible key thesis figures:

• multiple NAT boxes have a walker
• each reports their puncture rate on live network (outgoing / response = success rate)
• calculate average+diff success rate
• integrate in Quality Assurance infrastructure, jenkins.tribler.org
• Screenshot for thesis

[YourDaddyIsHere]

@synctext
This is the clean-walker:https://github.com/YourDaddyIsHere/Clean_Walker there is no dependency on tribler or dispersy codes (I keep using crypto.py of dispersy, but I rewrite part of the crypto.py so it doesn't has dependency on tribler or dispersy anymore)

It can connect with the live network (walk in multichain community). The walker is simple and it can only deal with 4 walker-message (introduction-request, introduction-response, puncture-request, puncture) + 2 identity-message (dispersy-missing-identity, dispersy-identity). For other messages types of multichain community, it doesn't support.

The codes are far more than 250 lines, the serialization/de-serialization of the 6 types of messages take around 500 lines, the walker's logic takes around 250 lines indeed.

For running it, just type "python walker.py". I already tested it with real trackers in live network, all types of messages works. I am hunting bugs on it, want to make it more robust.

For NAT puncturing test, I need to add some statistic modules on it (e.g. calculating puncturing rate for each NAT)

[synctext]

ToDo:

• add units tests ! !
• move all code to Twisted event reactor, explanation from @devos50 with others students.
• read Twisted tutorial

~/walker__clean_code>wc -l *py
  423 crypto.py
   66 Message.py
   40 Mylistener.py
  697 walker.py
  204 WcandidateGroup.py
   50 Wcandidate.py
 1480 total

Please improve your code for our next meeting and make it maintainable.
Ugly duplicate code: https://github.com/YourDaddyIsHere/Clean_Walker/blob/master/walker.py#L70
Ugly hardcoded stuff:

• Ugly definition of EMPTY_VOTE https://github.com/YourDaddyIsHere/Clean_Walker/blob/master/walker.py#L45
• magic without comment, what/why buffer: https://github.com/YourDaddyIsHere/Clean_Walker/blob/master/walker.py#L67
• _struct_4SH seems to be the easier to understand STRUCT_IP_AND_PORT https://github.com/YourDaddyIsHere/Clean_Walker/blob/master/walker.py#L90
• https://github.com/YourDaddyIsHere/Clean_Walker/blob/master/walker.py#L432
• https://github.com/YourDaddyIsHere/Clean_Walker/blob/master/walker.py#L466

Excellent, so next improvement of MultiChain we need Protocol Buffers + identity inclusion option. This last feature adds your full public key to every message you send. This wastes bandwidth, but removes the need for dispersy-identity message. Making messages atomic is also nice.

@egbertbouman You have more code improvement pointers?

[YourDaddyIsHere]

@synctext
Here is the first draft, not yet finished but can still tell the sketch of the story:
thesis-report.pdf

[synctext]

Comments:

• needs more science; like: We focus on advancing the state-of-the-art in a realistic scenario for creating trust. Creating trust has proven to be difficult, especially within distributed systems setting, without a central controlling server or single controlling organisational entity. We focus on the unsolved problem of creating trust when the majority within the system consists of attackers. To make this scenarion still somewhat feasible we assume a reputation function which can estimate the likelihood of an entity being an attacker. Within this thesis we focus on creating trust when honest entities are surrounded by an evil majority. For simplicity we fix our challenging scenarios to 70% attackers and 30% honest entities.
• abstract is too informal
• First line of first thesis chapter: Peer to Peer System is one of the architecture of Distributed System, according to [5], it can be further categorized as ”pure” Peer to Peer or ”hybrid” Peer to Peer. the reader is now confused by the four different concepts you introduce.
• chapter one: provide formal definition of peer discovery.
• hunting videos too informal.
• explain trustchain in detail (as many master thesis works do)
• explain Dispersy (message distribution to groups and persistent storage) and walker (NAT-puncture, peer discovery, network address discovery) in full detail
• all results are critically dependent on the input graph of TrustChain connections. this needs explanation and graphs showing the nature of the input data.
• Use possible a "related work chapter" to explain Tribler, walker, Dispersy,Sybil attack(+picture). Put all explanations in 1 chapter, not spread out across first three.
• Chapter Design of Transitive-Trust Walker. Contains more engineering details of work others did then your own work. Less byte positions, more what, why, how of your transitive trust walker design principles.
• 4.7 Reputation System start with transitive trust / PageRank / PimRank
• steps required to exhaust 95% of peers what is the scientific goal of this experiment?

[synctext]

Epic Quinten walker code in IPv8

[qstokkink]

@synctext that walker is actually already live in TrustChain on devel (see https://github.com/Tribler/tribler/blob/devel/Tribler/community/trustchain/community.py#L289).

Side note: because I wanted the IPv8 mechanism generic and decoupled it is also much more complicated. In fact, I consider the edge based walking to be the most complex code in IPv8.

[YourDaddyIsHere]

@synctext
Ah,that is a epic work.It is fundamental change in Walker strategy.

But the story of my thesis is adding improvement to "current dispersy walker" (the original one without live edge and take fully random walk). Can I still follow my story line by treating the original walker as current walker (it is now a "historical" walker, not a "current walker" any longer because of Quinten's work)? Because if I say the original walker does not exist any more, it will undermine the whole story line of my thesis...

Since all my experiments have been finished and I use the original walker as baseline, and the story line of the thesis report is also adding improvement to original walker, can I still use my current codes and follow my current story line? Because such big change (adding IPV8 and those new stuff to my clean slate walker) in the codes consumes to much time and I need to redo all experiments, which is also time consuming, but I really hurry to graduate, I am running out of budget...

My work now mainly follow the work of Pim Veldhuisen,adding improvement according to its limitation.

My story line is now:

In the experiment: I keep the edge between my walker and the trusted peer (the peer have blocks with us or the peers directly trusted by our trusted peers) alive, preventing NAT hole closed. So, instead of letting trusted nodes time out, we can make them available for longer time.

The teleport home walker also follow the strategy that: visit a neighbor A, A introduce B to me and my walker has a probability to teleport home, otherwise visit B and so on. That is the same with Pim Veldhuisen's simulation.

I also test another worker which take random walker but give the trusted peer higher probability.
I test the two new walker using the original walker (no live edge, take fully random walk) as base line.

The improvement compare with Pim Veldhuisen work is: Pim Veldhuisen give all peers infinite life span. Hence a high reputation peer will always stay in his top 10 peer list hence will cause load balancing issue. And keep a peer alive forever means we can not clean sybils in our peer list using time out. So I give the trusted peer finite life span (but still 10 times longer than normal peer), hence we can make trusted peer available for longer time and clean sybils by time out, and with a finite life span, a high reputation peer will not have global impact in the whole duration of experiment.

And... as you know, the experiments are done and the results are good. I have change the simulated network to 30% honest peers and 70% evil peers, the results are still good. But adding the new features according to Quintens works will cost too much time... I am really running out of budget...

[synctext]

storyline is still: what walker works best in an evil majority environment..

[YourDaddyIsHere]

ok, I will follow the current story line, trying to finish the new version of report before next week

[synctext]

latest thesis report URL: https://github.com/YourDaddyIsHere/MSc-Thesis/blob/master/thesis-report.pdf

[YourDaddyIsHere]

@synctext
That is not the latest... I forget to push the latest one to the repository these days.

I push the latest one a few minutes ago, now it is the latest one:
Thesis Report

I am drawing some graphs to better illustration, will keep update it until the next meeting in 6, Sep

[synctext]

commenting.. thesis-report (3).pdf

• awesome intro. please provide guidance for the reader first. Like "the core topic of this thesis might sounds rather technical and obscure, however, we can easily explain the concept of peer discovery with the following story."
• In this article, the task to find out the address of such nodes is called ”peer discovery”. replace: thesis.
• Chapter 2: "While the current Tribler Walker already has basic functionalities in place, it is not fully satisfying".
  Opening line should be more scientific, like: "Creating trust in the online world has been proven difficult. Wikipedia entries written by "AuthoritativeProfessor" might still contain falsehoods. This thesis work aims to create trust within a very challenging environment -- one without any central authority and without any infrastructure we can rely on".
• Move all Dispersy engineering details out of chapter 2
• 3.1 Storing Historic Behaviors -- move to problem description
• you can explain the problem of Sybil attacks without any engineering details. One possible angle is explaining about fake trump followers
  
• instead of Multichain, please quote our Trustchain Journal article
• possibly call Chapter 3 peer discovery and trust
• please don't call it "focused walking" not understandable for experts. "biased walk".
• design requirement: load balancing (+ remove the clean state remark).
• One easy to read illustration to show 1 real walker 300k honest peers and 700k evil ones
• "Figure 5.2: Sybil Prevention Validation" nobody understands this.
  More like: Experiments with various walker strategies for increasing attack strengths.
• before Figure 5.2 do an easy "peer discovery experiment"
  Conduct 10k walks for the 4 walker settings. 100k attack edges. Show number of steps taken on the X-axis, then plot on the Y-axis the increasing number of unique "peers discovered". Same figure and same experiment, but now with evil vs. honest ratio as Y-axis.
• Section 5.3 and others: explain in first sentence why you do this experiment
• Move section 5.3 as the first experiment: small network discovery (1 figure Fig 5.3 - 5.6)
• time as Y-axis?
• please use the Pim Veldhuizen figure notation for load balancing (a curve, sort node-ID by load)
• creative final experiment: try to attack your walker. Denial of Service or flooding it with numerous incoming walks. Show how damaging such an attack is. Simply be honest if it takes down the walker, "unsolved problem". You know which neighbors a target victim is connected to. Cost of an attack in Euro: 10TByte costs 30 Euro.
  Both crawl-request and intro-request.

[YourDaddyIsHere]

@synctext
latest version report

• all engineering details have been moved to Chapter 3, Chapter 2 now don't have any engineering details. I choose to follow the story told in Chapter 1 to introduce the concept of peer discovery, trust and potential attacks in high level, smoothly move from the story to the formal model of peer discovery strategy.

• I use the term "Multichain" and "TrustChain" interchangeably, because some of the references use the term "Multichain", replace all "Multichain" with "TrustChain" might make the reader confused, so I use the two terms interchangeably, according to the context

• The load balance graphs have been changed to Pim veldhuisen's fashion, y-axis is visit count, x-axis is the node id.

• For the experiment validating the sybil resilence ability, I add a graph where x-axis is time and y-axis is the number of discovered peers, that should give the reader a first impression of what is going on.

• For unsolved problem, I do the experiment: deployed the victim and attacker in two machines with exactly same amount of all resources. The victim is a standard Walker, the attacker is an attacking scripts keep sending introduction-request or crawl-request to the victim. According to the result, using introduction-request as weapon in DDoS will cost the attacker more resources than the victim. But using crawl-request will cost the victim much more resources while cost the attack few resources. (that holds true even when the victim reduce the number of blocks to return to 1)

By the way, because the reserved days for defense are October 23 to October 25, can we figure out the committee member in our next meeting (21,Sep)? There is only 4 weeks to go...

[synctext]

solid progress, good results

[devos50]

@YourDaddyIsHere note that Figure 3.6 in your thesis shows a block graph, used as input for the Temporal Pagerank algorithm, not NetFlow.

[synctext]

Comments on this thesis version:

• Making good progress!
• improve general readability and polish
• the address book should be over 10 kilometers thick (assuming 20 addresses per page and a 1000 page book is 10cm) include a little extra info
• the Ethernet, ARP, DHT, Bittorrent, Tribler explanation is a mere 2 pages is unreadable for non-experts.
• anywhere.But plus _ anyway.Given_ In multiple location the sentence spacing is lost.
• Use already language for experts in Chapter 2. No non-scientific examples needed. Use an example figure/graph with an attacker mode and Alice and Bob. First, Jack is young boy lives in a small village, he is the a participant in the introduction based peer discovery mention above. By using such method, he can in theory find anyone in the world. additionally In our daily life, people already develop defense mechanism to misleading attack, people will not buy a flight ticket to the other side of this planet
• Fake account, Sybil or one of the bot controlled by Trumph’s team (also no need to make it this big)?
• Only Trustchain please we will use the two terms interchangeably according to the context
• In this chapter, we will introduce Dispersy in details as well as the related work for the thesis.
  You also explain Bitcoin and Trustchain.
• The new Walker should not create serious load balancing problems Minor imbalance is OK :-)
• Figure 4.1: Walker Architecture please make more compact, big empty illustration. Use terms like Message parsing? Crawling versus peer discovery versus trust record discovery?
• Figure two steps and teleport home then start a new random path Please clarify, like put in "you" and make the (a) (b) step 4 hops. Then more dramatic reset home.
• mention open source and Github somewhere
• Figure 5.2: Random Walker Peer Discovery. Axis numbers not readable.
• Remove text lines above figures (duplicates caption text).
• Figure 5.5: Random Walker Load Balance please explain/check why a 50% reset walker discovers more then the random walker. Discovers 25-ish less peers.
• Graph of your re-visit again theory !
• 6.3 Unsolved Problem. 5.6 DDoS experimentation and vulnerability
• Be clear: even when a single request only returns a single block it creates a DDoS vulnerability! Mitigation ideas: unique numbers, signed request, proof-of-work puzzel

[YourDaddyIsHere]

@devos50
Oh,thank you,that is a mistake in caption. In previous paragraph I said Figure3.6 is for temporal page rank

[YourDaddyIsHere]

@synctext
We do not schedule a next meeting last time, should we have a meeting before the deadline of handing in the final report?
Since the defense is at 20,October, the deadline of handing in final report is around 13 October. I have time for every day of the following weeks.

[YourDaddyIsHere]

@synctext
Since the defense is in 20, October. The deadline for hand in the report is around 12, October. After that, I will have 1 week for preparing the presentation in defense, could we schedule a meeting at 15~19 October? I need some suggestions on my presentation. Otherwise I won't know whether it is good or crappy...

[YourDaddyIsHere]

The latest version of my thesis is in this repository.

I will update it multiple times every day until the mid night of 12, October (Wednesday this week) @

[YourDaddyIsHere]

@synctext
OK, it is almost final version now, I have the feedback from both committee members now, I have revised the thesis report according to their suggestions, I think we should have a talk tomorrow (11, October) then I have one day left to move to the final version.

[YourDaddyIsHere]

@synctext
Have some changes a few minutes ago.
the current slides:
thesis defense further simplified.pptx

[synctext]

Too many slides for a 30 minute presentation, Max 40min. First 8 slides, remove half. Just present solution 1, "in my thesis I looked at a smarter solution". In general slide 1-25 could be made more in-depth and scientific. Quick detailed comments:

• have an early slide called research question. Define thesis research question in a single sentence.
• slide 15 has too small letters to be readable
• slide 11, first mechanism called "outrun in numbers", not very scientific. second mechanism has no name.
• slide 16, current dispersy peer discovery mechanism (more scientific). Random connect overlay, no bias, no Sybil attack defense.
• slide 18, blockchain introduction; enlarge picture please.
• slide 23, Sybil attack model, [CITATION] SybilGuard paper
• slide 26 Distributed personalised Pagerank, [CITATION] teleport feature parameter Alpha
• slide 43, remove.
• ToAdd: is your research or your software ready for real-world usage?

[YourDaddyIsHere]

@devos50 @qstokkink
The defense is tomorrow (20 October) at 10:00 - 12:00 in the morning.
The room is HB.03.230 katwijkzaal

[qstokkink]

@YourDaddyIsHere I can't make it as I'm on holiday tomorrow, but good luck with your defense!

[synctext]

Final master thesis: Peer Discovery With Transitive Trust in Distributed System

[synctext]

Related work from modelling side using cellular automaton paradigm. Network Automata: Coupling structure and function in real-world networks. Our angle is not network topology, but the connectivity and trust integration. ToDo: Cellular Automata and game theory integration (e.g. Meritrank and AAMAS paper).