IPv8 distributed apps / plugins

[synctext]

Goal: Distributed apps with efficient execution model and event-processing.

Possible operation scenario: you download some blockchain "genesis code" to connect to the blockchain. You now can download the basic data and complete the bootstrap. This basic data would somehow be trustworthy and executable. Trust and abuse prevention is the key challenge.

this work fits into our long line of work around this theme.

Plugins and apps have been around for decades. In 1999 I've created one within Winamp. Numerous app ecosystems emerged, in 2008 a vulnerability was published in Facebook Apps.
Dealing with vulnerabilities, spam, and malware has proven to be a hard problem. Tezos aims to have a self-describing blockchain, but they did not yet address any of the fundamental security issues.

In 2008 the Tribler research team reached a key milestone: an app store and execution platform without any server. See our research report on this full self organising system, this work has a distant relation to the Ethereum smart contracts which launched years later. Real 2008 widget code without any servers, no app store and no app platform servers. In 2015 we created self-compiling Android apps ("Autonomous smartphone apps: self-compilation, mutation, and viral spreading").

Widget platform	estimated number of widgets	estimated average size per widget (KB)	estimated total size (MB)
Eclipse Plugins	1,200	900	1,054
Facebook Apps	40,000	–	–
Firefox extensions	3,100	500	1,500
Google Desktop Gadgets	1,250	80	100
Windows Vista Gadgets	5,800	140	780
Yahoo Widgets 5,000	600	2,930

Estimated repository size of various widget platforms in 2008

The 2008 widget source code was simply gossiped around between clients. Each client collected raw .py files and displayed a local app store interface like:

Our 2008 architecture for serverless widgets:

full 2009 thesis

[synctext]

[Copied from ticket #3670, moving activity here; oldest ticket] Master thesis idea:

• permissionless execution of Python code
  • No permission needed
  • Removes all dependency from servers
  • architecture without any single-point-of-failure
  • Open innovation and module growth
  • source code distribution and forking by design
• how to build trust for dAPP developers
  • star ratings for dAPP authors
  • code reviews and endorsements
  • prevent fake vote flooding (e.g. Sybil attack)
  • re-use Trustchain
• IPv8 execution engine
• dAPP discovery and update service
  • display all discovered dAPPs
  • View and click-to-execute untrusted code
• package manager
  • Python code is published as a swarm
  • Source code inside a bittorrent swarm
  • info_hash as a single requirement to identify, download, install, and run code.
• runtime environment
  • event-based intra-package communication bus (local-only)
  • dAPP-to-dAPP overlay, each running app joins the IPv8 overlay with the same app
  • dAPP version updates are gossiped around on this overlay
  • dAPPs: Loading Python modules/plug-ins at runtime

Possible sprints:

• seed swarm with Hello World; import and run
• add IPv8 and execution engine V1
• gossip random list of hashes of discovered code.
• implement client-server protocol of popular Android apps and possible re-use GCM push tokens

[synctext]

Security is a key concern In the first sprints I would recommend focusing on the core features first. Explore security after that. There are sandboxing features like RestrictedPython and chrootbuilder, etc.
A "Sandboxed Python" would let you permit or forbid modules, limit execution slices, permit or deny network traffic, constrain filesystem access to a particular directory (floated as "/"), and so on.

Fun part: IPv8 execution engine is itself obviously an IPv8 distributed app. Can upgrade itself.

[synctext]

security threat observed in the wild: plugins(docker images) which steal cpu cycles for mining: https://arstechnica.com/information-technology/2018/06/backdoored-images-downloaded-5-million-times-finally-removed-from-docker-hub/

[synctext]

Essentially, the public Internet is now a platform of apps on mobile devices

[synctext]

Towards first running code prototype:

• cmdline simplicity
• Libtorrent swarm, filled with execute.py source code file
• download hard-coded swarm
• unpack this swarm
• magical import and execute.
• possibly different executing approaches with different modules/packages/sandboxes.
• Hello World!\n

Goal: self-sustainable ecosystem. Dream outcome: Runs flawlessly and expands after thesis is completed :-)

[synctext]

Status update: experimented with IPv8, learning about related work.

• part-time thesis for now, double head TA.
• Dynamic loading: http://www.diveintopython.net/functional_programming/dynamic_import.html
• IPv8 as service or as a library?
  • 'the network is the code' :-)
  • each dApp has its own overlay (intra-dApp-overlay)
  • dAPP discovery and update service (inter-dApp-overlay)
• Advice: first running code with Libtorrent, dynamic import, then later expand with IPv8 magic
• note: the mechanisms are easy to fix and repair once you find out they are inefficient. You will not get it right the first time. Policies are much harder and difficult to understand. Their is no textbook on how to do this successfully. Key challenge is the governance, audit, and management layer, for instance, CPAN stats: "174,504 Perl modules in 38,832 distributions, written by 13,592 authors, mirrored on 263 servers".
• Early (social) design decision: anonymous code uploading not allowed (security risk)
• @mitchellolsthoorn : the balance between experimenting, exploration, and engineering.
• far future example Facebook-alternative dApp
• Next Meeting: code demo from open Github repo

[synctext]

Let's just go on another dreaming excursion...:

IPv8 dApps are the fundamental building blocks for the re-decentralisation of The Internet. It represents an evolution from monolithic code, REST-based micro-services, towards trustworthy global computing. For over 20 years we have dreamed of composable software. We expand upon the idea of smart contract, by making them scalable and smarter. We never had software repositories which are server-free and maintenance-free. Now we finally have build the first autonomous isolated software pieces which understand the foundations of trust, can made decisions and learn, manage their own money, trade on marketplaces, can self-replicate, can do self-compilation, self-organise online elections, and have long-term reputational memory.
Basic business dApps can be build for electronic signatures, legal document handling, product catalog publication, invoice processing, ordering, inventory finance, currency conversion, bank account management, micro-loans, and other basic banking functions.

Possible sprints to brainstorm about after Libtorrent basics are operational (our relentless incremental improvement methodology :-)

• dApp runtime support and communication paradigm
• dApp event communication
• Prototype for a week with (I'm assuming these are superior to old-skool REST):
  • https://pypi.org/project/amqp/
  • https://github.com/pika/pika
  • https://pika.readthedocs.io/en/0.9.13/modules/adapters/twisted.html
  • Read about Kafka, ActiveMQ, and Kestrel?

Even more ambitious is the offline-first approach, like Bramble, but they lack structural funding. Bramble builds upon Tor and handles all of the core functionality of "sending blocks of data back and forth, managing contacts, keeping channels between users secure and metadata-free, synchronizing state between a user's devices, and handling dependencies between pieces of data or expiring them when they get too old".

[mitchellolsthoorn]

Got the first prototype to work.
https://github.com/mitchellolsthoorn/ipv8-dapps-loader/

A torrent file is created from the 'execute.py' file which contains a hello world code example.
This torrent is seeded on the local network by the 'seeder.py'. The loader downloads this torrent and loads and runs the first dapp of this system

[synctext]

Solid progress!
Relentless incremental improvement, next step: trust, community building and code-reviews?

Brainstorm outcome: we use Github rating of software developers as a metric for quality of a code review. Just for initial bootstrapping. In later iterations we could hide the identity of the reviewers and have randomly picked witnesses validate the reputation of a software developer. Requirements for core functionality:

• a single code review is published in local Trustchain (sign hash of dApp with private key)
• app voting and discovery overlay (MvP: gossip 5 votes for dApps you reviewed)
• Can we safely import a limited-length unicode string from a malicious dApp (a.k.a. "dApp name")?

[synctext]

Brainstorming about the next upcoming sprint after skeleton voting is operational...

#2805 ticket holder @alexander-stannat has now been volunteered to focus on creating a dApp of trust! Goal is to provide visualization and calculation of trust. It would require an API for real-time data feed of Trustchain, GUI events, and user selection of nodes. Upon selecting a node it would display the public key of that node and current trust estimate. Please work together when ready for that.

[synctext]

Related work, "Crev: dependency vetting with a web of trust" (Alpha stage), https://news.ycombinator.com/item?id=18824923

[mitchellolsthoorn]

Progress update:
Voting logic is implemented.

At the moment, nodes vote (+1) on dapps when they receive newly spreaded dapps. The dapps are gossiped to all nodes once. The gossiping happens through voting, where popular or good dapps are gossiped more than others.

I ran a successful experiment where all selected dapps where discovered by the nodes set-up.

Possible plans for next sprint is:

• storing all discovered dapps in a DB
• make the discovered dapps visible
• make the user able to vote on dapps, instead of doing it automatically
• running an experiment with all components across multiple computers
• make an interface for dapps, instead of the current file with code

[synctext]

Meeting minutes:

• execution of .python inside a Bittorrent swarm
• Blockchain-based voting inside our IPv8 overlay
• uses Trustchain in IPv8: trustchain_peer = Peer(ECCrypto().generate_key(u"curve25519"))

    ipv8 = IPv8(configuration)
    trustchain_peer = Peer(ECCrypto().generate_key(u"curve25519"))
    trustchain_community = TrustChainTestnetCommunity(trustchain_peer, ipv8.endpoint, ipv8.network, working_directory='./'+str(i))
    ipv8.overlays.append(trustchain_community)
    ipv8.strategies.append((EdgeWalk(trustchain_community), 10))
    dapp_community = DAppCommunity(trustchain_peer, ipv8.endpoint, ipv8.network, trustchain=trustchain_community)

• cmdline-only sprint
• Is March Public Beta launch still an option? (stand-alone)
• Launch as an Android developers power-shell with code dissemination and trust?
  • runtime-import("Bittorrent-SHA1");
  • discovered-DApps();
  • normal shell commands
  • power-prompt toolkit seems one of many offerings Android?
• GUI solution for interaction?
  • runtime addition of GUI elements, screens, requester prompts and popups.
  • Usable with QT5, Kivy, Java or need for a domain-specific language: DApp-GUI?
  • Showstopper, for instance, need to compile QT5 for iOS when distributed in objective C.
  • Google to the rescue with Flutter
• Future sprints: Dart Hello World from inside swarm
• Downloads all DApps by default upto 10MByte maximum and seed
• Solve availability problem, but only vote if source code is reviewed and approved
• Future sprints: only download and seed if sufficient votes are cast (DDoS)
• Future sprints: multiple trust functions

ImportError and/or path error:

>python loader/runner.py 
Traceback (most recent call last):
  File "loader/runner.py", line 5, in <module>
    from community.dapp.community import DAppCommunity
  File "/home/pouwelse/GITHUB/ipv8-dapps-loader/loader/community/dapp/community.py", line 1, in <module>
    from block import DAppBlock
  File "/home/pouwelse/GITHUB/ipv8-dapps-loader/loader/community/dapp/block.py", line 1, in <module>
    from pyipv8.ipv8.attestation.trustchain.block import TrustChainBlock
ImportError: No module named pyipv8.ipv8.attestation.trustchain.block

[synctext]

Meeting minutes & progress:

• twistified, big refactor
• cmdline operational
• permanent database (no longer simple in-memory)
• next sprint: a simple Hello World and another Trustchain crawler service!
• cross-platform with Android support, possibly Android Webview {default GUI, with IPv8 services in background; dynamic GUI update)

Works!

2019-01-22T12:28:01+0100 [-] 
	        .___  _____                         
	      __| _/ /  _  \  ______  ______  ______
	     / __ | /  /_\  \ \____ \ \____ \/  ___/
	    / /_/  /    |    \   |_> >   |_> >___ \ 
	    \____  \____|__  /    __/ |   __/____  >
	         \/        \/ |__|    |__|       \/ 
	
2019-01-22T12:28:01+0100 [-] version 0.1
2019-01-22T12:28:01+0100 [-] [0] Show dApps
2019-01-22T12:28:01+0100 [-] [1] Create dApp
2019-01-22T12:28:01+0100 [-] [2] Exit
0
2019-01-22T12:28:03+0100 [-] Unhandled Error
	Traceback (most recent call last):
	  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 103, in callWithLogger
	    return callWithContext({"system": lp}, func, *args, **kw)
	  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 86, in callWithContext
	    return context.call({ILogContext: newCtx}, func, *args, **kw)
	  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 122, in callWithContext
	    return self.currentContext().callWithContext(ctx, func, *args, **kw)
	  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 85, in callWithContext
	    return func(*args,**kw)
	--- <exception caught here> ---
	  File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
	    why = selectable.doRead()
	  File "/usr/lib/python2.7/dist-packages/twisted/internet/process.py", line 291, in doRead
	    return fdesc.readFromFD(self.fd, self.dataReceived)
	  File "/usr/lib/python2.7/dist-packages/twisted/internet/fdesc.py", line 94, in readFromFD
	    callback(output)
	  File "/usr/lib/python2.7/dist-packages/twisted/internet/process.py", line 295, in dataReceived
	    self.proc.childDataReceived(self.name, data)
	  File "/usr/lib/python2.7/dist-packages/twisted/internet/_posixstdio.py", line 77, in childDataReceived
	    self.protocol.dataReceived(data)
	  File "/usr/lib/python2.7/dist-packages/twisted/protocols/basic.py", line 571, in dataReceived
	    why = self.lineReceived(line)
	  File "/home/pouwelse/GITHUB/ipv8-dapps-loader/twisted/plugins/cli_plugin.py", line 84, in lineReceived
	    self.menu_items[int(line)].values()[0]()
	  File "/home/pouwelse/GITHUB/ipv8-dapps-loader/twisted/plugins/cli_plugin.py", line 107, in show_dapps
	    msg(self._colorize("info_hash: " + dapp['info_hash'] + " name: " + dapp['name'], 'green'))
	exceptions.TypeError: cannot concatenate 'str' and 'buffer' objects

[mitchellolsthoorn]

Sprint Progress:

Fixed issues discovered during the current and previous sprint

The issues with the CLI interface and the menu were fixed. There were some conversion errors when printing out values from the database, which caused the errors seen on the previous sprint update. The program was also receiving a lot of packet malformatted errors on network packets. This turned out to be caused by an increased block size type that wasn't fixed in the Test TrustChain Network. A PR with an update was created and merged in IPv8 and uploaded to pip.

Database / Verification checks

The database was updated to store more structured data, so the program doesn't have to query the unstructured trustchain every time. This change also allowed for better checking of software guards like if the current dApp is known to the system before executing an action on it, and preventing double votes.

By making this change, there could grow inconsistencies between the trustchain and the database. This could happen through manual tampering of the database or trustchain, or through glitches in the blocks received flow. To prevent this, verification checks are executed 5 seconds after start-up, that detect these inconsistencies and fix them.

Automatic dApp crawling

The previous distribution mechanism of dApp loader only distributed dApps to connected peers when:

• A node created a new dApp
• A node voted on an existing dApp

This strategy could lead to a dead network with passive users and a slow onset time for dApps. To help solve this problem a dApp crawler was created and added, that searches connected peers for unknown dApps, to decrease this onset time. Since the crawler only crawls the first level connected peers, it doesn't impact the scalability significantly.

The crawling task is executed asynchronously 10 seconds after start-up and every hour afterward.

Trust - Double vote detection

To prevent malicious nodes from infecting the network with fake votes to promote certain dApps, a double vote detection check has been added to detect these. Upon detection, a message is shown to the user of the system to indicate this. This double vote detection happens on start-up but could be adjusted to run more often.

Libtorrent integration

In the previous sprint the program consisted out of multiple parts:

• dApp community
• loader
• seeder
• runner

These parts have now all been integrated into the dApp community. This integration led to some problems with the (dynamic) package import, but from preliminary testing, it should all work now.
Context options were added to the menu to reflect these:

Main Menu

• create test dapp (creates a fake test dapp package for testing)
• create dapp (creates a dapp from a dapp package with the package name provided by the user, this package should be located in the payloads directory in the state folder e.g. state_dir/payloads/package_name)

Context Menu (only after discovery)

• download dapp (downloads the dapp through libtorrent)
• run dapp (runs the dapp - first has to be downloaded)

Execution engine implementation

In the development of the execution engine, several problems were encountered:

• Overlays have a lot of common services that could conflict e.g. Trustchain
• These common services degrade the security and isolation of the running dApps e.g. by accessing each others components directly
• Overlays have different types of hierarchical levels e.g. service vs overlay component
• Some overlays use adapted common services e.g. Tracker
• dependencies are difficult to handle

hierarchical types

• Executable
  This was the basic test approach, but could be useful for certain dapps

• IPv8 overlay
  This is the original idea of adding overlays to the same IPv8 instance that dApp loader is using. This, however, causes problems with common services like TrustChain that need to be adapted for certain dapps. Also, port numbers would be more challenging to manage if extra services are started e.g. REST manager. This type is however needed/preferred when running multiple modularised application modules.

• Twisted service
  Twisted services could provide an alternative or secondary approach, where each service would run their own IPv8 instance and thus run on its own port. This would eliminate the common services problem and increase the security, but also the overhead.

Experiment - Executable: Distributing and running of test dApp

Experiment - Overlay: Distributing and running of IPv8 Crawler

Experiment - Service: Distributing and running of IPv8 Tracker

[synctext]

Success! 😄 Operational Trustchain crawler as a dApp. Meeting minutes, focus on either market dApp, trust dApp or implement dynamic GUI. Best focus on the toughest problem, with roadmap consequences: GUI stuff.

Sprint goal: in 2 weeks make prototype of dynamic GUI. prior notes "cross-platform with Android support, possibly Android Webview {default GUI, with IPv8 services in background; dynamic GUI update)". Prior work of QR code scanning and reading in Android. Great basic dApp dependency.. Test for Android: Trustchain crawler GUI.

Create demo .apk (ipv8-service and IPv8 app)

Future sprints: market dApp, Bitcoin wallet, QR-codes + IBANs. (ring-fenced local and global REST api)

2019-02-04T14:26:12+0100 [-] version 0.1
2019-02-04 14:26:12,698:DEBUG:dApp-community: Getting all dApps from catalog
2019-02-04 14:26:12,698:DEBUG:persistence: Getting all dApps from catalog
2019-02-04T14:26:12+0100 [-] 0 dApps found:
2019-02-04T14:26:12+0100 [-] [-1] Return to previous menu
0
2019-02-04T14:26:15+0100 [-] Unhandled Error
	Traceback (most recent call last):
	  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 103, in callWithLogger
	    return callWithContext({"system": lp}, func, *args, **kw)
	  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 86, in callWithContext
	    return context.call({ILogContext: newCtx}, func, *args, **kw)
	  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 122, in callWithContext
	    return self.currentContext().callWithContext(ctx, func, *args, **kw)
	  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 85, in callWithContext
	    return func(*args,**kw)
	--- <exception caught here> ---
	  File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
	    why = selectable.doRead()
	  File "/usr/lib/python2.7/dist-packages/twisted/internet/process.py", line 291, in doRead
	    return fdesc.readFromFD(self.fd, self.dataReceived)
	  File "/usr/lib/python2.7/dist-packages/twisted/internet/fdesc.py", line 94, in readFromFD
	    callback(output)
	  File "/usr/lib/python2.7/dist-packages/twisted/internet/process.py", line 295, in dataReceived
	    self.proc.childDataReceived(self.name, data)
	  File "/usr/lib/python2.7/dist-packages/twisted/internet/_posixstdio.py", line 77, in childDataReceived
	    self.protocol.dataReceived(data)
	  File "/usr/lib/python2.7/dist-packages/twisted/protocols/basic.py", line 571, in dataReceived
	    why = self.lineReceived(line)
	  File "/home/pouwelse/GITHUB/ipv8-dapps-loader/twisted/plugins/cli_plugin.py", line 134, in lineReceived
	    self.dapp_menu_items[int(line)].values()[0](line)
	  File "/home/pouwelse/GITHUB/ipv8-dapps-loader/twisted/plugins/cli_plugin.py", line 224, in download_dapp
	    self.dapp_community.download_dapp(self.context['info_hash'])
	exceptions.KeyError: 'info_hash'

[mitchellolsthoorn]

Sprint Progress:

This sprint I ran into a couple of hurdles that slowed down my progress.

My main focus was on the following tasks:

• Cleaning up some code, to make it easier to work with, less likely to break
• Solve bug in dApp crawler
• Implementing all functionality through a REST API interface
• Working on a website to function as a GUI that could then be used in the Android App interface
• Building a twisted web server
• Cross-compiling libtorrent and libtorrent bindings
• Trying to compile the mp3 hack, incorporating libtorrent and adding all required communities and dependencies
• Injecting this mp3 in the Android app

These experienced hurdles were:

• Looking up a lot of information on how certain things work e.g. python for Android, twisted web server
• Having trouble with the compilation step
• REST cross-origin request protection

Completed tasks:

• Crawler bug
• REST API
• Basic website
• Twisted web server

Task still working on:

• Compiling mp3 with integrated libtorrent
• Solving Cross-origin policy
• Testing if all separate components work as a whole in the Android App

REST API:

Basic Website:

[synctext]

Meeting minutes:

• next sprint goal
  • make it work :-)
  • getting ready for Android "power-shell" release
  • demo examples: runtime addition of GUI elements, screens, requester prompts and popups
• decision point: continue the Android-track or document for thesis.tex and focus remainder on PC/Tribler?
• Future sprints
  • decentral market (orderbook + Bitcoin wallets)
  • decide: minimal trustchain app voting or emphasis on code review
  • Pound to Euro trading as goal?

[synctext]

Todays meeting minutes:

• Solid progress on Android
• Operational dynamic GUI, runtime additions, dynamic code importing
• decision point:
  • document Android plug-in
  • write thesis chapter: "problem description"
  • continue technical work
    • create plugin for trust
    • event processing and basic GUI
    • running code in experimental stage
  • combo of two above issues
• Future Sprint ideas:
  • trust plugin and event processing
  • select from multiple available trust functions
  • trust in code, code reviews and voting
  • Decentral market (orderbook + wallets)
  • Ripple functionality; Pound to Euro trading as goal?
    • transfer money using IBAN with public accountability
    • is the money transferred?
    • prevents lots of scams: "yes I just transferred the money to you"
    • zero-knowledge proof magic?

[synctext]

Storyline input: This thesis presents a framework and implementations of a truly universal binary.

[mitchellolsthoorn]

Sprint Progress

This sprint I focused on:

• Android:
  • Making the Android app a little more robust
  • Show a loading screen while the backend is loading
  • Auto refreshing the dApp interface
  • Shutting the IPV8 service down when the app is closed
• dApp loader:
  • Creating an event processing system that dApps can use to communicate through locally. This allows multiple smaller dApps to function as a bigger one
  • Fiddling with the web imports for the Android client
• Trust plugin:
  • Getting to know how it works
  • Creating a community around it
  • Making a trust dApp
  • Converting trust code from script to proper algorithm (In Progress)
• Thesis:
  • Working on the problem description chapter (In Progress)

Due to computer networking lab preparation, I had less time than I would have wanted. I really wanted to get a good start on writing the thesis, but this didn't work out in the end. I also had a hard time figuring out how to get the actual problem and subject in writing. I think this is an important discussion point, because getting a clearer picture of this would help me in writing the thesis.

Things I would like to work on in the next sprint:

• Thesis
• Conference presentation
• Trust in code

Android app:

[synctext]

Meeting notes:
proposal for problem description. Risk is that it comes across as merely a plug-in system.
dramatic like: "this thesis presents a multi-plaform binary and new paradigm for trustworthy computing. We build on the large body of work around smart contracts and make it more generic, scalable, removed global consensus, and need for oracles." etc. Compare Debian, NodeJS, Winamp plugins, Facebook plugins, and Dapps. Argue that this thesis represents the next step in the continued evolution of computing models.

Next steps:

• thesis latex: finish 1 chapter in draft.
• wrap up the Android multi-platform binary + GUI
• define your Trust API v0.1 for the dApp
• scale! Not the crawler, but the walker (1 double-signed record/ introduction-request).
• load balanced random walker (live edge is future)
• trust as a microservice!

[mitchellolsthoorn]

Progress update

Worked on thesis the last week. Added draft of the introduction and the problem description. Had a hard time trying to separate the different sections: Introduction, Problem description, and related work.

Some personal notes:

• Should these be strictly separated or can they flow over into other sections?
• I am questioning when to introduce certain concepts
• How could I integrate parts of the paper in the thesis
• Should the design chapter follow chronological order or from overview to granular

Progress can be found in:
https://github.com/mitchellolsthoorn/master-thesis

[synctext]

• add 1-3 sentences at the very start which explain the thesis title
• introduction has storytelling structure. Use a more related work focused structure. Provide citations for these bold claims.
• intro: mention + screenshot prior work that social trust can be used; DevID prototype
• Problem Description
  • devise a software re-usability framework
  • validate this with real-world complexity
  • you solve world software problem, no mention of Tribler yet
    • make broader
    • Tribler is merely an example / possible application
  • many people tried, proven to be a hard problem
  • trade-off re-usability and usability
  • we propose a new balance between the two.
  • yet another learning attempt
  • our unique requirements: ToDo into a Related Work table (red and green ticks)
• Design Principles (more scientific high level; not TFTP)
  • trusted execution through openness, crowdsourcing, and voting (e.g. Ethereum VM)
    • NodeJS == social trust
    • Github == social
    • Solidity == ?
  • Event-based architecture
  • Internet-based (live overlay)
  • any-platform (even including Android, GUI framework; thus only common denominator: HTML)
• Implementation and experimentation
  • architecture picture of crawler
  • evaluate event handling system?

[synctext]

Scientific thesis title (less engineering focus):
"Implementation of a collective-intelligence coding framework"

[synctext]

Suggestion for problem description related work:
Some theoretical grounding for coding frameworks exists in the realm of "enterprise software". Evolvability is the cardinal concept in "Normalized Systems Theory", which states that the competitive environment changes rapidly of organizations and therefore they must adapt themselves accordingly in an agile way. This leads to software development tools with versioning in data, APIs, and data processing. This leads to a system with minimal dependencies. We reproduced the following four principles from [NST] here:

– Separation of Concerns, stating that a processing
function can only contain a single concern (i.e., change
driver or each part which can independently change);
– Data version Transparency, stating that a data structure
that is passed through the interface of a processing
function needs to exhibit version transparency (i.e., not
impacting processing functions in case it is updated);
– Action version Transparency, stating that a processing
function that is called by another processing function
needs to exhibit version transparency (i.e., not impacting
its calling processing functions in case it is
updated);
– Separation of States, stating that the calling of a
processing function within another processing function
should exhibit state keeping (i.e., before calling another
processing function, a state should be kept).

plus please discuss Lehman law

[mitchellolsthoorn]

Progress update:

Back from conference in the states.

Remaining days I continued working on writing of the thesis. I still have to resolve points from the previous meeting notes, but I wanted to focus first on getting some more concepts on paper, so it is easier to see the bigger picture and the storyline. Afterward, I will start refactoring to correct parts that could be made more scientific and less storytelling.

I have worked on creating diagrams for different parts of the application, so they can be more easily explained. Especially activity diagrams would help with explaining the process for some of the decentral operations. I have started writing the Design and Experiment/Evaluation chapter and the section about the Android app experiment.

In the coming two weeks I want to focus on finishing the code and writing tests.

[mitchellolsthoorn]

Suggestion new title:

• Balancing re-usability and usability in modern software development.
• Next evolution of modularized code execution
• Balancing code evolution and code development accessibility

[mitchellolsthoorn]

The Lehman law matches the topic very well. I can definitely use that. Thank you.

The other resource you mentioned about the Normalized System Theory seems to describe the development of highly abstracted and engineered software application that allow for agile development, without breaking and with minimal change. I don't know if this is very suitable

[synctext]

please mention: live code updates: https://github.com/julvo/reloading

[mitchellolsthoorn]

Need one more week to finish the final product and experiment. After that focus completely on adding remaining concepts and rewriting and refining the report.

Had trouble finding good metrics for the experiment that won't be very subjective or that depend on the composition of the network. Technical metrics are not suited best for comparison with other platforms.

Design chapter structure/notes:

• Modules search for dependencies downwards
  • View layer triggers search for logic layer
  • Logic layer triggers search for other logic layer
  • After search they register with the higher module
  • Module abstracts classes for registration mechanism
• Module distribution mechanism
  • Send message to all neighbours when voting
  • Send message to random neighbour every x time units to keep modules circulating
  • Random sending based on votes, lifetime, reputation
  • Each nodes downloads module to increase availability in the network
  • After x time units unused/unstored modules get deleted
  • Mention malicious nodes
  • Track of module is used to determine if it can be deleted
  • Find different module versions by module identifier
• Modules
  • Modules should have votes per version to prevent major code changes
  • Code should be viewable
  • Version number is hash code to prevent version tempering
  • Each module has own pub key for identifier
  • Decide to keep author in module identifier
  • Decide if each version should be a separate module, has downsides to upgrade speed
  • Key value store for storing settings in dependent modules
• Execution
  • Docker runner
  • Local runner
• Experiment
  • Distribution spread factor
  • New node Discovery time/factor
  • Module creation speed
  • Speed to distribute through network with instant voting
  • Speed to distribute through network for update
  • Bandwidth usage (hard to measure)
• Questions
  • Why Python? Because dynamic?
  • Why IPv8
  • Is it more usable?
• Motivation
  • Another attempt at holy Grail of software reusability

[synctext]

Quote @mitchellolsthoorn : Normalized System Theory == purist, holy grain and therefore inefficiency beyond usage. Too complex. Without simplicity all agility is lost.

Python does not need to be motivated. All engineering details are merely documented, not meticulously defended. "We use Trustchain as the distributed ledger in our experimental implementation."

IPv8 runtime model versus Ethereum fundamental brokenness (e.g. scalability and cost). The Ethereum model is not long-term sustainable:

Back in October’17, an investor sent 1,700 ETH to a contract (AirSwapDEX) with a gas price
of 400,000 Gwei and gas limit of 592,379. The Tx failed for some odd reason but the
investor was charged whopping 236 ETH ($122,086 as per today’s price)

"Trustchain does not have a native cybercurrency such as Ethereum, it provides a transaction recording ledger". Plus distributed trackers are currently still a central dependency (e.g. 25MByte bootstrap is still ongoing work).

Your thesis work requires a name for ease of referencing. Pasta-Frame: the next evolution of modularized code execution. Key property is permissionless code execution at near-zero cost. The code execution architecture defines the maximum complexity of the code that can be produced. JVM, Nodejs, CPAN perl modules and other real-world framework and the connection between modules determine the Maximum Complexity of Applications (MCA) which a single company, an global consortium or open source community can create. We devised the first architecture to take the MCA as the cardinal design optimisation. Science what defines MCA? What constrains/boosts MCA? The interconnection fabric is the key determinant for the MCA. How data flows between modules, how the future-proofing is arranges, how any piece of code can interconnect with any other code, and how can we devise the universal module interconnector?

Feel free to cite this 2003 work on dynamic configuration and dynamic runtime loading of code modules with a dependency graph with running code:

General remark: please always keep the science first and omit engineering details. Runs!

[synctext]

"Dependency managers now exist for essentially every programming language: Maven Central (Java), NuGet (.NET), Packagist (PHP), PyPI (Python), and RubyGems (Ruby) each host more than 100,000 packages. The arrival of this kind of fine-grained, widespread software reuse is one of the most consequential shifts in software development over the past two decades. And if we're not more careful, it will lead to serious problems." https://queue.acm.org/detail.cfm?id=3344149

[mitchellolsthoorn]

I continued working on the report. Although, the written isn't going as fast as I want. I struggled a lot during the last sprint as I am not making the progress I want to. I have rewritten the structure of the Introduction and moved parts of the structure around. I want to discuss if this new structure is a better fit for the thesis:

Introduction:

• Fancy introduction text to introduce the title and captivate the reader
• Explain what is already done by other people
• Explanation of what happened in the industry over the last couple of years
• Explanation of concepts used in the thesis
• Explain what is missing from the other research and what the thesis tries to solve
• Explain the structure of research/thesis
  Problem description:
• Explain in more details the problems that were left out in the current research and why these are important
• Explain scope of project
  Design:
• Explain the proposed solution to the mentioned problems through the framework

[synctext]

report (25).pdf review of current thesis:

• Vage "How to minimize the risk associated with the use of dependencies?". More concrete 'can we use social trust and crowdsourcing to improve security of libraries'?
• consider adding: "non-revocable publication of libraries"
• "It tries to find a balance between the software practices of Today and the impractical concepts
  of the future.", reformulate the future
• Chapter 2: "This work sets out to create a unified framework for reusable developer modules and user plug-ins built on top of IPv8."; make a clear opening sentence. Tribler comes later chapter: 'We use a concrete use-case driven methodology to advance the cause of software re-usability. '.
• live overlay: or integrated autonomous dissemination. No central entity or central governance, self-organising. No owner as person or organisation
• trustworthy code
  • open ecosystem
  • crowsourcing
  • trust function
• runtime engine
  • dynamic loading
  • integrated autonomous dissemination
  • seamless upgrading

ToDo: animated .GIF of whole trustchain .APK starting, loading, module fetching, trustchain starting, crawling, and browsing ? Use as leading example in thesis experiment? Merge Chapter 5 in other chapter?

[mitchellolsthoorn]

Progress update:

• Rewrote majority of introduction to be more polished
• Changed structure of problem description and added more information

[synctext]

LATEST raw .pdf of master thesis

• Re-write the 2.4. Runtime Engine more as a problem description, instead of a solution direction.
  Like: "A key bottleneck for re-usability and usability is the lack of support within the execution environment. Requirements for our envisioned runtime engine are as follows...". Requirement: "runtime support"?
• "Euthereum", modern spelling checker?
• Chapter "FBASE design"

[mitchellolsthoorn]

Progress update

• Fixed some points in the Problem description
• Rewrote the FBase design chapter
• Filled in the Final thesis document
• Want to have review version coming Friday

PDF

[synctext]

Comments:

• "3.5. Blockchain Storage"; organisational principles?

[synctext]

Comments:

• too informal The previous scenarios have the downside that they make it hard for modules to get of the ground.
• Prior (non-perfect) attempts to explain the design of a protocol
• A simple Bob and Alice story is missing. Either design or implementation dtails. Please list the (bytecode?) format of your votes, protocol message flow or even fancy state machine:-)
• Few hundred LoC? compact: https://github.com/Tribler/py-ipv8/pulls?q=is%3Apr+is%3Aclosed+author%3Amitchellolsthoorn
• Figure 4.3 naming of nice green-on-black screencapture
• please limit to 1 code extract, 1 readme and 1 screenshot
• Two decades of related work on epidemic protocols:
  • http://iptps06.cs.ucsb.edu/papers/Vulgaris-sub06.pdf
  • https://scholar.google.nl/scholar?q=epidemic+gossip+protocols
• 4.10 contains little on the key parameters
  This is represented in the blockchain by only one party signing the
  vote. Storing votes on the blockchain also prevents malicious use of votes to promote ones one module.
• How many seconds between sending out votes which you have collected from the original voter?
• gossip spreading experiment: please use plots
• less ambition for experiments: proof-of-scalability is hard; show that it works and is viable
• go for low hanging fruit, simple experiments: 1 week?
• coverage experiment: how long does it take for 100,10000 or 1 million modules and their votes to spread to 100% of nodes?
• Our 2008 gossip work https://www.semanticscholar.org/paper/Parallel-and-Distributed-Systems-Report-Series-%3A-An-Pouwelse-Yang/423253720d670bba267639473af8697e44f19879
• Our 2013 gossip report: https://pdfs.semanticscholar.org/94e6/26b7ff25b07d23fd345aad254abbdd0a574b.pdf

[synctext]

Remarks on latest version of master thesis.

• consider adding trust to the most prominent position in your thesis; into the title.
  "FBase: trustworthy code module execution"
• Intro, add concrete performance numbers from rant of Electron and other 99% overhead frameworks. Cross platform "run everywhere" is inefficient if you bundle Chrome in each app.
• Optionally add: where does "cross platform" end and code re-usage starts? (overlapping concerns?)
• great opening figure; please minor tweak by removing duplicity (Kernel vs. OS; service 1 vs. microservice 1)
  
• make figure 1.3 bigger (and bit more thick font) when possible
• Spice up the results of all your hard work (now merely 2 pages)
  • Do you have evidence that this is a possible superior approach versus Debian,Ethereum, NodeJS, Github releases (Trustworthy code repo)?
  • make another fancy graph/figure.
  • simple trustworthiness experiment, with 5 stars versus 25 stars?
  • you fully implemented a possible solution to seamless protocol upgrades, community splits and fork hell. See ancient Gnutella2 community warfare; Bitcoin civil crypto war. Moxie claimed closed/federated systems are superior because of this unsolved problem.
    • demonstrate this capability in an experiment
    • show upgrade time
  • plus, for instance, add table with Line of Code breakdown for 5.1
  • Left and right side-by-side screenshots of both Android and PC version GUI
  • Memory breakdown and CPU consumption graphs (constant CPU is fine quantification)
    
    
  • you implemented \subsection{"efficient cross-platform GUI"} 🥇 👾
• ToDo: review by key OSGi contributor of thesis

[synctext]

Discussion:

• asked a lot of graphs in prior meet, no finished and thesis.tex include extra results yet
• difficult to make a pretty graph of module upgrade. Only thing that I can think of is CPU graph plotting an doing figure annotation to mark start of module upgrade and completed module upgrade.
• correct, trust is probably the most interesting angle to done a prominent experiment on.
• idea to combine CPU plots with event bus experiment? e.g. to make it exciting; 1 figure, conduct four experiments with 4 workloads on the event bus (1 msg/second, 10,100,1000)?

[synctext]

Comments on THIS thesis version :

• Significant experiment progress!
• "A visualization of this architecture can be seen in Figure 5.5a", page 2
• Figure 1.3: "Chapter" for even more clarity
• Conclusions confront the requirements with the evaluation; not shown in Figure
• "different categories namely Principles" no caps I believe
• what is the 'superior' principle: self-sovereign or self-governing (Section 2.1)?
• "physical addresses (IP addresses)" 😄
• Our pull requests for these changes can be found on the library’s GitHub repository
• Componenets
• 25 starts
• Trustworthiness Experiment (5 stars vs 25 starts): nobody understands this, without reading the code.
  • specified to the level that it can be reproduced by any other smart master student of TUDelft
  • what is the goal
  • what is the desired outcome
  • Why and how
  • no explanation of reference to the gossip algorithm or Bittorrent seeding
• Figure 5.1 Axis unreadable
• Figure 5.5: cut header and zoom into screen itself; add details: Pentium Xeon Gold i9 6490 running Windows 10 pathlevel 13; Android 9.02 on Samsung S8+ 4G using Exynox 1.5 GHz CPU
• REPEAT: Do you have evidence that this is a possible superior approach versus Debian,Ethereum, NodeJS, Github releases (Trustworthy code repo)?
• REPEAT: idea to combine CPU plots with event bus experiment? e.g. to make it exciting; 1 figure, conduct four experiments with 4 workloads on the event bus (1 msg/second, 10,100,1000)?
  • Steady CPU usage for 100,000 events per second at 8%
  • Combine several runs into a average CPU utilisation graph; drop the time axis; each experiment is a single data point
  • X-axis: workload versus CPU utilisation
  • 100k events, 1M events, 10M events; something
  • Please only draw a line when you have 8 datapoints
• email presentation Friday for early feedback round

[grimadas]

Something to consider for the future:

Python sandboxing and creating a secure subset

https://github.com/InQuest/python-sandboxapi
https://github.com/Cisco-Talos/pyrebox
https://news.ycombinator.com/item?id=8280429
https://github.com/dsagal/pynbox

For IoT: https://micropython.org/

Smart contracts and Python:
https://py-evm.readthedocs.io/en/latest/
https://github.com/koshikraj/neo-ico-template
https://github.com/koshikraj/neo-python-contracts
https://neo-python.readthedocs.io/en/latest/basicusage.html

[synctext]

YouTube export plugin: https://github.com/mps-youtube/pafy

[synctext]

The work pioneered by FBase can be used to build adversarial compatibility for Big Tech.

https://www.eff.org/deeplinks/2020/07/legislative-path-interoperable-internet

We’ll present three specific kinds of interoperability mandate, borrowed from the ACCESS Act’s framing. These
are data portability, back-end interoperability, and delegability. Each one provides a piece of the puzzle: portability
allows users to take their data and move to another platform; back-end interoperability lets users of upstart
competitors interact with users of large platforms; and delegability allows users to interact with content from the
big platforms through an interface of their choosing. All three address different ways that large platforms consolidate
and guard their power.

Plus: https://www.eff.org/deeplinks/2019/10/adversarial-interoperability

[synctext]

From Modules, monoliths, and microservices on design of module systems, like FBase. Top-line goals of module systems are always the same:

• Isolate each bit of code from the other bits.
• Re-connect those bits only where explicitly intended (through a well-defined interface).
• Guarantee that bits you change will still be compatible with the right other bits.
• Upgrade, downgrade, and scale some bits without having to upgrade all the other bits simultaneously.

[synctext]

Solid progress made.
After releasing the "Unstoppable DAO for Web3 Disruption" we have the Internet-deployed fabric for plug-ins (now called smart contracts). With our freedom of computing Android plugin system we can load&run any trustworthy code.

@grimadas found a great solution for run-time.

• For Python: https://github.com/Tribler/trustchain-superapp#freedom-of-computing-app
• For Kotlin: https://developer.algorand.org/tutorials/build-algorand-android-smart-contract-using-kotlin/#10stateless-smart-contract

Adversarial compatibility with WhatsApp comes closer: https://github.com/tgalal/yowsup Python lib for that!

Related work {fails to address trust issues} : POSE: Practical Off-chain Smart Contract Execution. Network and Distributed Systems Security (NDSS) Symposium 2023, [Conference or Workshop Item].

In POSE, each smart contract runs on its own subset of TEEs randomly
selected from all TEEs registered to the network. One of the selected TEEs
is responsible for the execution of a smart contract.