WebMemex/memextension

A browser extension that lets you grow your personal web of knowledge.

• 📥 Store pages and 🔍 find them back in their context
• 📝 Create pages to add notes and quotes
• 🔗 Create links to organise your web by your associations
• 📡 Publish your personal web on the world wide web

Because a row of dozens of tabs or bookmarks needs better organisation.

Because our minds organise by assocation, not by folders or filetypes.

Because browsers were never intended to be just viewers.

Let's make web browsers the web weavers they ought to be.

Project status

Sorry to disappoint, but we do not have the promised solution yet. 😒

This project is in full development. You are most welcome to contribute! See Hacking below about how to build and run it from source and teach it new tricks.

Approach

As it stands now, it is a WebExtension (thus should work on most modern browsers), bundled by browserify with some babel ES6–7→ES5 compilation, that logs and stores visited pages in PouchDB, and provides a viewer for this data based on React+Redux. See Code Anatomy below for a full tour du code.

The project strategy is to combine and integrate features from other projects, and to factor out developed functionality into separate modules wherever it seems sensible. This extension could then be regarded as a bunch of different (but related) features a browser ought to have, bundled together for quick installation.

See our initial blog post for the feature roadmap of the next few months.

Contribute

Got feedback, bug fixes, new features, tips? Give a shout. 📢

Pop in on #webmemex on Freenode, send a PR or open an issue on GitHub, or send me (Gerben) a message.

Coding, design, communication, there is plenty to be done. Soon we should even have a tiny bit of funding available to reward contributions. 💸

Hacking

Like playing with ES6, WebExtension browser APIs, React, Redux, PouchDB? Come play along! 🎉

Build and run it

1. Clone this repo.
2. Get Node/NPM.
3. Run make.
4. Load it in Firefox or Chromium/Chrome:
   • In Firefox (≥49): run make fx-run (or run web-ext directly for more control). Alternatively, go to about:debugging, choose 'Load Temporary Add-on', and pick extension/manifest.json from this repo.
   • In Chromium/Chrome: go to Tools→Extensions, enable 'Developer mode', 'Load unpacked extension...', pick the extension/ folder from this repo.
   • Others browsers: let know if you have tested it!

Automatic recompilation

If the steps above worked, running make watch will trigger a quick recompilation every time a source file has been modified.

If you are testing in Firefox through make fx-run/web-ext, the extension should also reload automatically. Otherwise, depending on which part of the code you change, you may have to reload the extension in your browser:

• If your edits affected only the overview interface, just refresh/reopen it.
• However, if you changed the background script, you have will to reload the extension: find it back in the list of temporary add-ons/extensions and click Reload.
• If you changed the 'content_script', it seems browser-dependent whether newly opened pages will get the new version. Better reload the extension to be sure.

Code Anatomy

WebExtension parts

To comply with the anatomy of a WebExtension, it consists of three main parts (found in extension/ after compilation):

• background.js always runs, in an 'empty invisible tab', listening for messages and events.
• content_script.js is loaded into every web page that is visited. It is invisible from that web page's own scripts, and can talk to the background script.
• overview/overview.html, with the resources in that folder, provides the main user interface.

The parts communicate in two ways:

• Messaging through browser.sendMessage, usually done implicitly by using a remote procedure call (util/webextensionRPC.js).
• Through the in-browser PouchDB database, they get to see the same data, and can react to changes made by other parts.

Besides these parts, browser-polyfill.js provides the promise-based browser API, that simply wraps Chromium/Chrome's callback-based chrome API, in order to make the same code run in different browsers (and to structure the callback mess).

Source organisation

To keep things modular, the source code in src/ is not split in exactly those the three parts of the extension, but is rather grouped by functionality. Some folders may end up being factored out into separate repos later on, or at some point perhaps even into separate but interacting browser extensions.

src/activity-logger/: activity logger

This logs every page visit in PouchDB. Soon it should also watch for user interactions, for example to remember which parts of a page you have read.

Currently, for every visit, a new page object is created in the database, to represent the visited page itself. This object should soon be deduplicated when the same page is visited multiple times. After creating a new page object, the next module is triggered to start analysing the page.

src/page-analysis: page analysis

This extracts and stores information about the page in a given tab, such as:

• The plain text of the page, mainly for the full-text search index.
• Metadata, such as its author, publication date, etcetera.
• A screenshot for visual recognition.

src/overview/: overview

The overview is the user interface that opens in a tab of its own. It is built with React and Redux, which create a somewhat complex but nicely organised application structure.

See src/overview/Readme.md for more details.

src/search: document search

Functions for finding relevant knowledge in the user's memory. Currently provides a simple full-text keyword search through visited pages using pouchdb-quick-search. This can be improved in many ways, because we are searching through a person's memory, not just some arbitrary document collection. For example, we can use created assocations and browsing paths to better understand what one is looking for.

src/dev/: development tools

Tools to help during development. They are not used in production builds.

src/util/: utilities

Contains small generic things, stuff that is not project-specific. Things that could perhaps be packaged and published as an NPM module some day.

...: other stuff

The build process is a Makefile, that runs some npm commands specified in package.json, which in turn start the corresponding tasks in gulpfile.babel.js (transpiled by settings in .babelrc). All lurking there so you only have to type make to get things running.

So much for the code tour. 💤 Any questions? ☝️