Skip to content

Latest commit

 

History

History
615 lines (497 loc) · 25.2 KB

README.md

File metadata and controls

615 lines (497 loc) · 25.2 KB
Raw

syncall

Coverage Status pre-commit PyPI version Downloads Code style: black

Description

syncall is your one-stop software to bi-directionally synchronize and keep in sync the data from a variety of services. The framework is targeted towards, but not limited to, the synchronization of note-taking and task management data. Each synchronization comes with its own executable which handles the synchronization services/sides at hand.

One of the main goals of syncall is to be extendable. Thus it should be easy to introduce support for either a new service / synchronization side (e.g., ClickUp) or a new synchronization altogether (e.g., ClickUp <-> Google Keep) given that you implement the corresponding synchronization sides and conversion methods. See also the CONTRIBUTING guide to get started.

At the moment the list of supported synchronizations is the following:

Description Executable
README Taskwarrior ⬄ Google Tasks tw-gtasks-sync
README Taskwarrior ⬄ Google Calendar tw-gcal-sync
README Taskwarrior ⬄ Notion Checkboxes tw-notion-sync
README Taskwarrior ⬄ Google Keep Checkboxes tw-gkeep-sync
README Taskwarrior ⬄ Asana Tasks tw-asana-sync
README Taskwarrior ⬄ Generic Caldav server tw-caldav-sync
README Local Files ⬄ Google Keep Notes fs-gkeep-sync

Each of the above should support bi-directional synchronization between the said services. This means that on an addition, modification, or deletion of an item on one side, a corresponding addition, modification or deletion of the counterpart item will occur on the other side so that the two sides are eventually in sync. All synchronizations also support conflict resolution meaning that it can successfully deal with item edits on both sides.

Currently unless the executable at hand specifies otherwise, the following conflict resolution strategies are available:

  1. MostRecentRS
  2. LeastRecentRS
  3. AlwaysFirstRS
  4. AlwaysSecondRS

By the end of the run, it should show you a summary of what's been done, like the following.

Google Keep
-----------
        * Items created: 3
        * Items updated: 2
        * Items deleted: 1

Taskwarrior
-----------
        * Items created: 1
        * Items updated: 0
        * Items deleted: 0

Refer to the corresponding README from the list above for instructions specific to the synchronization with that particular service. Before jumping to that though, please complete the installation instructions below.

Installation instructions

Requirements

  • Python version >= 3.8
  • For the integrations that require Taskwarrior - (Installation instructions) version >=2.6 is required.

Installation Options

You have to specify at least one extra. To do so use the [] syntax in pip:

# for installing integration with google (e.g. Google Keep / Calendar) and Notion
pip3 install syncall[notion,google]

Here's some of the available options for installing it:

  • From PyPI - e.g., Specify three extras for integrations - Google (Calendar/Tasks), Google Keep, Notion):

    pip3 install --user --upgrade syncall[notion,google,gkeep]

  • From Github - e.g., Specify two extras:

    pip3 install --user "syncall[gkeep,fs] @ git+https://github.com/bergercookie/syncall"

  • Download and install devel branch locally - bleeding edge

    git clone https://github.com/bergercookie/syncall
cd syncall
git checkout devel
pip3 install --user --upgrade .[gkeep,fs,google,tw,caldav,asana]

  • Setup using poetry - handy for local development and for isolation of dependencies:

    git clone https://github.com/bergercookie/syncall
poetry install --all-extras
# get an interactive shell
poetry shell

# now the executables of all the services should be in your PATH for the
# current shell and you can also edit the source code without further
# re-installation ...

Sample Usage Instructions

Here's the CLI help page for the synchronizations available.

tw_gtasks_sync --help 
Usage: tw_gtasks_sync [OPTIONS]

  Synchronize lists from your Google Tasks with filters from Taskwarrior.

  The list of TW tasks is determined by a combination of TW tags and a TW
  project while the list in GTasks should be provided by their name. if it
  doesn't exist it will be crated

Options:
  -l, --gtasks-list TEXT          Name of the Google Tasks list to synchronize
                                  (will be created if not there)
  --google-secret FILE            Override the client secret used for the
                                  communication with the Google APIs
  --oauth-port INTEGER            Port to use for OAuth Authentication with
                                  Google Applications
  -t, --taskwarrior-tags TEXT     Taskwarrior tags to synchronize
  -p, --tw-project TEXT           Taskwarrior project to synchronize
  --list-combinations             List the available named TW<->Google Tasks
                                  combinations
  --list-resolution-strategies    List all the available resolution strategies
                                  and exit
  -r, --resolution-strategy [MostRecentRS|LeastRecentRS|AlwaysFirstRS|AlwaysSecondRS]
                                  Resolution strategy to use during conflicts
  -b, --combination TEXT          Name of an already saved TW<->Google Tasks
                                  combination
  -s, --save-as TEXT              Save the given TW<->Google Tasks filters
                                  combination using a specified custom name.
  --prefer-scheduled-date         Prefer using the "scheduled" date field
                                  instead of the "due" date if the former is
                                  available
  -v, --verbose
  --version                       Show the version and exit.
  --help                          Show this message and exit.
tw_gcal_sync --help 
Usage: tw_gcal_sync [OPTIONS]

  Synchronize calendars from your Google Calendar with filters from
  Taskwarrior.

  The list of TW tasks is determined by a combination of TW tags and a TW
  project while the calendar in GCal should be provided by their name. if it
  doesn't exist it will be crated

Options:
  -c, --gcal-calendar TEXT        Name of the Google Calendar to synchronize
                                  (will be created if not there)
  --google-secret FILE            Override the client secret used for the
                                  communication with the Google APIs
  --oauth-port INTEGER            Port to use for OAuth Authentication with
                                  Google Applications
  -t, --taskwarrior-tags TEXT     Taskwarrior tags to synchronize
  -p, --tw-project TEXT           Taskwarrior project to synchronize
  --list-combinations             List the available named TW<->Google
                                  Calendar combinations
  --list-resolution-strategies    List all the available resolution strategies
                                  and exit
  -r, --resolution-strategy [MostRecentRS|LeastRecentRS|AlwaysFirstRS|AlwaysSecondRS]
                                  Resolution strategy to use during conflicts
  -b, --combination TEXT          Name of an already saved TW<->Google
                                  Calendar combination
  -s, --save-as TEXT              Save the given TW<->Google Calendar filters
                                  combination using a specified custom name.
  --prefer-scheduled-date         Prefer using the "scheduled" date field
                                  instead of the "due" date if the former is
                                  available
  --default-event-duration-mins INTEGER
                                  The default duration of an event that is to
                                  be created on Google Calendar [in minutes]
  -v, --verbose
  --version                       Show the version and exit.
  --help                          Show this message and exit.
tw_notion_sync --help 
Usage: tw_notion_sync [OPTIONS]

  Synchronise filters of TW tasks with the to_do items of Notion pages

  The list of TW tasks is determined by a combination of TW tags and TW
  project while the notion pages should be provided by their URLs.

Options:
  -n, --notion-page TEXT          Page ID of the Notion page to synchronize
  --token, --token-pass-path TEXT
                                  Path in the UNIX password manager to fetch
  -t, --taskwarrior-tags TEXT     Taskwarrior tags to synchronize
  -p, --tw-project TEXT           Taskwarrior project to synchronize
  -r, --resolution-strategy [MostRecentRS|LeastRecentRS|AlwaysFirstRS|AlwaysSecondRS]
                                  Resolution strategy to use during conflicts
  -b, --combination TEXT          Name of an already saved TW<->Notion
                                  combination
  --list-combinations             List the available named TW<->Notion
                                  combinations
  -s, --save-as TEXT              Save the given TW<->Notion filters
                                  combination using a specified custom name.
  -v, --verbose
  --version                       Show the version and exit.
  --help                          Show this message and exit.
tw_gkeep_sync --help 
Usage: tw_gkeep_sync [OPTIONS]

  Synchronize Notes from your Google Keep with filters from Taskwarrior.

  The list of TW tasks is determined by a combination of TW tags and a TW
  project while the note in GKeep should be specified using their full name.
  if it doesn't exist it will be created.

  This service will create TaskWarrior tasks with the specified filter for
  each one of the checkboxed items in the specified Google Keep note and will
  create Google Keep items for each one of the tasks in the Taskwarrior
  filter. You have to first "Show checkboxes" in the Google Keep Note in order
  to use it with this service.

Options:
  -k, --gkeep-note TEXT           Full title of the Google Keep Note to
                                  synchronize - Make sure you enable the
                                  checkboxes
  --user, --user-pass-path TEXT   Path in the UNIX password manager to fetch
                                  the Google username from
  --passwd, --passwd-pass-path TEXT
                                  Path in the UNIX password manager to fetch
                                  the Google password from
  --token, --token-pass-path TEXT
                                  Path in the UNIX password manager to fetch
                                  the google keep token from
  -t, --taskwarrior-tags TEXT     Taskwarrior tags to synchronize
  -p, --tw-project TEXT           Taskwarrior project to synchronize
  --list-combinations             List the available named TW<->Google Keep
                                  combinations
  -r, --resolution-strategy [MostRecentRS|LeastRecentRS|AlwaysFirstRS|AlwaysSecondRS]
                                  Resolution strategy to use during conflicts
  -b, --combination TEXT          Name of an already saved TW<->Google Keep
                                  combination
  -s, --save-as TEXT              Save the given TW<->Google Keep filters
                                  combination using a specified custom name.
  -v, --verbose
  --version                       Show the version and exit.
  --help                          Show this message and exit.
tw_asana_sync --help 
Usage: tw_asana_sync [OPTIONS]

Options:
  --token, --token-pass-path TEXT
                                  Path in the UNIX password manager to fetch
  -w, --asana-workspace-gid TEXT  Asana workspace GID used to filter tasks
  -W, --asana-workspace-name TEXT
                                  Asana workspace name used to filter tasks
  --list-asana-workspaces         List the available Asana workspaces
  -t, --taskwarrior-tags TEXT     Taskwarrior tags to synchronize
  -p, --tw-project TEXT           Taskwarrior project to synchronize
  -r, --resolution-strategy [MostRecentRS|LeastRecentRS|AlwaysFirstRS|AlwaysSecondRS]
                                  Resolution strategy to use during conflicts
  -b, --combination TEXT          Name of an already saved TW<->Asana
                                  combination
  --list-combinations             List the available named TW<->Asana
                                  combinations
  -s, --save-as TEXT              Save the given TW<->Asana filters
                                  combination using a specified custom name.
  -v, --verbose
  --version                       Show the version and exit.
  --help                          Show this message and exit.
tw_caldav_sync --help 
Usage: tw_caldav_sync [OPTIONS]

  Synchronize lists of tasks from your caldav Calendar with filters from
  Taskwarrior.

  The list of TW tasks is determined by a combination of TW tags and a TW
  project. Use `--all` to synchronize all tasks.

  The calendar in Caldav should be provided by their name. If it doesn't exist
  it will be created.

Options:
  --caldav-calendar TEXT          Name of the caldav Calendar to sync (will be
                                  created if not there)
  --caldav-url TEXT               URL where the caldav calendar is hosted at
                                  (including /dav if applicable)
  --caldav-user TEXT              The caldav username for the given caldav
                                  instance
  --caldav-passwd, --caldav-passwd-pass-path TEXT
                                  Path in the UNIX password manager to fetch
                                  the caldav password from
  --caldav-passwd-cmd CMD
                                  A command (which is run in a shell) that
                                  outputs the caldav password on stdout
  --all, --taskwarrior-all-tasks  Sync all taskwarrior tasks [potentially very
                                  slow]
  -t, --taskwarrior-tags TEXT     Taskwarrior tags to synchronize
  -p, --tw-project TEXT           Taskwarrior project to synchronize
  --30-days, --only-modified-last-30-days
                                  Only synchronize Taskwarrior tasks that have
                                  been modified in the last 30 days
  --list-combinations             List the available named TW<->Caldav
                                  combinations
  -r, --resolution-strategy [MostRecentRS|LeastRecentRS|AlwaysFirstRS|AlwaysSecondRS]
                                  Resolution strategy to use during conflicts
  -b, --combination TEXT          Name of an already saved TW<->Caldav
                                  combination
  -s, --save-as TEXT              Save the given TW<->Caldav filters
                                  combination using a specified custom name.
  -v, --verbose
  --version                       Show the version and exit.
  --help                          Show this message and exit.
fs_gkeep_sync --help 
Usage: fs_gkeep_sync [OPTIONS]

  Synchronize Notes from your Google Keep with text files in a directory on
  your filesystem.

  You can only synchronize a subset of your Google Keep notes based on a set
  of provided labels and you can specify where to create the files by
  specifying the path to a local directory. If you don't specify Google Keep
  Labels it will synchronize all your Google Keep notes.

  For each Google Keep Note, fs_gkeep_sync will create a corresponding file
  under the specified root directory with a matching name. Any addition,
  deletion and modification of the files on the filesystem will result in the
  corresponding addition, deletion and modification of the corresponding
  Google Keep item. The same holds the other way around.

Options:
  -k, --gkeep-labels TEXT         Google Keep labels whose notes to
                                  synchronize
  -i, --gkeep-ignore-labels TEXT  Google Keep labels whose notes will be
                                  explicitly ignored
  --user, --user-pass-path TEXT   Path in the UNIX password manager to fetch
                                  the Google username from
  --passwd, --passwd-pass-path TEXT
                                  Path in the UNIX password manager to fetch
                                  the Google password from
  --token, --token-pass-path TEXT
                                  Path in the UNIX password manager to fetch
                                  the google keep token from
  --ext, --filename-extension TEXT
                                  Use this extension for locally created files
  --fs, --fs-root TEXT            Directory to consider as root for
                                  synchronization operations
  --list-combinations             List the available named Filesystem<->Google
                                  Keep combinations
  -r, --resolution-strategy [MostRecentRS|LeastRecentRS|AlwaysFirstRS|AlwaysSecondRS]
                                  Resolution strategy to use during conflicts
  -b, --combination TEXT          Name of an already saved Filesystem<->Google
                                  Keep combination
  -s, --save-as TEXT              Save the given Filesystem<->Google Keep
                                  filters combination using a specified custom
                                  name.
  -v, --verbose
  --version                       Show the version and exit.
  --help                          Show this message and exit.

Mechanics / Automatic synchronization

To achieve synchronization between two arbitrary services, we use a push-pull mechanism which is far easier and less troublesome than an automatic synchronization solution. This means that you have to explicitly call the executable for the synchronization you want to achieve. If you want to automate this, consider adding a cron job or a systemd timer.

crontab -e
...

# Add the following to sync every 10' - modify the arguments according to your
# preferences and according to the instructions of the corresponding executable
# for example for `tw_gcal_sync`:
#
# See output and potential errors in your system logs (e.g., `/var/log/syslog`)
*/10 * * * * tw_gcal_sync -c "TW Reminders" -t "remindme"

FAQ

How do I reset the synchronization and start it from scratch?

  1. Remove the combination file that corresponds to your synchronization. For example, if you're executing synchronization of Google Keep with Taskwarrior, like the following, your combination name is testnote__None__test_tag.

    tw_gkeep_sync -t test_tag -k "testnote"

    The executable also mentions the combination name at the end of the run.

    ...
14:00:03.41 | INFO      | Flushing data to remote Google Keep...
14:00:04.32 | SUCCESS   | Sync completed successfully. You can now use the -b/--combination option to refer to this particular combination

  tw_gkeep_sync --combination testnote__None__test_tag

    For this combination, on Linux, remove ~/.config/syncall/testnote__None__test_tag.yaml.

  2. Remove the section for your combination in the <sideA_sideB_configs.yaml> configuration file under the ~/.config/syncall/ directory.

    This section will have the same name as the combination file deleted in the earlier step and will look like this:

    ---
testnote__None__test_tag:
  gkeep_note: testnote
  tw_project: null
  tw_tags: !!python/tuple
    - test_tag

  3. Remove the items of one of the sides. Keep e.g. the items from the Google Keep note and delete the tasks of the tag/project you are using for synchronization.

  4. Rerun synchronization from scratch to populate the one side with the items of the other side.

I want to synchronize more than a single (Notion page / TW filter), (Google Calendar / TW filter) etc. combination.

See combinations.md.

Tab Completion

All synchronization executables support tab auto-completion for their options for bash, zsh and fish. You can find them under completions/

Miscellaneous

Self Promotion

If you find this tool useful, please star it on Github

TODO List

See ISSUES list for the things that I'm currently either working on or interested in implementing in the near future. In case there's something you are interesting in working on, don't hesitate to either ask for clarifications or just do it and directly make a PR.