Tractive: migrating from Trac to GitHub

Tractive is a tool that helps you migrate Trac instances to GitHub.

Specifically, it converts Trac tickets into GitHub Issues by accessing the Trac database’s issues and posts the change history of each ticket as issue comments.

Tractive is written using re-tooled code based on the excellent trac-hub migration tool.

Note	This tool was created in order to perform Trac-to-GitHub migration for the IETF.

Installation

Prerequisites

1. Ruby

2. Git

3. reposurgeon, which works on Linux (manual compile recommended for the latest versions) and macOS (use Homebrew)

4. trac-admin must be installed in order to extract Trac attachments

Steps

Install it via:

$ gem install tractive

Usage

Prerequisites

1. A GitHub Account as the importer

Steps to migrate from Trac to GitHub

1. Migrate your Trac-managed SVN repositories to GitHub. (Migrating SVN to GitHub using reposurgeon)

2. Build the SVN Revision to Git Commit RevMap. (Building the Trac Revision to Git Commit RevMap)

3. Bootstrap your Tractive config file. (Boostrapping the Tractive configuration file)

4. Configure mapping in your Tractive config file. (Working with the Tractive configuration file, and mapping)

5. Run Tractive (Running Tractive)

   $ tractive <options>

Migrating SVN to GitHub using reposurgeon

reposurgeon is the leading tool today (and under active development) that allows conversion among multiple version control systems.

We strongly recommend you to use the excellent reposurgeon for this task.

In the case of Trac to GitHub, we are mostly looking at SVN to Git.

Note	For further details, please see the reposurgeon guide.

Typically these are the steps to take:

1. In an empty directory, run repotool --initialize {name-of-repo}. Enter that the source VCS is svn, the destination is git. The directory will then be populated with a number of files, including:

   • Makefile includes multiple useful tasks, requires user configuration.

   • {name-of-repo}.lift, a file to store reposurgeon conversion configuration.

   • {name-of-repo}.map, a reposurgeon specific map of SVN users to GitHub users/emails)

   • {name-of-repo}.opts, to enter pre-read reposurgeon options.

2. Customize these lines of the Makefile with the SVN URL and reposurgeon "read options":

   REMOTE_URL = https://svn.ietf.org/svn/tools/mailarch
   READ_OPTIONS = --branchify=trunk:branch/*:personal/*/*:tags/*:*

3. Update the {name-of-repo}.lift with additional reposurgeon commands (see the reference guide) if necessary. e.g.

   branch rename refs/heads/master refs/heads/main
   
   msgin --create <<EOF
   Tag-Name: git-conversion
   Tagger: IETF Bot <[email protected]> America/Los_Angeles
   
   Marks the spot where this repository was migrated from Subversion to git.
   EOF

4. Download the SVN repository in mirror mode to the {name-of-repo}-mirror/ directory. This is necessary for faster processing and so that reposurgeon won’t attempt to download from the REMOTE_URL.

5. Generate the user map from the SVN repository with make stubmap. Edit the {name-of-repo}.map to correct/fix it if necessary.

6. Run make to generate {name-of-repo}.svn and {name-of-repo}.fo. The resulting Git repository will be at {name-of-repo}-git/.

7. The {name-of-repo}.fo file is a SVN Revision to timestamp mapping. Tractive uses this file to build the SVN Revision to Git Commit SHA hash mapping.

Building the Trac Revision to Git Commit RevMap

RevMap is a mapping file that contains a one-to-many mapping of an SVN revision number to its corresponding Git commit SHA hash(es).

Note	An SVN revision may map to multiple, identical Git commits; see reposurgeon documentation for more details.

With the {name-of-repo}.fo file generated by reposurgeon (or any other tool for that matter, as long as the format is identical), we can build the SVN revision to Git commit SHA "RevMap".

A typical {name-of-repo}.fo file looks like this:

SVN:9	2012-10-10T23:50:[email protected]
SVN:10	2012-10-10T23:50:[email protected]
SVN:12	2012-10-10T23:52:[email protected]
SVN:13	2012-10-10T23:52:[email protected]
...
SVN:33	2013-02-07T04:01:[email protected]
SVN:34	2013-02-12T17:46:[email protected]
SVN:35.2	2013-02-12T20:15:[email protected]
...

The left column is the SVN revision, while the right column provides a timestamp for which you can find with Git. If you enter {name-of-repo}-git/, you can use the git scommit and git scommits commands (defined in the reposurgeon guide) to view the corresponding Git commit(s).

Notice that:

• typically there is a single timestamp per SVN Revision. In this case each timestamp can be identified with one Git commit via git scommit {timestamp}.

• sometimes two SVN revisions have identical timestamps. In this case you will need to use git scommits {timestamp} to see all these commits.

• sometimes one SVN revision maps to multiple Git commits. In this case you will need to use git scommits {timestamp} to see all these commits.

With this information you can now build the RevMap with:

tractive generate revmap \
  --svn-url <url of the SVN repository> \
  --rev-timestamp-file <reposurgeon timestamp map, e.g. {name-of-repo}.fo> \
  --git-local-repo-path <path to converted Git repository, e.g. {name-of-repo}-git> \
  --revmap-output-file <output path of RevMap.txt>

The generated RevMap will be then used in a Tractive run so that the migrated issues and commits will replace references to SVN revisions with the corresponding Git commit SHA, enabling GitHub to expose those linkages in the user interface.

Table 1. Options for tractive generate revmap command

Option	Description	Type
--svn-url	(required unless --svn-local-path set) SVN repository URL that should be used in RevMap generation. The URL must start with the http:// or https:// prefix (not the svn:// prefix).	String
--svn-local-path	(required unless -svn-url set) SVN local repository path that should be used in RevMap generation. You can clone the svn repo locally and provide its local path if the svn repository is offline.	String
--rev-timestamp-file	(required) Specify the input file that contains SVN revision and timestamps that should be used in RevMap generation. The format of the file must follow the reposurgeon .fo format.	String
--git-local-repo-path	(required) Local Git repository path that should be used in RevMap generation. The path must be a Git repository with a .git folder within.	String
--revmap-output-file	(required) Output file path to save the generated RevMap.	String

Boostrapping the Tractive configuration file

General

Tractive uses a YAML configuration file that contains configuration to export data from Trac and then import to GitHub.

Setting up the configuration file

Copy the example YAML configuration and adapt it as necessary:

cp config.example.yaml config.yaml
vi config.yaml

Then, at a minimum, you need to setup the following sections for bootstrapping (the next step) to work:

1. Trac configuration (Trac configuration)

2. GitHub configuration (GitHub configuration)

Bootstrapping the configuration file with information

The Tractive configuration file is used to perform the actual migration actions, and therefore depends heavily on the contents of the content to be migrated.

Tractive can read certain information from the Trac database to allow for easier mapping of users, labels and milestones.

The following command provides that information.

tractive -i

Working with the Tractive configuration file, and mapping

The configuration file contains the following sections.

Trac configuration

trac:

(mandatory) Trac (data source) configuration options.

database:

Database access URL. The database URL follows the Sequel scheme. SQLite, MySQL, Postgres endpoints supported.

ticketbaseurl:

URL of the Trac "tickets" interface.

EXAMPLE:

trac:
  # For MySQL: mysql2://user:password@host:port/database
  database: sqlite://db/trac.db
  ticketbaseurl: https://example.org/trac/foobar/ticket

ticketbaseurl:

The Trac Url which will be added in Github issues. A link will be added in the footer of Github issue to link it to Trac ticket.

GitHub configuration

github:

(mandatory) GitHub (migration target) configuration options.

repo:

Target GitHub organization and repo name as {github-org}/{repo-name}.

token:

Personal Access Token of the GitHub user used to import. This token can be generated under GitHub Settings > Personal Access Tokens. e.g. 'ghp_fpsc4de1f0c46e01576810740c9242097cba4619486'.

local_repo_path:

Local path to the migrated Git repository. e.g. '/Users/user/repo-git'.

revmap_path:

Local path to the RevMap file generated via the tractive generate revmap command.

EXAMPLE:

github:
  repo: 'example-org/target-repository'
  token: 'ghp_fpsc4de1f0c46e01576810740c9242097cba4619486'
  local_repo_path: '/Users/user/repo-git'

revmap_path: ./example-revmap.txt

Attribute/Label mapping

Since GitHub’s issue tracker does not have a first-class notion of ticket priority, type, and version information, Tractive supports expressing these in the form of labels.

The pattern of a mapping is like:

+

{trac-ticket-type}:
  {trac-ticket-type-value}: {github-label-value}

labels:

provides custom label mappings.

type:

Type of the Trac ticket. e.g.

  defect:      defect
  task:        task
  enhancement: feature request
  cleanup:     cleanup

component:

Component of the Trac ticket. e.g.

  configuration:
    name: conf
    color: ff00ff
  documentation:
    name: doc
    color: 00ff00

resolution:

Resolution of the Trac ticket. e.g.

  fixed:      fixed
  invalid:    invalid
  wontfix:    wontfix
  duplicate:  duplicated
  worksforme: worksforme
  obe:        obe

platform:

Platform related to the Trac ticket. e.g.

  Linux:   Linux
  Windows: Windows

severity:

Severity of the Trac ticket. (also called Priority in Trac) e.g.

  trivial:
    name: trivial
    color: ff0000
  major:
    name: major
    color: b44647
  minor:
    name: minor
    color: f7347a
  medium:
    name: medium
    color: f3c77c

priority:

Priority of the Trac ticket.

  Low:
    name: low
    color: 22dd00
  High:
    name: high
    color: ff0000

tracstate:

Status of the Trac ticket.

  accepted:
    name: accepted
    color: 22dd00
  assigned:
    name: assigned
    color: aadd88
  closed:
    name: closed
    color: ee00aa
  new:
    name: new
    color:

Note	As component, severity, priority and tracstate are converted into labels on github so there is an option to specify the color for those labels.

User mapping

users:

a one-to-one mapping between Trac usernames or email addresses to GitHub usernames for users, in the following pattern:

users:
  {Trac email or username}:
    email: {Github email}
    name: {name of the person}
    username: {username on GitHub}
  ...

EXAMPLE:

users:
  [email protected]:
    email: [email protected]
    name: Matthew
    username: example-matt
  valencia:
    email: valencia
    name: Valencia
    username: example-vale

If you don’t want to map a user, you can just leave the username empty like below:

users:
  [email protected]:
    email: [email protected]
    name: Matthew
    username:

Milestone mapping

milestones:

mapping of milestones.

{milestone-name}:

ID of the milestone.

name:

Name of the milestone.

due:

Due date in POSIX msec.

completed:

Date of completion in POSIX msec.

description:

Description of the milestone

EXAMPLE:

milestones:
  '2021_02':
    name: '2021_02'
    due: 1392595200000000
    completed: 1415959156000000
    description: ''

Attachments migration configuration

ticket | wiki:

specifies the options for the tickets or wikis

delete_mocked:

Whether to delete mocked tickets after migration or not

attachments:

specifies method of obtaining attachments from Trac.

url:

URL to obtain Trac attachments from

hashed:

Whether the url has hased or plain image names and ids

export_folder:

folder where the attachments will be downloaded to from Trac.

export_script:

output of a script that utilizes trac-admin to download all attachments from Trac.

Note	To delete the issues, an organization owner must enable deleting an issue for the organization’s repositories, and you must have admin or owner permissions in the repository. For more information, see "deleting an issue".

ticket:
  delete_mocked: true
  attachments:
    url: https://abc.com/raw-attachment/ticket
    hashed: true
    export_folder: ./attachments
    export_script: attachments.sh

wiki:
  attachments:
    url: https://abc.com/raw-attachment/wiki
    hashed: true

By using the -i option, you can easily produce a YAML file with labels, users, milestones etc. You can copy the output into the config.yaml file and adapt it as required.

Once the command is completed you should run attachments.sh to export the attachments.

Running Tractive

Thereafter just invoke tractive to read from the default config.yaml path:

tractive

By default, tractive assumes the file config.yaml to be in the same directory as the command was run.

You can also specify the configuration file on the command line:

tractive -c foo.yaml

Filtering Trac tickets by status

Add the -o flag to only import the tickets that are not in a closed status:

tractive -o

Single post mode

If you want all Trac comments and changes within one Trac ticket to be compiled into a single issue at GitHub:

tractive -S

Resuming Trac ticket import

To resume the migration at a given Trac ticket ID, use -s:

tractive -s 42

Matching Trac ticket IDs and GitHub issues IDs

You may want to ensure that the imported Trac ticket IDs are identical to the GitHub Issues ID. This section applies when migrating to an existing or empty GitHub repository,

Tractive can help you create dummy tickets (and close them) for IDs missing in Trac (because they were deleted). This works even if you run it multiple times.

tractive -M

Note	When converting your Trac setup to GitHub, it is prudent to first try the migration into a test repository which you can delete afterwards. If the run was smooth and delivers the expected results, you can re-run the migration for the real target repository.

As the process can be interrupted, you can always specify the first ID number to migrate. In this case you need to provide the -s argument for the first ID not available in Github.

tractive -M -s 601

Fast issue import

By default, Tractive will verify that the created issue numbers match the ticket IDs of the corresponding Trac ticket and error-exit if the numbers do not match.

The fast import option allows you to disable this safe-checking behavior.

In order to utilize this feature, you should also disable user interactions by setting Limit to repository collaborators under your repository settings. Alternatively, when migrating issues to a new repository, import the issues on a test-repository and rename the repository to the final name when the import went satisfactory.

You can disable this check by using the fast option:

tractive -F

In effect your import will be much faster since there is no ID synchronization (but after the script has finished, it can still take some time until the issues are created on github).

If you know that the ticket IDs will not match, e.g. there are existing GitHub issues that are not created by import, using the fast import option is obligatory. In this case, you must specify the ID of the first Trac ticket to be migrated (even if it is 1):

tractive -F -s 1

References

Command line options

Option	Description	Type
-A, --attachment-exporter	Generate an attachment exporter script according to config.yaml.	String
-a, --attachment-url	Add URL to Cloud Host where attachment files are available.	String
-c	Set the configuration file. Default value: tractive.config.yaml.	String
-d, --dry-run	Write exported data to a local file instead of pushing it to Github.	Boolean
-e, --export-attachments	Export attachments from the database according to config.yaml.	String
-F, --fast-import	Import without safety-checking GitHub issue numbers.	Boolean
-f, --filter	Filter Trac tickets that you want to import. The following options are allowed (at least one necessary): column-name operator column-value include-null	Boolean
--column-name	Name of the column to filter.	String
--operator	Operator for filter. Example of operators include LIKE and =.	String
--column-value	Value of the column to filter.	String
--include-null	Include rows having null value in filtered column.	Boolean
-h, help	Display the Tractive help message, or you can provide a command to know more about a single command via tractive help {command}.	N/A
-I <path>, --import-from-file=<path>	Import issues from a JSON file specified at <path>.	String
-i, --info	Reports existing labels and users in the database.	Boolean
-L, --log-file	Name of the log file to output logs.	String
-M, --mockup	Create mocked closed issues on Github for deleted tickets on Trac.	Boolean
-o, --opened-only	Skips the import of closed tickets.	Boolean
-r, --rev-map-file	Specify path of the RevMap file for migration.	String
-S, --single-post	Put all Trac ticket comments in the first GitHub issue comment.	Boolean
-s <ID>, --start-at <ID>	Start migration from ticket with number <ID>.	String
-v, --verbose	Enable verbose mode.	Boolean

Migrating Wikis

You need the following for wiki migration

• Clone of the github wiki repo.

Then you can run the migration with:

tractive migrate-wikis \
  --attachment-base-url <url for the attachments> \
  --trac-database-path <full path of trac database> \
  --repo-path <path to cloned Git wiki repository, e.g. {name-of-repo}.wiki>

After that open a terminal in the git wiki folder and run git push

Table 2. Options for tractive migrate-wikis command

Option	Description	Type
a, --attachment-base-url	(required unless specified in config file) If attachment files are reachable via a URL we reference this here.	String
-d, --trac-database-path	(required unless specified in config file) Full path of the Trac sqlite3 database export file.	String
-r, --repo-path	(required) Full path to the root of the git-repository that is our destination.	String
-c, --config	Path to config file.	String
-e, --wiki-extensions	Space separated list of extensions or filenames (if the file don’t have an extension). This is required to convert file SVN source links to Github links. This is used to determine if a path belongs to a file or a directory.	Array
-f, --source-folders	Space separated list of non standard folders in SVN that are used to find if a path is complete or partial.	Array

Implementation details

Usage of GitHub Issue Import API

Tractive uses the GitHub Issue Import API to create Issues.

While this API is not available via GitHub’s official API bindings (e.g. Octokit), it offers several advantages over the normal Issue creation API:

• will not trigger abuse detection warnings and will not get blocked

• does not send out email notifications on issue changes

• does not increase your contribution count (especially if you attempt multiple import tries)

• faster than with the GitHub v3 Issues API

• allows setting the correct creation/closed date

• creates atomic changes without allowing users to interfere in the creation of a single issue and its comments.

The caveat is that there is still no way of migrating an issue or comment attributing to a third-party user account without using that user’s GitHub account, but this is likely a security concern that will not be addressed by GitHub.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/ietf-ribose/tractive. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the code of conduct.

Code of conduct

Everyone interacting in the Tractive project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.

License

Tractive and its supporting code are licensed under a BSD-style licence.

Code inherited from trac-hub is under Matthias Vallentin’s BSD 3-Clause License.

Tractive is funded and developed by Ribose Inc.