Skip to content

Latest commit

 

History

History
142 lines (133 loc) · 9.16 KB

README.md

File metadata and controls

142 lines (133 loc) · 9.16 KB
Raw

gitrtc

rtc component to git repository migration tool, with Django framework

Credits: https://github.com/rtcTo/rtc2git

Progress

  • The tool is good for changeset history collection and stream branching determine
  • The tool is good for the actual migration
  • The tool is good for the validation
  • The tool tags rtc baselines it collected

What does it do?

  • it collect changeset history for all of the streams of a component
    • latest rtc client no longer has 1000 history issue, so use it
  • it compares these history and determine the proper branching points
    • for complicated branches, you need perform further with options --branchfurther --streamparent parent_stream_name
  • it migrate the streams and forcely verify the trunk stream of its branching points and baselines, non-trunk streams optional
  • it collect and generate important baselines and tag them in the git
  • it creates workspace git_history_ComponentName (for changeset history collection, can be safely deleted, component level)
  • it creates certain workspaces for each stream migrating
    • git_verify_StreamName (verify baselines for the stream, can be deleted safely)
    • git_migrate_ComponentName_StreamName (for the migration, should keep it for incremental updates)

Requirements

  • tool assumes:
    • comoponent name does not contain space
    • component name and stream name does not contains colon
    • both does not contain shell escaped special characters
    • baseline name and comment does not contain shell sensitive characters other than space and colon
  • this intends to support only latest rtc client (that lscm show history returns unlimited items, the earlier version only return 100 or 1000) -- if your rtc is older, you need provide the history files for all streams to migrate
  • SCM tools (recommend lscm for performance, tune java heap size to 4G+ if you have large amount of changesets in streams)
  • python3 + Django<2 + django-mptt + docutils
  • mysql + mysqlclient (optional, if your component has 10K+ changesets for performance consideration, I did 50K+ with sqlite3 though)
  • use ubuntu 16.04 x64 when doing the migration, with latest scmtools (or use the docker image)

Tested on

Steps

1. install scmtools and login with proper account as the alias rtc

# lscm logout -r rtc
# lscm login -r https://rtc/url -u your_account -P your_password -n rtc -c (if you password is TOO SPECIAL, change it simpler)
# lscm list connections  (you should see the line with something including repo/url and your_account)
# lscm help  (see help information)

2. install python3

3. install python3 modules

# pip3 install "Django<2" django-mptt docutils
# pip3 install mysqlclient (optional, only if select to use mysql database)

4. fork, clone and configure the tool

  • edit local_settings.py, you can specify your rtc information to automate login and relogin (rtc login / relogin works in shell environment, you might need change your password to make it work, confirm with cmd line test in step 1 above)
  • edit local_settings.py, you can configure your mail function
  • edit local_settings.py, COMPONENT_CREATORS provide information for the component creator info (by default I only figure out user name of the creator, but do not know the email, this variable provides this info, if it is not found, I will use [email protected] instead)
  • edit local_settings.py to decide using sqlite or mysql for database backend. (optional)

5. run the tool:

./djangoapp.py --component component_name --streambase your_trunk_stream_for_the_component [--infoinit | --infoshow | --infoupdate | --inforeport | --migrate | --infoverify] [other options]

  • --infoinit (run only once, you should keep using fake component like 'init' and fake streambase like 'init_trunk')
    • retrieve all of your project areas
    • retrieve their related streams
    • retrieve components for each stream
    • usually you only need to do this once, result will be saved as rtc/fixtures/infoinit.json and will be loaded when you process other components
    • you only need to update this when you add/remove streams or modified components in streams
  • --infoshow [--streams[id1,id2,]] [--writehistory]
    • list streams with their ids that you can specify in options like --streams
    • --streams [id1,id2,]
      • show informations to a subset of streams
    • --writehistory
      • write changeset history of streams
  • --infoupdate
    • list all of the baselines for the component
    • for each stream containing the component, do the following with streambase the first one to process
      • list and create tree of changeset for the stream
      • list baselines with the corresponding changesets associated
      • determine branching point against streambase (you may need to do further compare for branches with common branching point on trunk, with --branchfurther --streamparent ParentStreamName)
    • you can pickup new delivered changesets with --incremental
    • you can do a lightweight analysis to determine branching information with --shortcut_analyze
  • --inforeport [--withbaselines] [--withchangesets] [--streams [id1,id2,]] [--levelrange]
    • report rtc information and component specific numbers
      • --withchangesets show changeset information on branching points and ending points
      • --withbaselines show baseline information, help
      • --streams [id1,id2,...] limit the streams to show in the report if you have too many or you only interest in some of them
      • --levelrange [0,1000] --levelinterval 10 used when --withchangesets
  • --migrate [--allstreams] [--streams [id1,id2]] [--withvalidation]
    • migrate only the trunk with --migrate only
    • you can migrate any streams using --streams [stream_id, stream_id2]
    • you can try to migrate all streams with --allstreams
    • you can use --incremental to pickup recently delivered changesets (collected already by --infoupdate --incremental)
    • you can use --withvalidation to do baseline and branching point validations while migrating
  • --infoverify [--withbranchingpoints ] [--streams [id1,id2] | --allstreams]
    • verify baseline in stream of the trunk by default
    • specify --allstreams to validate all of the streams migrated
    • with --streams [id1,id2] to validate specified streams
    • with --withbranchingpoints to also validate the branching points if there is and not valided yet

Examples

1 ./djangoapp.py --component init --streambase init_trunk --infoinit (only once)

  1. this generate rtc/fixtures/infoinit.json which will be loaded for each component you are going to migrate

2 ./djangoapp.py --component yarn --streambase yarn_trunk --infoupdate

  1. ./djangoapp.py --component yarn --streambase yarn_trunk --infoupdate --branchfuther --streamparent yarn_IOP_working

-. if all of you streams are directly cut from trunk, this step is not necessary -. if your branching if complicated, you would spend quite some time in this step

  1. ./djangoapp.py --component yarn --streambase yarn_trunk --infoupdate --incremental --streams [200]

-. perform an update of stream with id 200, this will pickup any new baselines and changesets since last --infoupdate

3 ./djangoapp.py --component yarn --streambase yarn_trunk --inforeport

# ./djangoapp.py --component yarn --streambase yarn_trunk --inforeport --withbaselines
# ./djangoapp.py --component yarn --streambase yarn_trunk --inforeport --withchangesets
# ./djangoapp.py --component yarn --streambase yarn_trunk --inforeport --withbaselines --withchangesets --streams [3,5,7]

4. ./djangoapp.py --component yarn --streambase yarn_trunk --migrate (this migrate the trunk stream)

  1. ./djangoapp.py --component yarn --streambase yarn_trunk --migrate --streams [200]

-. migrate a single stream whose id is 200

  1. ./djangoapp.py --component yarn --streambase yarn_trunk --migrate --allstreams

-. tries to migrate all streams. If your stream relationship is complicated, you might need to run multiple times

  1. ./djangoapp.py --component yarn --streambase yarn_trunk --migrate --incremental

-. perform an updated migration of trunk stream to git

  1. ./djangoapp.py --component yarn --streambase yarn_trunk --migrate --incremental --streams [200]

-. perform an updated migration of stream with id 200 to git

  1. ./djangoapp.py --component yarn --streambase yarn_trunk --migrate --withvalidation --allstreams

-. perform the migration with validatoins of all streams

  1. ./djangoapp.py --component yarn --streambase yarn_trunk --migrate --tagbaselines

-. create git tags corresponding rtc baselines (of those validated)

5 ./djangoapp.py --component yarn --streambase yarn_trunk --infoverify (this verifies the trunk stream's baselines)

  1. ./djangoapp.py --component yarn --streambase yarn_trunk --infoverify --streams [id1,id2]

-. this verifies streams specified

  1. ./djangoapp.py --component yarn --streambase yarn_trunk --infoverify --allstreams

-. this verifies all streams

  1. ./djangoapp.py --component yarn --streambase yarn_trunk --infoverify --withbranchingpoints

-. this verifies the branching points of a stream, you can specify streams to verify against