[Work in progress] Implement concurrent resync#852

Merged

plwhite merged 98 commits intomasterfrom smc-concurrent-resync

Nov 17, 2015

+4,678-1,139

Commits on Nov 10, 2015

Prototype etcd driver process, replaces etcd watch loop.
Shaun Crampton
committed
Move etcd driver into its own package.
Shaun Crampton
committed
Move etcd driver code into own package, Felix now starts the driver.
Shaun Crampton
committed
Move gevent-specific function to geventutils. Set up logging in driver.
Shaun Crampton
committed
Work in progress on productising etcd driver.
Shaun Crampton
committed
Move complex HWM calculations into a dedicated class. General tidy-up.
Shaun Crampton
committed
Optimization: avoid trie lookup for most events.
Shaun Crampton
committed
Use /run/felix-driver.sck for socket.
Shaun Crampton
committed
Experiment: use a BytesIO to avoid calling sendall as often.
Shaun Crampton
committed
Refactor etcd driver into a class. Add init message.
Shaun Crampton
committed
Move Ready key check and config loading to driver.
Shaun Crampton
committed
Pass config back ot driver once it's been validated.
Shaun Crampton
committed
Driver now sends status messages to Felix.
Shaun Crampton
committed
Minor cleanups: logging and comments.
Shaun Crampton
committed
Handle errors from etcd when loading snapshot.
Shaun Crampton
committed
Handle HTTP and etcd errors in watcher loop.
Shaun Crampton
committed
Check cluster ID on etcd responses.
Shaun Crampton
committed
Suppress error when reads time out on a watch.
Shaun Crampton
committed
PReliminary requirements update.
Shaun Crampton
committed
Allow ':' and '.' in etcd keys.
Shaun Crampton
committed
Do not handle any updates from driver until begin_polling is set.
Shaun Crampton
committed
In driver, trigger resync if ready key is unset.
Shaun Crampton
committed
Pull out protocol constants into new file.
Shaun Crampton
committed
Cleanup: remove now-unused code.
Shaun Crampton
committed
Rename driver handler methods.
Shaun Crampton
committed
Implement in-sync handling for tag manager.
Shaun Crampton
committed
Implement EndpointManager/DispatchChain in-sync processing.
Shaun Crampton
committed
Block ProfileRules startup until we're in sync.
Shaun Crampton
committed
Make Felix's etcd watching thread more cooperative.
Shaun Crampton
committed
Only log tag update message if we actually updated tags.
Shaun Crampton
committed
Fix up tests, remove now-invalid tests.
Shaun Crampton
committed
Add new deb dependencies.
Shaun Crampton
committed
Defensively check for process adoption in driver.
Shaun Crampton
committed
Defensive: encode keys that use characters not allowed in the trie.
Shaun Crampton
committed
Fix that in-sync was logged at error.
Shaun Crampton
committed
Add msgpack dependency.
Shaun Crampton
committed
Add versions to felix_requirements to match built packages.
Shaun Crampton
committed
Rev build number for upgrade testing.
Shaun Crampton
committed
Trigger status cleanup and hosts ipset updates from in-sync message.
Shaun Crampton
committed
Fix python-msgpack version.
Shaun Crampton
committed
Tidy up fetcd.py:
Shaun Crampton
committed
Implement periodic resync.
Shaun Crampton
committed
Fic incorrect use of send() vs sendall().
Shaun Crampton
committed
Make sure status messages are flushed immediately.
Shaun Crampton
committed
Ensure start-of-dat cleanup only runs once when we Felix moves to being in sync.
Shaun Crampton
committed
Trigger resync if the whole /calico/v1 directory disappears.
Shaun Crampton
committed
Fix UT broken by variable rename.
Shaun Crampton
committed
clean up fetcd.py. Add stats, comments and logging.
Shaun Crampton
committed
Move Felix socket handling into utility classes. Clean up EtcdEvent class.
Shaun Crampton
committed
Refactor driver to use MessageReader/Writer classes.
Shaun Crampton
committed
Fix up imports.
Shaun Crampton
committed
Add configuration for driver's log file location.
Shaun Crampton
committed
Set syslog executable name in driver.
Shaun Crampton
committed
Remove fake stack and use function calls instead.
Shaun Crampton
committed
Make URL calculation common; other minor cleanups.
Shaun Crampton
committed
Add comments to HWM tracker.
Shaun Crampton
committed
Add UTs for HWM tracking.
Shaun Crampton
committed
Factor out shared request logic in driver.
Shaun Crampton
committed
Fix hard-coded URL for etcd.
Shaun Crampton
committed
Add initial FV-level test for driver.
Shaun Crampton
committed
Fix thread shutdown; make sure threads are daemons and join them correctly.
Shaun Crampton
committed
Minor fixes: prevent blocking forever on events.
Shaun Crampton
committed
Record ready flag in HWM cache so that we spot if it disappears.
Shaun Crampton
committed
Add FV-level test for mainline resync processing in driver.
Shaun Crampton
committed
Minor cleanups; get driver tests working with other tests.
Shaun Crampton
committed
Add UTs for protocol.py read/write function. Minor fix to reader.
Shaun Crampton
committed
Cover remaining lines in protocol.py.
Shaun Crampton
committed
Streamline driver test, avoid messy stack trace on success.
Shaun Crampton
committed
Make timeout=1 the default for MessageReader. Avoids accidentally blocking forever.
Shaun Crampton
committed
More driver FV tests covering various error cases.
Shaun Crampton
committed
Move stub code to own file.
Shaun Crampton
committed
Remove now-unused parse_if_* functions.
Shaun Crampton
committed
Cover new config.
Shaun Crampton
committed
Cover new EndpointManager function.
Shaun Crampton
committed
Minor fix: removing chains should be synchronous.
Shaun Crampton
committed
Prevent accidental start of a real LocalEndpoint during test.

fasaxc
authored and
Shaun Crampton
committed
Full coverage for EndpointManager.
Shaun Crampton
committed
Add UTs for ProfileManager.
Shaun Crampton
committed
Remove now-unused force_reprogram flag. Add UT for ipsets.py.
Shaun Crampton
committed
Exclude new test files from coverage.
Shaun Crampton
committed
Additional UT for etcd driver.
Shaun Crampton
committed
First batch of tests for new fetcd function.
Shaun Crampton
committed
Cover dispatch functions in fetcd.
Shaun Crampton
committed
More coverage for fetcd.py.
Shaun Crampton
committed
Additional coverage for etcdutils.py.
Shaun Crampton
committed
Add etcddriver to coveragerc includes.
Shaun Crampton
committed
Cover __main__.py.
Shaun Crampton
committed
More coverage for driver.py and some minor fixes:
Shaun Crampton
committed
Convince PyCharm's linter that patch.object exists. Removes many warnings.
Shaun Crampton
committed
More coverage for driver.py and hwm.py.
Shaun Crampton
committed
Cover splitter.py.
Shaun Crampton
committed
Update dependencies.
Shaun Crampton
committed
Add calico.etcddriver to list of tests in tox.ini.
Shaun Crampton
committed
Update changelogs.
Shaun Crampton
committed
Rework driver tests to handle concurrent requests from old/new watcher.
Shaun Crampton
committed

Commits on Nov 12, 2015

Code review markups for concurrent resync function.
Shaun Crampton
committed
Code review markups: bound size of watcher queue.
Shaun Crampton
committed
Code review markups: don't stop the watcher unless we have to.
Shaun Crampton
committed