Skip to content
This repository has been archived by the owner on Oct 22, 2019. It is now read-only.

Two hosts managed to get the same block #58

Closed
plwhite opened this issue Dec 2, 2015 · 3 comments
Closed

Two hosts managed to get the same block #58

plwhite opened this issue Dec 2, 2015 · 3 comments

Comments

@plwhite
Copy link
Contributor

plwhite commented Dec 2, 2015

The code I was running does not include the fix to issue #47 - if you think that's the issue, feel free to comment and close without wasting any time on it.

I have 500 hosts trying to grab addresses at once. See the following.

core@plw-etcd-00 ~ $ etcdctl ls /calico/ipam/v2 --recursive | grep 192.168.242.192-26
/calico/ipam/v2/host/host-0282/ipv4/block/192.168.242.192-26
/calico/ipam/v2/host/host-0268/ipv4/block/192.168.242.192-26
/calico/ipam/v2/assignment/ipv4/block/192.168.242.192-26
core@plw-etcd-00 ~ $ etcdctl get /calico/ipam/v2/assignment/ipv4/block/192.168.242.192-26
{"attributes": [{"handle_id": null, "secondary": {}}], "cidr": "192.168.242.192/26", "affinity": "host:host-0282", "allocations": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}

I have some trace of host-0268 that is probably relevant.

2015-12-02 09:55:17,151:INFO:[140336892864256]:start_job(126): Starting job 1-test-containers-host-0268 on host-0268, {u'environment': {u'PING_TARGET': u'172.31.0.111'}, u'manual_network': {u'profile_ids': [u'prof-110']}, u'container': u'10.240.0.6:5000/ping-agent', u'name': u'host-0268_001-to-172.31.0.111-host-0170', u'exec': [u'python', u'/usr/src/app/ping-agent.py']}
2015-12-02 09:55:17,151:DEBUG:[140336892864256]:read(466): Issuing read for key /calico/bgp/v1/host/plw-host-0268/ip_addr_v4 with args {}
2015-12-02 09:55:17,162:DEBUG:[140336892864256]:_make_request(387): "GET /v2/keys/calico/bgp/v1/host/plw-host-0268/ip_addr_v4 HTTP/1.1" 200 142
2015-12-02 09:55:17,163:DEBUG:[140336892864256]:read(466): Issuing read for key /calico/bgp/v1/host/plw-host-0268/ip_addr_v6 with args {}
2015-12-02 09:55:17,165:DEBUG:[140336892864256]:_make_request(387): "GET /v2/keys/calico/bgp/v1/host/plw-host-0268/ip_addr_v6 HTTP/1.1" 200 131
2015-12-02 09:55:17,166:DEBUG:[140336892864256]:read(466): Issuing read for key /calico/v1/ipam/v4/pool/ with args {'recursive': True}
2015-12-02 09:55:17,172:DEBUG:[140336892864256]:_make_request(387): "GET /v2/keys/calico/v1/ipam/v4/pool/?recursive=true HTTP/1.1" 200 466
2015-12-02 09:55:17,173:INFO:[140336892864256]:auto_assign_ips(410): Auto-assign 1 IPv4, 0 IPv6 addrs
2015-12-02 09:55:17,173:DEBUG:[140336892864256]:read(466): Issuing read for key /calico/ipam/v2/host/host-0268/ipv4/block/ with args {}
2015-12-02 09:55:17,175:DEBUG:[140336892864256]:_make_request(387): "GET /v2/keys/calico/ipam/v2/host/host-0268/ipv4/block/ HTTP/1.1" 200 267
2015-12-02 09:55:17,175:DEBUG:[140336892864256]:_auto_assign_block(534): Auto-assigning from block 192.168.242.192/26
2015-12-02 09:55:17,176:DEBUG:[140336892864256]:_auto_assign_block(536): Auto-assign from 192.168.242.192/26, retry 0
2015-12-02 09:55:17,176:DEBUG:[140336892864256]:read(466): Issuing read for key /calico/ipam/v2/assignment/ipv4/block/192.168.242.192-26 with args {}
2015-12-02 09:55:17,177:DEBUG:[140336892864256]:_make_request(387): "GET /v2/keys/calico/ipam/v2/assignment/ipv4/block/192.168.242.192-26 HTTP/1.1" 200 638
2015-12-02 09:55:17,178:ERROR:[140336892864256]:process_jobs(615): Failed to create container
Traceback (most recent call last):
  File "./host-agent.py", line 612, in process_jobs
    etcd_client, ipam_client, docker_client)
  File "./host-agent.py", line 165, in start_job
    attributes={})
  File "/usr/local/lib/python2.7/site-packages/pycalico/ipam.py", line 412, in auto_assign_ips
    attributes, pool[0])
  File "/usr/local/lib/python2.7/site-packages/pycalico/ipam.py", line 465, in _auto_assign
    attributes)
  File "/usr/local/lib/python2.7/site-packages/pycalico/ipam.py", line 542, in _auto_assign_block
    affinity_check=affinity_check)
  File "/usr/local/lib/python2.7/site-packages/pycalico/block.py", line 170, in auto_assign
    (self.host_affinity, affinity_id))
NoHostAffinityWarning: Host affinity is host-0282 (not host-0268)
@plwhite
Copy link
Contributor Author

plwhite commented Dec 2, 2015

And from host-0282, the competing entity.

2015-12-02 09:55:16,921:DEBUG:[140692400150272]:_new_affine_block(140): Checking if block 192.168.242.192/26 is free.
2015-12-02 09:55:16,921:DEBUG:[140692400150272]:read(466): Issuing read for key /calico/ipam/v2/assignment/ipv4/block/192.168.242.192-26 with args {}
2015-12-02 09:55:16,922:DEBUG:[140692400150272]:_make_request(387): "GET /v2/keys/calico/ipam/v2/assignment/ipv4/block/192.168.242.192-26 HTTP/1.1" 404 126
2015-12-02 09:55:16,922:DEBUG:[140692400150272]:_new_affine_block(145): Found block 192.168.242.192/26 free.
2015-12-02 09:55:16,923:DEBUG:[140692400150272]:write(378): Writing  to key /calico/ipam/v2/host/host-0282/ipv4/block/192.168.242.192-26 ttl=None dir=False append=False
2015-12-02 09:55:17,010:DEBUG:[140692400150272]:_make_request(387): "PUT /v2/keys/calico/ipam/v2/host/host-0282/ipv4/block/192.168.242.192-26 HTTP/1.1" 201 151
2015-12-02 09:55:17,010:DEBUG:[140692400150272]:_compare_and_swap_block(78): CAS Write new block <pycalico.block.AllocationBlock object at 0x7ff579be4d90>
2015-12-02 09:55:17,011:DEBUG:[140692400150272]:write(378): Writing {"attributes": [], "cidr": "192.168.242.192/26", "affinity": "host:host-0282", "allocations": [null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]} to key /calico/ipam/v2/assignment/ipv4/block/192.168.242.192-26 ttl=None dir=False append=False
2015-12-02 09:55:17,112:DEBUG:[140692400150272]:_make_request(387): "PUT /v2/keys/calico/ipam/v2/assignment/ipv4/block/192.168.242.192-26 HTTP/1.1" 201 641
2015-12-02 09:55:17,113:DEBUG:[140692400150272]:_auto_assign_block(534): Auto-assigning from block 192.168.242.192/26
2015-12-02 09:55:17,113:DEBUG:[140692400150272]:_auto_assign_block(536): Auto-assign from 192.168.242.192/26, retry 0

@robbrockbank
Copy link
Contributor

The stack trace suggests this is not the latest release of libcalico. I believe this issue is resolved with the latest libcalico. Could you retest with that?

@plwhite
Copy link
Contributor Author

plwhite commented Dec 15, 2015

This no longer repros with the latest code. Closing.

@plwhite plwhite closed this as completed Dec 15, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@robbrockbank @plwhite