warning for skipped hrefs leading to unhanded exception #113

jludwig · 2014-09-08T23:19:20Z

Hello,

First of all, great work with this project. I am very close to throwing away Outlook completely. This is the last step!

Now for the issue:

I am using Davmail as the middleman between Outlook Exchange 2011 and vdirsyncer, and for the most part it works flawlessly. However, when I try syncing the 'all staff' calendar, the biggest one, I get a page full of warnings that look like this:

warning: Skipping identical href: /public/example.com/All%20Staff%20Calendar/Foobar%20-%20Appt.EML

After about five minutes, it reaches the end and gives back an unhanded exception that says:

Unhandled exception occured while syncing all_staff_calendar.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 393, in sync_collection
    force_delete=status_name in force_delete
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 129, in sync
    list_b = prepare_list(storage_b, b_href_to_status)
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 76, in prepare_list
    prefetch(storage, rv, download)
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 86, in prefetch
    for href, item, etag in storage.get_multi(hrefs):
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 333, in get_multi
    .format(href, hrefs_left))
KeyError: "/public/example.com/Shared%20All%20Staff/Foobar%20Leave%20early.EML doesn't exist in set([massive freaking array])

When I add debug messaging, it looks pretty standard. These are the last lines displayed before the error:

debug: X-MICROSOFT-CDO-APPT-SEQUENCE:0
debug: X-MICROSOFT-CDO-OWNERAPPTID:2112067640
debug: X-MICROSOFT-CDO-BUSYSTATUS:FREE
debug: X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
debug: X-MICROSOFT-CDO-ALLDAYEVENT:TRUE
debug: X-MICROSOFT-CDO-IMPORTANCE:1
debug: X-MICROSOFT-CDO-INSTTYPE:0
debug: X-MICROSOFT-DISALLOW-COUNTER:FALSE
debug: END:VEVENT
debug: END:VCALENDAR
debug: </C:calendar-data><D:getetag>2014-05-05T20:29:17Z</D:getetag></D:prop><D:status>HTTP/1.1 200 OK</D:status></D:propstat></D:response></D:multistatus>

Please let me know any additional information that would help. Thank you.

The text was updated successfully, but these errors were encountered:

untitaker · 2014-09-09T06:43:10Z

Could you post the contents of the massive freaking array? If you have privacy concerns over it, you may send it to [email protected], although i doubt it contains any personal information.

jludwig · 2014-09-12T20:02:34Z

I gotta commend you for your rapid response time.

git+https://github.com/untitaker/vdirsyncer.git#outlook_fixes did in fact fix that error (thank you!), however now on my all_staff_calendar, I'm getting an error that looks like this:

Unhandled exception occured while syncing all_staff_calendar.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 393, in sync_collection
    force_delete=status_name in force_delete
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 129, in sync
    b_idents = _prepare_idents(storage_b, b_href_to_status)
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 76, in _prepare_idents
    _prefetch(storage, rv, download)
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 86, in _prefetch
    for href, item, etag in storage.get_multi(hrefs):
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 324, in get_multi
    .find('{DAV:}prop') \
AttributeError: 'NoneType' object has no attribute 'find'

I'm unsure whether this has something to do with this issue (allowing what were previously errors proceed and report back as warnings) or whether I should open a new ticket.

untitaker · 2014-09-12T20:07:07Z

That irritates me... while i can see why that error would happen, i don't know
how it could happen in your particular instance, because the "propstat" tags
are all there. Could you send me debug logs once again?

Thanks,
Markus

On Fri, Sep 12, 2014 at 01:02:34PM -0700, Justin Ludwig wrote:

I gotta commend you for your rapid response time.

git+https://github.com/untitaker/vdirsyncer.git#outlook_fixes did in fact fix that error (thank you!), however now on my all_staff_calendar, I'm getting an error that looks like this:
Unhandled exception occured while syncing all_staff_calendar.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 393, in sync_collection
    force_delete=status_name in force_delete
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 129, in sync
    b_idents = _prepare_idents(storage_b, b_href_to_status)
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 76, in _prepare_idents
    _prefetch(storage, rv, download)
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 86, in _prefetch
    for href, item, etag in storage.get_multi(hrefs):
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 324, in get_multi
    .find('{DAV:}prop') \
AttributeError: 'NoneType' object has no attribute 'find'
I'm unsure whether this has something to do with this issue (allowing what were previously errors proceed and report back as warnings) or whether I should open a new ticket.

Reply to this email directly or view it on GitHub:
#113 (comment)

untitaker · 2014-09-20T12:10:39Z

The commit from the outlook_fixes branch was 1c1aade, FTR.

untitaker · 2014-09-25T18:31:16Z

Justin sent me debug logs in private, and it seems to me that the server is sometimes case-sensitive and sometimes not. vdirsyncer makes two requests to fetch both item types. This makes the server respond with two items:

/foo/bar/baz/FOO.eml
/foo/bar/baz/foo.eml

If vdirsyncer tries to fetch both of those items, the server returns one of those items twice. vdirsyncer warns that the item got sent twice, and raises an exception because the other is missing.

@dmfs, do you have any experience with the server mentioned in the OP?

untitaker · 2014-09-25T18:33:36Z

Also, @jludwig, could you try to use

item_types =

for the relevant storage section? This should cause vdirsyncer to send only one request, and bypass the problem without actually fixing it.

dmfs · 2014-09-25T22:54:47Z

@untitaker I know that we do have a couple of users using davmail, but afaik there are no open issues (beside some SSL issues maybe).
I'm just wondering why those files have the extension ".eml". Are they actually iCalendar files? What is the content-type returned for those files?

untitaker · 2014-09-26T05:31:15Z

The file extension is indeed irritating, but the items seem to be normal ics files, and the server advertizes them as such.

untitaker · 2014-09-26T17:50:57Z

@jludwig can you post version information re Davmail? I want to file a bug report on Davmail first and see how it works out.

See Davmail/Outlook, #113

untitaker · 2014-09-26T18:13:40Z

@jludwig Also, could you check out the outlook_fixes branch again?

pip uninstall vdirsyncer
pip install git+https://github.com/untitaker/vdirsyncer.git#outlook_fixes

untitaker · 2014-09-26T18:14:18Z

...but now, remove the item_types parameter and add:

case_sensitive = False

jludwig · 2014-09-26T22:46:36Z

I had the same results on bothO
DavMail for MacOSX-Java7 version 4.5.1-2303 (current version I'm using)
and the standard
DavMail for MacOSX v4.5.1-2303 (version I used on my old computer).

This is the error I received this time:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 393, in sync_collection
    force_delete=status_name in force_delete
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 129, in sync
    b_idents = _prepare_idents(storage_b, b_href_to_status)
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 76, in _prepare_idents
    _prefetch(storage, rv, download)
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 86, in _prefetch
    for href, item, etag in storage.get_multi(hrefs):
  File "/usr/local/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 327, in get_multi
    .find('{DAV:}prop') \
AttributeError: 'NoneType' object has no attribute 'find'

My config looks like this:

[general]
status_path = ~/.vdirsyncer/status/

[pair all_staff]
a = all_staff_local
b = all_staff_remote

[storage all_staff_local]
type = filesystem
path = ~/.calendars/all_staff
fileext = .ics

[storage all_staff_remote]
type = caldav
url = http://localhost:1080/public/path/to/cal
case_sensitive = False
...

Full debug sent privately.

Version 0.3.0 ============= *released on 20 September 2014* - Add ``verify_fingerprint`` parameter to :py:class:`vdirsyncer.storage.HttpStorage`, :py:class:`vdirsyncer.storage.CaldavStorage` and :py:class:`vdirsyncer.storage.CarddavStorage`, see issue `#99`_ and pull request `#106`_. - Add ``passwordeval`` parameter to :ref:`general_config`, see issue `#108`_ and pull request `#117`_. - Emit warnings (instead of exceptions) about certain invalid responses from the server, see issue `#113`_. This is apparently required for compatibility with Davmail. .. _`#99`: pimutils/vdirsyncer#99 .. _`#106`: pimutils/vdirsyncer#106 .. _`#108`: pimutils/vdirsyncer#108 .. _`#113`: pimutils/vdirsyncer#113 .. _`#117`: pimutils/vdirsyncer#117

See Davmail/Outlook, #113

untitaker · 2015-01-01T18:03:36Z

Closing this as a duplicate of #144.

See Davmail/Outlook, #113

untitaker added type: bug mod: DAV labels Sep 18, 2014

untitaker added a commit that referenced this issue Sep 26, 2014

Deal with case-insensitive servers

01b0c33

See Davmail/Outlook, #113

jludwig closed this as completed Sep 26, 2014

jludwig reopened this Sep 26, 2014

untitaker added the unconfirmed label Nov 6, 2014

untitaker mentioned this issue Dec 2, 2014

sync against davmail results in several 'Server sent item twice' and an exception #144

Closed

untitaker added a commit that referenced this issue Dec 3, 2014

Deal with case-insensitive servers

1eefb30

See Davmail/Outlook, #113

untitaker added a commit that referenced this issue Dec 6, 2014

Deal with case-insensitive servers

1b81915

See Davmail/Outlook, #113

untitaker added a commit that referenced this issue Dec 6, 2014

Deal with case-insensitive servers

951d4df

See Davmail/Outlook, #113

untitaker added a commit that referenced this issue Dec 6, 2014

Deal with case-insensitive servers

76a9308

See Davmail/Outlook, #113

untitaker mentioned this issue Dec 6, 2014

Deal with case-insensitive servers #146

Closed

untitaker added a commit that referenced this issue Dec 6, 2014

Deal with case-insensitive servers

be14457

See Davmail/Outlook, #113

untitaker added a commit that referenced this issue Dec 8, 2014

Deal with case-insensitive servers

a30165c

See Davmail/Outlook, #113

untitaker removed the unconfirmed label Dec 8, 2014

untitaker added a commit that referenced this issue Dec 8, 2014

Deal with case-insensitive servers

4b3f96d

See Davmail/Outlook, #113

untitaker added a commit that referenced this issue Dec 8, 2014

Deal with case-insensitive servers

4c5f09e

See Davmail/Outlook, #113

untitaker added cat: server compat type: 3rd party and removed cat: server compat labels Dec 31, 2014

untitaker closed this as completed Jan 1, 2015

untitaker added a commit that referenced this issue Jan 8, 2015

Deal with case-insensitive servers

eac992d

See Davmail/Outlook, #113

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

warning for skipped hrefs leading to unhanded exception #113

warning for skipped hrefs leading to unhanded exception #113

jludwig commented Sep 8, 2014

untitaker commented Sep 9, 2014

jludwig commented Sep 12, 2014

untitaker commented Sep 12, 2014

untitaker commented Sep 20, 2014

untitaker commented Sep 25, 2014

untitaker commented Sep 25, 2014

dmfs commented Sep 25, 2014

untitaker commented Sep 26, 2014

untitaker commented Sep 26, 2014

untitaker commented Sep 26, 2014

untitaker commented Sep 26, 2014

jludwig commented Sep 26, 2014

untitaker commented Jan 1, 2015

warning for skipped hrefs leading to unhanded exception #113

warning for skipped hrefs leading to unhanded exception #113

Comments

jludwig commented Sep 8, 2014

untitaker commented Sep 9, 2014

jludwig commented Sep 12, 2014

untitaker commented Sep 12, 2014

untitaker commented Sep 20, 2014

untitaker commented Sep 25, 2014

untitaker commented Sep 25, 2014

dmfs commented Sep 25, 2014

untitaker commented Sep 26, 2014

untitaker commented Sep 26, 2014

untitaker commented Sep 26, 2014

untitaker commented Sep 26, 2014

jludwig commented Sep 26, 2014

untitaker commented Jan 1, 2015