507 error during collection sync

[siderai]

Calling calendar_principal.calendar(name=calendar_name).objects(load_objects=False) for google calendar and yandex calendar somehow raises the following error: caldav.lib.error.ResponseError: ResponseError at 'HTTP/1.1 507 Insufficient Storage', reason no reason. Some accounts or particular calendars raise this error consistently, others do not at all, and I don't understand why.

I haven't found any documented solutions neither at the platforms nor within the library.

I get that 5** status code indicates a server-side problem, but is there anything I can do client-side to prevent this from happening?

Any ideas on how to handle it?

[tobixen]

I'm intending to create some frameworks for letting the library work around known incompatibility-issues.

There are different pathways to fetching all objects on a calendar, if I remember correct the .objects will do a sync-token request. At the other hand you have .events() and .todos() which will search for events and tasks. If the first doesn't work out, then try the latter.

[siderai]

objects() returns a SynchronizableCalendarObjectCollection, which allows you to get periodic updates on events without storing them locally (sync() method). So querying by sync token is preferable. I don't need to get all the events in the query, but only a snapshot and then updates later on.

I like this mechanism, but I'm concerned about an incompatibility occurring here and there.

Where do you think the root of the problem is: in the library implementation or in google services?
In the first case we can think of a workaround, but in the second case I can only put up with it or use another version of google calendar API.

[tobixen]

I'm not quite sure where the problem is. It's been a long time since I tested the library against Google, and there is nothing in my notes that it didn't support sync token. There is also a link in the documentation towards https://developers.google.com/calendar/caldav/v2/guide - and on that link it says clearly that rfc6578 should be supported - and that is the sync-token standard. So it seems like it should work, I don't know what could be wrong.

I do have a Google account, so theoretically I should be able to check this up myself - but I don't think I will have any time this week nor the next.

[tobixen]

Is this still an issue?

[tobixen]

I will look into Google when I'm working on #513. Procrastinating this issue a bit longer.

[tobixen]

I don't know if I will have time to look into Google before the next release, but this issue is almost certainly fixed already - recently a workaround was fixed so that any sync-operations on servers not supporting sync-tokens will transparently work by fetching the full calendar on every sync. As for now, this workaround may require explicitly configuring that the server does not support sync-tokens - I haven't written test code for the automatic fallback yet.

Here is an issue analysis by Claude, with some corrections by me (the analysis below asserted that the fix was released in 2.0 - in reality it's just merged into master, and will be released in 2.2 hopefully this week):

Issue #340 Analysis: 507 Error During Collection Sync

Date: 2025-12-03
Issue URL: #340
Status: Open (Milestone v2.0)
Original Report: October 31, 2023

Issue Summary

Users reported receiving HTTP 507 "Insufficient Storage" errors when calling calendar.objects(load_objects=False) on Google Calendar and Yandex Calendar. The issue was inconsistent - affecting some accounts consistently while never affecting others.

The .objects() method uses sync-token requests (RFC 6578), and the 507 error suggests the server is refusing to process the sync-collection REPORT request.

---

Is it still relevant?

NO - Likely RESOLVED by the transparent fallback mechanism recently implemented and scheduled for release in version 2.2.

---

What Changed Since the Issue Was Reported

Unreleased Changes (from CHANGELOG.md)

The CHANGELOG explicitly documents that this issue should now be resolved:

Line 27-28:

When using the sync-token API now on servers not supporting sync-token, the full calendar will be fetched on every sync. The change is intended to be un-breaking, but for people having very big calendars and syncing them to a mobile device with limited memory, bandwidth, CPU and battery, this change may be painful.

Line 31:

Transparent handling of calendar servers not supporting sync-tokens. The API will yield the same result, albeit with more bandwidth and memory consumption.

Current Implementation (collection.py:1143-1255)

The objects_by_sync_token() method now includes:

1. Pre-flight check (lines 1169-1180):

   • Checks if server supports sync-token feature
   • If sync-token feature is marked "unsupported", skips sync-token attempt entirely
   • Detects "fake" tokens (starting with "fake-") and uses fallback immediately

2. Try-catch with fallback (lines 1182-1216):

   try:
       # Attempt sync-collection REPORT
       # ...
   except (error.ReportError, error.DAVError) as e:
       ## Server doesn't support sync tokens or the sync-collection REPORT failed
       log.info(
           f"Sync-collection REPORT failed ({e}), falling back to full retrieval"
       )
       ## Fall through to fallback implementation

3. Fallback implementation (lines 1218-1255):

   • Uses self.search() to retrieve all objects
   • Generates a "fake" sync token based on current state
   • Returns same interface (SynchronizableCalendarObjectCollection)
   • Transparent to user - same API, same return type

How This Fixes Issue #340

The 507 error is now caught and handled:

1. Server returns 507 "Insufficient Storage"
2. HTTP response with status >= 300 triggers ReportError (davclient.py:736-739):

   if response.status >= 300:
       raise error.ReportError(
           f"{response.status} {response.reason} - {response.raw}"
       )

3. Exception is caught by except (error.ReportError, error.DAVError) block
4. Fallback to self.search() is used instead
5. User gets the same result, just via a different mechanism

User perspective:

• Before: Crashed with 507 error
• After: Works transparently, just uses more bandwidth/memory

---

Why the 507 Error Occurred

HTTP 507 "Insufficient Storage" is an unusual error for sync-collection. Possible reasons:

1. Server bug - Google/Yandex incorrectly returns 507 instead of 501 "Not Implemented"
2. Account-specific limits - Some accounts may have quotas or limits on sync operations
3. Large calendars - Server may refuse sync-collection for very large calendars
4. Server-side issues - Temporary storage issues on server infrastructure

The inconsistency (some accounts always fail, others never fail) suggests it's related to account-specific characteristics (calendar size, account type, quota limits).

---

Current State Assessment

What exists now:

1. ✅ Transparent fallback - Automatically falls back when sync-collection fails
2. ✅ Feature detection - Remembers when server doesn't support sync-tokens
3. ✅ Fake token system - Emulates sync-token behavior for servers without support
4. ✅ Error handling - Catches all DAVError and ReportError exceptions
5. ✅ Logging - Logs when fallback is used (at INFO level)

What's NOT done:

1. ❌ No specific handling for 507 - Treated same as any other error
2. ❌ No server-specific workarounds - Could detect Google/Yandex and skip sync-token preemptively
3. ❌ No caldav-server-tester check - Could test for 507 error specifically
4. ❌ Issue not closed - Still in milestone v2.0, but should be resolved

---

Testing the Fix

To verify this issue is resolved, someone with access to an account that previously exhibited the 507 error should test:

from caldav import DAVClient

client = DAVClient(url="...", username="...", password="...")
calendar = client.principal().calendar(name="...")

# This should now work without raising an exception
objects = calendar.objects(load_objects=False)

# Should complete successfully, albeit possibly slower than sync-token
for obj in objects:
    print(obj.url)

Expected behavior:

• No exception raised
• Log message at INFO level: "Sync-collection REPORT failed (...), falling back to full retrieval"
• Objects retrieved successfully via search() instead of sync-collection

---

Recommended Actions

Option 1: Close the Issue (Recommended)

Rationale:

• The transparent fallback mechanism should handle this
• The implementation catches all errors >= 300, including 507
• User gets same functionality, just via different mechanism
• CHANGELOG explicitly mentions this was addressed

Steps:

1. Add comment explaining that issue should be resolved by v2.2 fallback mechanism
2. Ask original reporter to test with latest version
3. If confirmed working, close issue
4. If still failing, investigate why fallback isn't working

Option 2: Add Server-Specific Workaround (If Still Needed)

If testing shows the issue still occurs (which would be surprising), add Google/Yandex to compatibility hints:

# In compatibility_hints.py
FEATURES = {
    "google": {
        "sync-token": {"support": "unsupported", "behaviour": "Returns 507 error"}
    },
    "yandex": {
        "sync-token": {"support": "unsupported", "behaviour": "Returns 507 error"}
    }
}

This would prevent even attempting sync-collection for these servers.

Option 3: Add caldav-server-tester Check (Nice to Have)

Add a check specifically for 507 error during sync-collection:

class CheckSyncToken(Check):
    def _run_check(self):
        try:
            objects = self.calendar.objects(load_objects=False)
            self.set_feature("sync-token")
        except error.ReportError as e:
            if "507" in str(e):
                self.set_feature("sync-token", {
                    "support": "unsupported",
                    "behaviour": "Server returns 507 Insufficient Storage"
                })
            else:
                self.set_feature("sync-token", "broken")

However, this check may not be necessary since the fallback is transparent.

---

Analysis of Code Flow

Before v2.1 (Issue occurred):

1. User calls calendar.objects(load_objects=False)
2. Method attempts sync-collection REPORT
3. Server returns 507 error
4. ReportError raised
5. Exception propagates to user ❌

After v2.2 (Issue should be resolved):

1. User calls calendar.objects(load_objects=False)
2. Method checks if sync-token is known to be unsupported (no, first time)
3. Method attempts sync-collection REPORT
4. Server returns 507 error
5. ReportError raised
6. Exception caught by try/except ✅
7. Logs: "Sync-collection REPORT failed (507 Insufficient Storage - ...), falling back to full retrieval"
8. Falls through to fallback implementation
9. Uses self.search() to get all objects
10. Generates fake sync token
11. Returns SynchronizableCalendarObjectCollection
12. User gets same result ✅

Future calls (after feature detection):

If the client supports feature detection and the 507 is consistently returned:

1. Feature detection marks sync-token as "unsupported" or "broken"
2. Future calls skip sync-collection entirely
3. Goes directly to fallback (step 9 above)
4. No wasted REPORT request

---

Related Code Locations

• /home/tobias/caldav/caldav/collection.py:1143-1255 - objects_by_sync_token() with fallback
• /home/tobias/caldav/caldav/collection.py:1211-1216 - Exception catching and fallback trigger
• /home/tobias/caldav/caldav/davclient.py:736-739 - Where ReportError is raised for status >= 300
• /home/tobias/caldav/caldav/lib/error.py:108-109 - ReportError class definition
• /home/tobias/caldav/CHANGELOG.md:27-31 - Documentation of the fix

---

Conclusion

This issue is almost certainly RESOLVED by the transparent fallback mechanism implemented in version 2.0.

The implementation now:

1. ✅ Catches the 507 error (via except (error.ReportError, error.DAVError))
2. ✅ Falls back to full calendar retrieval via search()
3. ✅ Returns same interface to user
4. ✅ Logs the fallback for debugging

Recommended action:

1. Add a comment to the issue explaining the v2.2 changes
2. Ask the original reporter to test with caldav >= 2.2
3. Close the issue if confirmed working
4. Only reopen if testing shows the issue still occurs

The transparent fallback means users no longer need to worry about server-specific sync-token issues. The API "just works," albeit with higher bandwidth usage for servers that don't properly support sync-tokens or return errors like 507.

---

Open Questions

1. Will anyone be testing this with Google/Yandex Calendar after v2.2 is released?
2. Should we add explicit 507 handling with a more specific error message?
3. Should we document the fallback behavior in user-facing documentation?
4. Should we add feature detection to automatically mark Google/Yandex as "sync-token: unsupported"?

None of these questions block closing the issue - they're just potential improvements.