Pre-release v1.5 fixes

CHANGELOG.md

@@ -8,25 +8,43 @@ This project should adhere to [Semantic Versioning](https://semver.org/spec/v2.0
 
 ## [2.0.0] - [Unreleased]
 
-In version 2.0, the requests library will be replaced with niquests or httpx.  See https://github.com/python-caldav/caldav/issues/457.  Master branch is currently running niquests.
+In version 2.0, the requests library will be replaced with niquests or httpx.  See https://github.com/python-caldav/caldav/issues/457.  Master branch is currently running niquests.  Work by @ArtemIsmagilov, https://github.com/python-caldav/caldav/pull/455
 
-In version 2.0, support for python 3.7 will be dropped, possibly also python 3.8.  Master branch supports both as for now.
+In version 2.0, support for python 3.7 and python 3.8 will be officially dropped.  Master branch *should* support both as for now, but Python 3.7 is no longer tested.
 
 ## [1.5.0] - [Unreleased]
 
 ### Deprecated
 
-Python 3.7 is no longer tested - but it should work.  Please file a bug report if it doesn't work.  (Note that the caldav library pulls in many dependencies, and not all of them supports dead snakes).
+Python 3.7 is no longer tested (dependency problems) - but it should work.  Please file a bug report if it doesn't work.  (Note that the caldav library pulls in many dependencies, and not all of them supports dead snakes).
 
-### Changed
+### Fixed
 
+* Servers that return a quoted URL in their path will now be parsed correctly by @edel-macias-cubix in https://github.com/python-caldav/caldav/pull/473
+* Compatibility workaround: If `event.load()` fails, it will retry the load by doing a multiget - https://github.com/python-caldav/caldav/pull/460 and  https://github.com/python-caldav/caldav/pull/475 - https://github.com/python-caldav/caldav/issues/459
+* Compatibility workaround: A problem with a wiki calendar fixed by @soundstorm in https://github.com/python-caldav/caldav/pull/469
+* Blank passwords should be acceptable - https://github.com/python-caldav/caldav/pull/481
+* Compatibility workaround: Accept XML content from calendar server even if it's marked up with content-type text/plain by @niccokunzmann in https://github.com/python-caldav/caldav/pull/465
+* Bugfix for saving component failing on multi-component recurrence objects - https://github.com/python-caldav/caldav/pull/467
 * Some exotic servers may return object URLs on search, but it does not work out to fetch the calendar data.  Now it will log an error instead of raising an error in such cases.
+* Some workarounds and fixes for getting tests passing on all the test servers I had at hand in https://github.com/python-caldav/caldav/pull/492
+
+### Changed
+
 * The `tests/compatibility_issues.py` has been moved to `caldav/compatibility_hints.py`, this to make it available for a caldav-server-tester-tool that I'm splitting off to a separate project/repository, and also to make https://github.com/python-caldav/caldav/issues/402 possible.
 
 #### Refactoring
 
 * Minor code cleanups by github user @ArtemIsmagilov in https://github.com/python-caldav/caldav/pull/456
-* The very much overgrown `objects.py`-file has been split into three.
+* The very much overgrown `objects.py`-file has been split into three - https://github.com/python-caldav/caldav/pull/483
+* Refactor compatibility issues by @tobixen in https://github.com/python-caldav/caldav/pull/484
+* Refactoring of `multiget` in https://github.com/python-caldav/caldav/pull/492
+
+### Documentation
+
+* Add more project links to PyPI by @niccokunzmann in https://github.com/python-caldav/caldav/pull/464
+* Document how to use tox for testing by @niccokunzmann in https://github.com/python-caldav/caldav/pull/466
+* Readthedocs integration has been repaired (https://github.com/python-caldav/caldav/pull/453 - but eventually the fix was introduced directly in the master branch)
 
 #### Test framework
 
@@ -39,20 +57,18 @@ Python 3.7 is no longer tested - but it should work.  Please file a bug report i
     * Allows offline testing of my upcoming `check_server_compatibility`-script
   * Also added the possibility to tag test servers with a name
 * Many changes done to the compatibility flag list (due to work on the server-checker project)
+* Functional tests for multiget in https://github.com/python-caldav/caldav/pull/489
 
 ### Added
 
+* support easy search for journals by @tobixen in https://github.com/python-caldav/caldav/pull/486
+* Methods for verifying and adding reverse relations - https://github.com/python-caldav/caldav/pull/336
+* Easy creation of events and tasks with alarms, search for alarms - https://github.com/python-caldav/caldav/pull/221
 * Work in progress: `auto_conn`, `auto_calendar` and `auto_calendars` may read caldav connection and calendar configuration from a config file, environmental variables or other sources.  Currently I've made the minimal possible work to be able to test the caldav-server-tester script.
 * By now `calendar.search(..., sort_keys=("DTSTART")` will work.  Sort keys expects a list or a tuple, but it's easy to send an attribute by mistake.  https://github.com/python-caldav/caldav/issues/448 https://github.com/python-caldav/caldav/pull/449
-* Compatibility workaround: If `event.load()` fails, it will retry the load by doing a multiget - https://github.com/python-caldav/caldav/pull/475 - https://github.com/python-caldav/caldav/issues/459
 * The `class_`-parameter now works when sending data to `save_event()` etc.
 * Search method now takes parameter `journal=True`.  ref https://github.com/python-caldav/caldav/issues/237 and https://github.com/python-caldav/caldav/pull/486
 
-### Fixed
-
-* Bugfix for saving component failing on multi-component recurrence objects - https://github.com/python-caldav/caldav/pull/467
-* Readthedocs integration has been repaired (https://github.com/python-caldav/caldav/pull/453 - but eventually the fix was introduced directly in the master branch)
-
 
 ## [1.4.0] - 2024-11-05
 
@@ -86,8 +102,6 @@ Python 3.7 is no longer tested - but it should work.  Please file a bug report i
 
 ### Added
 
-* Methods for verifying and adding reverse relations - https://github.com/python-caldav/caldav/pull/336
-* Easy creation of events and tasks with alarms, search for alarms - https://github.com/python-caldav/caldav/pull/221
 * Allow to reverse the sorting order on search function  by @twissell- in https://github.com/python-caldav/caldav/pull/433
 * Work on integrating typing information. Details in https://github.com/python-caldav/caldav/pull/358
 * Remove dependency on pytz. Details in https://github.com/python-caldav/caldav/issues/231 and https://github.com/python-caldav/caldav/pull/363

caldav/calendarobjectresource.py

@@ -61,7 +61,9 @@
     from typing import Self
 
 from .davobject import DAVObject
-from .elements import cdav, dav
+from .elements.cdav import CalendarData
+from .elements import cdav
+from .elements import dav
 from .lib import error
 from .lib import vcal
 from .lib.error import errmsg
@@ -396,11 +398,6 @@ def fix_reverse_relations(self, pdb: bool = False) -> list:
         """
         return self._handle_reverse_relations(verify=True, fix=True, pdb=pdb)
 
-    ## TODO: fix this (and consolidate with _handle_relations / set_relation?)
-    # def ensure_reverse_relations(self):
-    #     missing_relations = self.check_reverse_relations()
-    #     ...
-
     def _get_icalendar_component(self, assert_one=False):
         """Returns the icalendar subcomponent - which should be an
         Event, Journal, Todo or FreeBusy from the icalendar class
@@ -606,10 +603,10 @@ def load(self, only_if_unloaded: bool = False) -> Self:
 
         try:
             r = self.client.request(str(self.url))
+            if r.status == 404:
+                raise error.NotFoundError(errmsg(r))
         except:
-            return self.load_by_multiget()
-        if r.status == 404:
-            raise error.NotFoundError(errmsg(r))
+            self.load_by_multiget()
         self.data = vcal.fix(r.raw)
         if "Etag" in r.headers:
             self.props[dav.GetEtag.tag] = r.headers["Etag"]
@@ -623,15 +620,17 @@ def load_by_multiget(self) -> Self:
         with a multiget query
         """
         error.assert_(self.url)
-        href = self.url.path
-        prop = dav.Prop() + CalendarData()
-        root = cdav.CalendarMultiGet() + prop + dav.Href(value=href)
-        response = self.parent._query(root, 1, "report")
-        results = response.expand_simple_props([CalendarData()])
-        error.assert_(len(results) == 1)
-        data = results[href][CalendarData.tag]
-        error.assert_(data)
-        self.data = data
+        mydata = self.parent._multiget(event_urls=[self.url], raise_notfound=True)
+        try:
+            url, self.data = next(mydata)
+        except StopIteration:
+            ## We shouldn't come here.  Something is wrong.
+            ## TODO: research it
+            ## As of 2025-05-20, this code section is used by
+            ## TestForServerECloud::testCreateOverwriteDeleteEvent
+            raise error.NotFoundError(self.url)
+        assert_(self.data)
+        assert_(next(mydata, None) is None)
         return self
 
     ## TODO: self.id should either always be available or never

caldav/collection.py

@@ -59,7 +59,9 @@
 from .calendarobjectresource import Journal
 from .calendarobjectresource import Todo
 from .davobject import DAVObject
-from .elements import cdav, dav
+from .elements.cdav import CalendarData
+from .elements import cdav
+from .elements import dav
 from .lib import error
 from .lib import vcal
 from .lib.python_utilities import to_wire
@@ -558,15 +560,14 @@ def save(self):
             self._create(id=self.id, name=self.name, **self.extra_init_options)
         return self
 
-    ## TODO: this is missing test code.
-    ## TODO: needs refactoring:
-    ## Objects found may be Todo and Journal, not only Event.
-    ## Replace the last lines with _request_report_build_resultlist method
-    def calendar_multiget(self, event_urls: Iterable[URL]) -> List[_CC]:
+    # def data2object_class
+
+    def _multiget(
+        self, event_urls: Iterable[URL], raise_notfound: bool = False
+    ) -> Iterable[str]:
         """
-        get multiple events' data
-        @author mtorange@gmail.com
-        @type events list of Event
+        get multiple events' data.
+        TODO: Does it overlap the _request_report_build_resultlist method
         """
         if self.url is None:
             raise ValueError("Unexpected value None for self.url")
@@ -580,16 +581,40 @@ def calendar_multiget(self, event_urls: Iterable[URL]) -> List[_CC]:
         )
         response = self._query(root, 1, "report")
         results = response.expand_simple_props([cdav.CalendarData()])
-        rv = [
-            Event(
+        if raise_notfound:
+            for href in response.statuses:
+                status = response.statuses[href]
+                if "404" in status:
+                    raise error.NotFoundError(f"Status {status} in {href}")
+        for r in results:
+            yield (r, results[r][cdav.CalendarData.tag])
+
+    ## Replace the last lines with
+    def multiget(
+        self, event_urls: Iterable[URL], raise_notfound: bool = False
+    ) -> Iterable[_CC]:
+        """
+        get multiple events' data
+        TODO: Does it overlap the _request_report_build_resultlist method?
+        @author mtorange@gmail.com (refactored by Tobias)
+        """
+        results = self._multiget(event_urls, raise_notfound=raise_notfound)
+        for url, data in results:
+            yield self._calendar_comp_class_by_data(data)(
                 self.client,
-                url=self.url.join(r),
-                data=results[r][cdav.CalendarData.tag],
+                url=self.url.join(url),
+                data=data,
                 parent=self,
             )
-            for r in results
-        ]
-        return rv
+
+    def calendar_multiget(self, *largs, **kwargs):
+        """
+        get multiple events' data
+        @author mtorange@gmail.com
+        (refactored by Tobias)
+        This is for backward compatibility.  It may be removed in 3.0 or later release.
+        """
+        return list(self.multiget(*largs, **kwargs))
 
     ## TODO: Upgrade the warning to an error (and perhaps critical) in future
     ## releases, and then finally remove this method completely.

caldav/compatibility_hints.py

@@ -77,6 +77,9 @@
     'no_scheduling_mailbox':
         """Parts of RFC6833 is supported, but not the existence of inbox/mailbox""",
 
+    'no_scheduling_calendar_user_address_set':
+        """Parts of RFC6833 is supported, but not getting the calendar users addresses""",
+
     'no_default_calendar':
         """The given user starts without an assigned default calendar """
         """(or without pre-defined calendars at all)""",
@@ -268,6 +271,9 @@
     'no_search':
         """Apparently the calendar server does not support search at all (this often implies that 'object_by_uid_is_broken' has to be set as well)""",
 
+    'no_search_openended':
+        """An open-ended search will not work""",
+
     'no_events_and_tasks_on_same_calendar':
         """Zimbra has the concept of task lists ... a calendar must either be a calendar with only events, or it can be a task list, but those must never be mixed"""
 }
@@ -419,10 +425,11 @@
     'fragile_sync_tokens', ## no issue raised yet
     'vtodo_datesearch_nodtstart_task_is_skipped', ## no issue raised yet
     'broken_expand_on_exceptions', ## no issue raised yet
-    'date_todo_search_ignores_duration'
+    'date_todo_search_ignores_duration',
     'calendar_color',
     'calendar_order',
-    'vtodo_datesearch_notime_task_is_skipped'
+    'vtodo_datesearch_notime_task_is_skipped',
+    "no_alarmsearch",
 ]
 
 google = [
@@ -448,6 +455,9 @@
 
 nextcloud = [
     'date_search_ignores_duration',
+    'unique_calendar_ids',
+    'broken_expand',
+    'no_delete_calendar',
     'sync_breaks_on_delete',
     'no_recurring_todo',
     'combined_search_not_working',
@@ -488,7 +498,8 @@
     'text_search_not_working',
     'no_relships',
     'isnotdefined_not_working',
-    'robur_rrule_freq_yearly_expands_monthly'
+    'no_alarmsearch',
+    'broken_expand',
 ]
 
 posteo = [
@@ -498,6 +509,7 @@
     'no_recurring_todo',
     'no_sync_token',
     'combined_search_not_working',
+    'no_alarmsearch',
     'broken_expand',
 ]
 
@@ -527,7 +539,22 @@
 
     ## Purelymail claims that the search indexes are "lazily" populated,
     ## so search works some minutes after the event was created/edited.
-    'search_delay'
+    'search_delay',
+
+    ## I haven't raised this one with them yet
+    'no_alarmsearch',
+]
+
+gmx = [
+    "no_scheduling_mailbox",
+    "no_mkcalendar",
+    "search_needs_comptype",
+    #"text_search_is_case_insensitive",
+    "no_freebusy_rfc4791",
+    "no_expand",
+    "no_search_openended",
+    "no_sync_token",
+    "no_scheduling_calendar_user_address_set",
 ]
 
 # fmt: on

caldav/davclient.py

@@ -283,6 +283,7 @@ def find_objects_and_props(self) -> Dict[str, Dict[str, _Element]]:
         self.sync_token will be populated if found, self.objects will be populated.
         """
         self.objects: Dict[str, Dict[str, _Element]] = {}
+        self.statuses: Dict[str, str] = {}
 
         if "Schedule-Tag" in self.headers:
             self.schedule_tag = self.headers["Schedule-Tag"]
@@ -300,6 +301,7 @@ def find_objects_and_props(self) -> Dict[str, Dict[str, _Element]]:
             ## but then there was https://github.com/python-caldav/caldav/issues/136
             if href not in self.objects:
                 self.objects[href] = {}
+                self.statuses[href] = status
 
             ## The properties may be delivered either in one
             ## propstat with multiple props or in multiple
@@ -358,7 +360,7 @@ def _expand_simple_prop(
             error.assert_(len(values) == 1)
             return values[0]
 
-    ## TODO: "expand" does not feel quite right.
+    ## TODO: word "expand" does not feel quite right.
     def expand_simple_props(
         self,
         props: Iterable[BaseElement] = None,
@@ -897,6 +899,7 @@ def auto_conn(
 
             try:
                 idx = int(name)
+                name = None
             except ValueError:
                 idx = None
             try:

caldav/davobject.py

@@ -216,11 +216,11 @@ def _query(
             body = to_wire(body)
             if (
                 ret.status == 500
-                and b"getetag" not in body
-                and b"<C:calendar-data/>" in body
+                and b"D:getetag" not in body
+                and b"<C:calendar-data" in body
             ):
                 body = body.replace(
-                    b"<C:calendar-data/>", b"<D:getetag/><C:calendar-data/>"
+                    b"<C:calendar-data", b"<D:getetag/><C:calendar-data"
                 )
                 return self._query(
                     body, depth, query_method, url, expected_return_value
@@ -298,7 +298,7 @@ def get_properties(
                 ## ... but it gets worse ... when doing a propfind on the
                 ## principal, the href returned may be without the slash.
                 ## Such inconsistency is clearly a bug.
-                log.error(
+                log.warning(
                     "potential path handling problem with ending slashes.  Path given: %s, path found: %s.  %s"
                     % (path, exchange_path, error.ERR_FRAGMENT)
                 )
@@ -323,15 +323,15 @@ def get_properties(
             ## Ref https://github.com/python-caldav/caldav/issues/191 ...
             ## let's be pragmatic and just accept whatever the server is
             ## throwing at us.  But we'll log an error anyway.
-            log.error(
+            log.warning(
                 "Possibly the server has a path handling problem, possibly the URL configured is wrong.\n"
                 "Path expected: %s, path found: %s %s.\n"
                 "Continuing, probably everything will be fine"
                 % (path, str(list(properties)), error.ERR_FRAGMENT)
             )
             rc = list(properties.values())[0]
         else:
-            log.error(
+            log.warning(
                 "Possibly the server has a path handling problem.  Path expected: %s, paths found: %s %s"
                 % (path, str(list(properties)), error.ERR_FRAGMENT)
             )

caldav/lib/error.py

@@ -62,7 +62,7 @@ def assert_(condition: object) -> None:
             raise
 
 
-ERR_FRAGMENT: str = "Please consider raising an issue at https://github.com/python-caldav/caldav/issues or reach out to t-caldav@tobixen.no, include this error and the traceback and tell what server you are using"
+ERR_FRAGMENT: str = "Please consider raising an issue at https://github.com/python-caldav/caldav/issues or reach out to t-caldav@tobixen.no, include this error and the traceback (if any) and tell what server you are using"
 
 
 class DAVError(Exception):