Dec release

[robbrad]

Summary by CodeRabbit

• New Features

  • Added support for Blackpool, Harlow, and Isle of Anglesey councils.
  • Enhanced UPRN handling and validation across multiple councils.

• Bug Fixes

  • Improved parsing logic for Fife, Merton, Mid Sussex, North Hertfordshire, Rushmoor, Birmingham, Chelmsford, and other councils.
  • Fixed address lookup and collection date extraction issues.

• Documentation

  • Updated council documentation with new parameters and lookup instructions.

• Chores

  • Updated GitHub Actions workflows and dependencies.
  • Refined test data configurations.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Warning

Rate limit exceeded

@robbrad has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 16 minutes and 5 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between 108d4ae and 129d79d.

📒 Files selected for processing (7)

• .github/workflows/release.yml (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/BlackpoolCouncil.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/FarehamBoroughCouncil.py (3 hunks)
• uk_bin_collection/uk_bin_collection/councils/HarlowCouncil.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/MidSussexDistrictCouncil.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/SouthHollandDistrictCouncil.py (2 hunks)
• uk_bin_collection/uk_bin_collection/councils/SouthLanarkshireCouncil.py (2 hunks)

Walkthrough

This PR updates GitHub Actions workflows to use actions/checkout@v6, adds three new council scrapers (Blackpool, Harlow, Isle of Anglesey), refactors existing council implementations from Selenium/API to HTTP-based approaches or vice versa, updates start dates for bin collection schedules, adds test data entries, and updates configuration files and wiki documentation.

Changes

Cohort / File(s)	Summary
GitHub Actions Workflow Updates .github/workflows/behave_pull_request.yml, behave_schedule.yml, bump.yml, codeql-analysis.yml, docker-image.yml, ha_compatibility_test.yml, hacs_validation.yml, lint.yml, release.yml, rollback-release.yml, validate-release-ready.yml, wiki.yml	Updated actions/checkout@v5 to actions/checkout@v6 across all jobs and workflows.
New Council Scrapers uk_bin_collection/.../BlackpoolCouncil.py, HarlowCouncil.py, IsleOfAngleseyCouncil.py	Added three new council implementations with UPRN-based or postcode-based lookups and API/HTTP parsing for collection data.
Selenium-to-HTTP Refactors uk_bin_collection/.../MidSussexDistrictCouncil.py, NorthHertfordshireDistrictCouncil.py	Replaced Selenium-driven flows with requests/BeautifulSoup or Cloud9 mobile API-based approaches and UPRN validation.
API/Approach Migrations uk_bin_collection/.../FifeCouncil.py, MertonCouncil.py, WiltshireCouncil.py	Updated to alternative implementation approaches: FifeCouncil switched to Selenium-driven interactive workflow; MertonCouncil to FixMyStreet polling; WiltshireCouncil to events-list extraction.
HTML Parsing & Selector Updates uk_bin_collection/.../BirminghamCityCouncil.py, ChelmsfordCityCouncil.py, NorwichCityCouncil.py, WestOxfordshireDistrictCouncil.py, WinchesterCityCouncil.py	Modified HTML selectors, table/element targeting, and parsing logic to adapt to site structure changes or improve robustness.
Date Constant & Logic Updates uk_bin_collection/.../ArgyllandButeCouncil.py, EdinburghCityCouncil.py, FarehamBoroughCouncil.py, NorthumberlandCouncil.py, RushmoorCouncil.py, SouthGloucestershireCouncil.py, SouthLanarkshireCouncil.py, SouthamptonCityCouncil.py, ThurrockCouncil.py	Updated start dates to 2025, added UPRN normalization, refined collection date logic, and improved parsing guards.
Minor Header/Request Updates uk_bin_collection/.../ArmaghBanbridgeCraigavonCouncil.py, HaltonBoroughCouncil.py, KingsLynnandWestNorfolkBC.py, LondonBoroughLambeth.py	Added user-agent headers, refined Bin_Type logic, and improved request headers for compatibility.
Configuration & Dependency Updates pyproject.toml, uk_bin_collection_api_server/requirements.txt	Added new version file entry in pyproject.toml; updated Connexion and uk-bin-collection to specify version constraints in requirements.txt.
Test Data & Documentation uk_bin_collection/tests/input.json, wiki/Councils.md	Added new council entries (Blackpool, Harlow, Isle of Anglesey) and updated existing entries with new parameters and configuration; updated wiki with new council sections and parameter guidance.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Areas requiring extra attention:

• IsleOfAngleseyCouncil.py — New implementation with multi-step session management, address lookup, and error handling across auth, resolution, and schedule-fetch flows; careful review of JSON parsing, API error diagnostics, and null-safety checks needed.
• MidSussexDistrictCouncil.py, NorthHertfordshireDistrictCouncil.py — Major refactors replacing Selenium with HTTP/API; verify request flows, error handling paths, and data extraction logic match expected output format.
• FifeCouncil.py — Significant control-flow change from API-based to Selenium-driven with webdriver lifecycle management; ensure driver cleanup and WebDriverWait patterns are correct.
• MertonCouncil.py, WiltshireCouncil.py — Substantial parsing logic changes; verify new HTML structure expectations and date parsing/year-rollover logic.
• uk_bin_collection/tests/input.json — Large dataset changes across multiple councils with new configuration patterns (UPRN vs. postcode/house_number, Selenium vs. API); cross-reference with corresponding council implementation changes.
• Heterogeneous council updates — 25+ files with varied change types (dates, selectors, logic, headers) require individual reasoning per file rather than pattern-based verification.

Possibly related issues

• Change to Mid Sussex Council Website #1721 — Addresses Mid Sussex Selenium timeout failures by replacing the Selenium-driven implementation with a requests/BeautifulSoup workflow, directly solving the reported issue.

Possibly related PRs

• fix: Council Fix Pack - December 2025 #1705 — Introduces overlapping changes to the same council scrapers (Blackpool, Harlow, Fife, Rushmoor, Mid Sussex, etc.) and test input data, with identical modifications across multiple files.
• feat: add direct mobile API support to North Hertfordshire District Council scraper #1715 — Replaces NorthHertfordshireDistrictCouncil Selenium flow with Cloud9 mobile API UPRN-based implementation, matching the lookup_uprn and fetch_mobile_api additions in this PR.
• Isle of Anglesey support #1729 — Introduces IsleOfAngleseyCouncil support with the same council class implementation and test input entry additions.

Suggested reviewers

• dp247

---

🐰 With workflows upgraded and councils refactored,
New parsers emerge, old Selenium detracted,
UPRN lookups and API calls aligned,
The bin collection data: precise and refined! ✨🗑️

Pre-merge checks and finishing touches

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.
Title check	❓ Inconclusive	The title 'Dec release' is vague and generic, using non-descriptive terminology that does not convey meaningful information about the substantial changes in this pull request.	Replace with a specific title describing the main changes, such as 'Update GitHub Actions checkout to v6 and enhance council data parsers' or 'Upgrade GitHub Actions and refactor bin collection council modules'.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov]

❌ 7 Tests Failed:

Tests completed	Failed	Passed	Skipped
27	7	20	0

View the full list of 7 ❄️ flaky test(s)

uk_bin_collection.tests.step_defs.test_validate_council::test_scenario_outline[HaltonBoroughCouncil]

Flake rate in main: 100.00% (Passed 0 times, Failed 227 times)

Stack Traces | 45.4s run time

fixturefunc = <function scrape_step at 0x7f1341c760c0>
request = <FixtureRequest for <Function test_scenario_outline[HaltonBoroughCouncil]>>
kwargs = {'context': <test_validate_council.Context object at 0x7f1343376150>, 'headless_mode': 'True', 'local_browser': 'False', 'selenium_url': 'http://localhost:4444'}

    def call_fixture_func(
        fixturefunc: _FixtureFunc[FixtureValue], request: FixtureRequest, kwargs
    ) -> FixtureValue:
        if is_generator(fixturefunc):
            fixturefunc = cast(
                Callable[..., Generator[FixtureValue, None, None]], fixturefunc
            )
            generator = fixturefunc(**kwargs)
            try:
                fixture_result = next(generator)
            except StopIteration:
                raise ValueError(f"{request.fixturename} did not yield a value") from None
            finalizer = functools.partial(_teardown_yield_fixture, fixturefunc, generator)
            request.addfinalizer(finalizer)
        else:
            fixturefunc = cast(Callable[..., FixtureValue], fixturefunc)
>           fixture_result = fixturefunc(**kwargs)

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.12/lib/python3.12.../site-packages/_pytest/fixtures.py:898: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../tests/step_defs/test_validate_council.py:101: in scrape_step
    context.parse_result = CollectData.run()
uk_bin_collection/uk_bin_collection/collect_data.py:101: in run
    return self.client_code(
uk_bin_collection/uk_bin_collection/collect_data.py:121: in client_code
    return get_bin_data_class.template_method(address_url, **kwargs)
uk_bin_collection/uk_bin_collection/get_bin_data.py:61: in template_method
    bin_data_dict = self.get_and_parse_data(this_url, **kwargs)
uk_bin_collection/uk_bin_collection/get_bin_data.py:84: in get_and_parse_data
    bin_data_dict = self.parse_data("", url=address_url, **kwargs)
.../uk_bin_collection/councils/HaltonBoroughCouncil.py:85: in parse_data
    WebDriverWait(driver, 10).until(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <selenium.webdriver.support.wait.WebDriverWait (session="b37b0da0019ce3194839f4b0d4a6aedc")>
method = <function presence_of_element_located.<locals>._predicate at 0x7f1356217f60>
message = ''

    def until(self, method: Callable[[D], Union[Literal[False], T]], message: str = "") -> T:
        """Wait until the method returns a value that is not False.
    
        Calls the method provided with the driver as an argument until the
        return value does not evaluate to ``False``.
    
        Parameters:
        -----------
        method: callable(WebDriver)
            - A callable object that takes a WebDriver instance as an argument.
    
        message: str
            - Optional message for :exc:`TimeoutException`
    
        Return:
        -------
        object: T
            - The result of the last call to `method`
    
        Raises:
        -------
        TimeoutException
            - If 'method' does not return a truthy value within the WebDriverWait
            object's timeout
    
        Example:
        --------
        >>> from selenium.webdriver.common.by import By
        >>> from selenium.webdriver.support.ui import WebDriverWait
        >>> from selenium.webdriver.support import expected_conditions as EC
    
        # Wait until an element is visible on the page
        >>> wait = WebDriverWait(driver, 10)
        >>> element = wait.until(EC.visibility_of_element_located((By.ID, "exampleId")))
        >>> print(element.text)
        """
        screen = None
        stacktrace = None
    
        end_time = time.monotonic() + self._timeout
        while True:
            try:
                value = method(self._driver)
                if value:
                    return value
            except self._ignored_exceptions as exc:
                screen = getattr(exc, "screen", None)
                stacktrace = getattr(exc, "stacktrace", None)
            if time.monotonic() > end_time:
                break
            time.sleep(self._poll)
>       raise TimeoutException(message, screen, stacktrace)
E       selenium.common.exceptions.TimeoutException: Message: 
E       Stacktrace:
E       #0 0x563a7b19baea <unknown>
E       #1 0x563a7abe7cdb <unknown>
E       #2 0x563a7ac3a6c4 <unknown>
E       #3 0x563a7ac3a901 <unknown>
E       #4 0x563a7ac898b4 <unknown>
E       #5 0x563a7ac86c87 <unknown>
E       #6 0x563a7ac2caca <unknown>
E       #7 0x563a7ac2d7d1 <unknown>
E       #8 0x563a7b162ab9 <unknown>
E       #9 0x563a7b165a8c <unknown>
E       #10 0x563a7b14bd49 <unknown>
E       #11 0x563a7b166685 <unknown>
E       #12 0x563a7b1336c3 <unknown>
E       #13 0x563a7b1887d8 <unknown>
E       #14 0x563a7b1889b3 <unknown>
E       #15 0x563a7b19aa83 <unknown>
E       #16 0x7f0a02d38aa4 <unknown>
E       #17 0x7f0a02dc5a64 __clone

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.12/lib/python3.12.../webdriver/support/wait.py:146: TimeoutException

uk_bin_collection.tests.step_defs.test_validate_council::test_scenario_outline[LondonBoroughLambeth]

Flake rate in main: 25.50% (Passed 149 times, Failed 51 times)

Stack Traces | 18.5s run time

fixturefunc = <function validate_output_step at 0x7f0a7678a200>
request = <FixtureRequest for <Function test_scenario_outline[LondonBoroughLambeth]>>
kwargs = {'context': <test_validate_council.Context object at 0x7f0a8e100dd0>}

    def call_fixture_func(
        fixturefunc: _FixtureFunc[FixtureValue], request: FixtureRequest, kwargs
    ) -> FixtureValue:
        if is_generator(fixturefunc):
            fixturefunc = cast(
                Callable[..., Generator[FixtureValue, None, None]], fixturefunc
            )
            generator = fixturefunc(**kwargs)
            try:
                fixture_result = next(generator)
            except StopIteration:
                raise ValueError(f"{request.fixturename} did not yield a value") from None
            finalizer = functools.partial(_teardown_yield_fixture, fixturefunc, generator)
            request.addfinalizer(finalizer)
        else:
            fixturefunc = cast(Callable[..., FixtureValue], fixturefunc)
>           fixture_result = fixturefunc(**kwargs)

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.12/lib/python3.12.../site-packages/_pytest/fixtures.py:898: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../tests/step_defs/test_validate_council.py:114: in validate_output_step
    assert file_handler.validate_json_schema(
.../step_defs/step_helpers/file_handler.py:40: in validate_json_schema
    validate(instance=json_data, schema=schema)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

instance = {'bins': []}
schema = {'$ref': '#/definitions/BinData', '$schema': 'http://json-schema.org/draft-06/schema#', 'definitions': {'Bin': {'addit...ems': {'$ref': '#/definitions/Bin'}, 'minItems': 1, 'type': 'array'}}, 'required': ['bins'], 'title': 'BinData', ...}}}
cls = <class 'jsonschema.validators.Draft6Validator'>, args = (), kwargs = {}
validator = Draft6Validator(schema={'$ref': '#/definitions/BinData', '$schema': 'http://json-...ft-06/schema#', 'definitions': {'B...nitions/Bin'}, 'minItems': 1, 'type': 'array'}}, 'required': ['bins'], 'title': 'BinData', ...}}}, format_checker=None)
error = <ValidationError: '[] should be non-empty'>

    def validate(instance, schema, cls=None, *args, **kwargs):  # noqa: D417
        """
        Validate an instance under the given schema.
    
            >>> validate([2, 3, 4], {"maxItems": 2})
            Traceback (most recent call last):
                ...
            ValidationError: [2, 3, 4] is too long
    
        :func:`~jsonschema.validators.validate` will first verify that the
        provided schema is itself valid, since not doing so can lead to less
        obvious error messages and fail in less obvious or consistent ways.
    
        If you know you have a valid schema already, especially
        if you intend to validate multiple instances with
        the same schema, you likely would prefer using the
        `jsonschema.protocols.Validator.validate` method directly on a
        specific validator (e.g. ``Draft202012Validator.validate``).
    
    
        Arguments:
    
            instance:
    
                The instance to validate
    
            schema:
    
                The schema to validate with
    
            cls (jsonschema.protocols.Validator):
    
                The class that will be used to validate the instance.
    
        If the ``cls`` argument is not provided, two things will happen
        in accordance with the specification. First, if the schema has a
        :kw:`$schema` keyword containing a known meta-schema [#]_ then the
        proper validator will be used. The specification recommends that
        all schemas contain :kw:`$schema` properties for this reason. If no
        :kw:`$schema` property is found, the default validator class is the
        latest released draft.
    
        Any other provided positional and keyword arguments will be passed
        on when instantiating the ``cls``.
    
        Raises:
    
            `jsonschema.exceptions.ValidationError`:
    
                if the instance is invalid
    
            `jsonschema.exceptions.SchemaError`:
    
                if the schema itself is invalid
    
        .. rubric:: Footnotes
        .. [#] known by a validator registered with
            `jsonschema.validators.validates`
    
        """
        if cls is None:
            cls = validator_for(schema)
    
        cls.check_schema(schema)
        validator = cls(schema, *args, **kwargs)
        error = exceptions.best_match(validator.iter_errors(instance))
        if error is not None:
>           raise error
E           jsonschema.exceptions.ValidationError: [] should be non-empty
E           
E           Failed validating 'minItems' in schema['properties']['bins']:
E               {'type': 'array', 'items': {'$ref': '#/definitions/Bin'}, 'minItems': 1}
E           
E           On instance['bins']:
E               []

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.12/lib/python3.12.../site-packages/jsonschema/validators.py:1332: ValidationError

uk_bin_collection.tests.step_defs.test_validate_council::test_scenario_outline[NorthHertfordshireDistrictCouncil]

Flake rate in main: 56.36% (Passed 96 times, Failed 124 times)

Stack Traces | 0.551s run time

fixturefunc = <function scrape_step at 0x7f8eef42b6a0>
request = <FixtureRequest for <Function test_scenario_outline[NorthHertfordshireDistrictCouncil]>>
kwargs = {'context': <test_validate_council.Context object at 0x7f8effdefd40>, 'headless_mode': 'True', 'local_browser': 'False', 'selenium_url': 'http://localhost:4444'}

    def call_fixture_func(
        fixturefunc: _FixtureFunc[FixtureValue], request: FixtureRequest, kwargs
    ) -> FixtureValue:
        if is_generator(fixturefunc):
            fixturefunc = cast(
                Callable[..., Generator[FixtureValue, None, None]], fixturefunc
            )
            generator = fixturefunc(**kwargs)
            try:
                fixture_result = next(generator)
            except StopIteration:
                raise ValueError(f"{request.fixturename} did not yield a value") from None
            finalizer = functools.partial(_teardown_yield_fixture, fixturefunc, generator)
            request.addfinalizer(finalizer)
        else:
            fixturefunc = cast(Callable[..., FixtureValue], fixturefunc)
>           fixture_result = fixturefunc(**kwargs)

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.12/lib/python3.12.../site-packages/_pytest/fixtures.py:898: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../tests/step_defs/test_validate_council.py:101: in scrape_step
    context.parse_result = CollectData.run()
uk_bin_collection/uk_bin_collection/collect_data.py:101: in run
    return self.client_code(
uk_bin_collection/uk_bin_collection/collect_data.py:121: in client_code
    return get_bin_data_class.template_method(address_url, **kwargs)
uk_bin_collection/uk_bin_collection/get_bin_data.py:61: in template_method
    bin_data_dict = self.get_and_parse_data(this_url, **kwargs)
uk_bin_collection/uk_bin_collection/get_bin_data.py:84: in get_and_parse_data
    bin_data_dict = self.parse_data("", url=address_url, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <NorthHertfordshireDistrictCouncil.CouncilClass object at 0x7f8eef111ca0>
page = ''
kwargs = {'council_module_str': 'NorthHertfordshireDistrictCouncil', 'dev_mode': False, 'headless': True, 'local_browser': False, ...}
bins_with_sort_date = [], uprn = '100081258147'
api_response = {'wasteCollectionDates': {'cacheKey': '1.0.959', 'container1CollectionDetails': None, 'container2CollectionDetails': None, 'container3CollectionDetails': None, ...}}
waste_collection_dates = {'cacheKey': '1.0.959', 'container1CollectionDetails': None, 'container2CollectionDetails': None, 'container3CollectionDetails': None, ...}
container_num = 8

    def parse_data(self, page: str, **kwargs) -> dict:
        """
        Parse bin collection data using the Cloud9 mobile API.
    
        Args:
            page: Unused (kept for interface compatibility)
            **kwargs: Must contain either 'uprn' or both 'postcode' and 'paon'
    
        Returns:
            dict: Bin collection data in standard format
        """
        bins_with_sort_date = []
    
        # Get UPRN either directly or via lookup
    
        uprn = kwargs.get("uprn")
    
        if uprn:
            check_uprn(uprn)
        else:
            # Try to lookup UPRN from postcode and house number
            # This is provided to maintain backward compatibility with the existing postcode/paon input method
            postcode = kwargs.get("postcode")
            paon = kwargs.get("paon")
            check_postcode(postcode)
            check_paon(paon)
    
            # Attempt UPRN lookup using postcode and paon
            uprn = lookup_uprn(postcode=postcode, paon=paon)
    
    
        # Fetch data from mobile API
        api_response = fetch_mobile_api(uprn)
    
        # Parse the API response - Cloud9 API returns WasteCollectionDates with 8 containers
        # Response structure: {"wasteCollectionDates": {"container1CollectionDetails": {...}, ...}}
        waste_collection_dates = api_response.get("wasteCollectionDates", {})
    
        if not waste_collection_dates:
            raise ValueError(
                f"No wasteCollectionDates found in API response. API response: {json.dumps(api_response)[:200]}"
            )
    
        # Process all 8 possible containers
        for container_num in range(1, MOBILE_API_NUM_CONTAINERS + 1):
            container_key = f"container{container_num}CollectionDetails"
            container = waste_collection_dates.get(container_key)
    
            if not container or not isinstance(container, dict):
                continue
    
            # Extract collection date
            collection_date_str = container.get("collectionDate", "")
    
            # Skip empty collection dates
            if not collection_date_str:
                continue
    
            # Extract container description (bin type)
            bin_type = container.get("containerDescription", f"Container {container_num}")
            try:
                collection_datetime = datetime.fromisoformat(collection_date_str)
            except ValueError:
                # skip bins with invalid date format and continue processing
                continue
    
            # Parse the date - API returns ISO format like "2025-11-25T00:00:00"
            bin_entry = {
                "type": bin_type,
                "collectionDate": collection_datetime.strftime(date_format),
                "_sort_date": collection_datetime
            }
    
            bins_with_sort_date.append(bin_entry)
    
        if not bins_with_sort_date:
>           raise ValueError(
                "No valid bin collection data could be extracted from the API response"
            )
E           ValueError: No valid bin collection data could be extracted from the API response

.../uk_bin_collection/councils/NorthHertfordshireDistrictCouncil.py:234: ValueError

uk_bin_collection.tests.step_defs.test_validate_council::test_scenario_outline[NorthumberlandCouncil]

Flake rate in main: 75.22% (Passed 56 times, Failed 170 times)

Stack Traces | 52.4s run time

fixturefunc = <function scrape_step at 0x7f8eef42b6a0>
request = <FixtureRequest for <Function test_scenario_outline[NorthumberlandCouncil]>>
kwargs = {'context': <test_validate_council.Context object at 0x7f8effdefd40>, 'headless_mode': 'True', 'local_browser': 'False', 'selenium_url': 'http://localhost:4444'}

    def call_fixture_func(
        fixturefunc: _FixtureFunc[FixtureValue], request: FixtureRequest, kwargs
    ) -> FixtureValue:
        if is_generator(fixturefunc):
            fixturefunc = cast(
                Callable[..., Generator[FixtureValue, None, None]], fixturefunc
            )
            generator = fixturefunc(**kwargs)
            try:
                fixture_result = next(generator)
            except StopIteration:
                raise ValueError(f"{request.fixturename} did not yield a value") from None
            finalizer = functools.partial(_teardown_yield_fixture, fixturefunc, generator)
            request.addfinalizer(finalizer)
        else:
            fixturefunc = cast(Callable[..., FixtureValue], fixturefunc)
>           fixture_result = fixturefunc(**kwargs)

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.12/lib/python3.12.../site-packages/_pytest/fixtures.py:898: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../tests/step_defs/test_validate_council.py:101: in scrape_step
    context.parse_result = CollectData.run()
uk_bin_collection/uk_bin_collection/collect_data.py:101: in run
    return self.client_code(
uk_bin_collection/uk_bin_collection/collect_data.py:121: in client_code
    return get_bin_data_class.template_method(address_url, **kwargs)
uk_bin_collection/uk_bin_collection/get_bin_data.py:61: in template_method
    bin_data_dict = self.get_and_parse_data(this_url, **kwargs)
uk_bin_collection/uk_bin_collection/get_bin_data.py:84: in get_and_parse_data
    bin_data_dict = self.parse_data("", url=address_url, **kwargs)
.../uk_bin_collection/councils/NorthumberlandCouncil.py:56: in parse_data
    cookie_button = wait.until(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <selenium.webdriver.support.wait.WebDriverWait (session="d7d59f0aa6408fb8ac75dec4c5f30f97")>
method = <function element_to_be_clickable.<locals>._predicate at 0x7f8eef0e7600>
message = ''

    def until(self, method: Callable[[D], Union[Literal[False], T]], message: str = "") -> T:
        """Wait until the method returns a value that is not False.
    
        Calls the method provided with the driver as an argument until the
        return value does not evaluate to ``False``.
    
        Parameters:
        -----------
        method: callable(WebDriver)
            - A callable object that takes a WebDriver instance as an argument.
    
        message: str
            - Optional message for :exc:`TimeoutException`
    
        Return:
        -------
        object: T
            - The result of the last call to `method`
    
        Raises:
        -------
        TimeoutException
            - If 'method' does not return a truthy value within the WebDriverWait
            object's timeout
    
        Example:
        --------
        >>> from selenium.webdriver.common.by import By
        >>> from selenium.webdriver.support.ui import WebDriverWait
        >>> from selenium.webdriver.support import expected_conditions as EC
    
        # Wait until an element is visible on the page
        >>> wait = WebDriverWait(driver, 10)
        >>> element = wait.until(EC.visibility_of_element_located((By.ID, "exampleId")))
        >>> print(element.text)
        """
        screen = None
        stacktrace = None
    
        end_time = time.monotonic() + self._timeout
        while True:
            try:
                value = method(self._driver)
                if value:
                    return value
            except self._ignored_exceptions as exc:
                screen = getattr(exc, "screen", None)
                stacktrace = getattr(exc, "stacktrace", None)
            if time.monotonic() > end_time:
                break
            time.sleep(self._poll)
>       raise TimeoutException(message, screen, stacktrace)
E       selenium.common.exceptions.TimeoutException: Message: 
E       Stacktrace:
E       #0 0x55718cc92aea <unknown>
E       #1 0x55718c6decdb <unknown>
E       #2 0x55718c7316c4 <unknown>
E       #3 0x55718c731901 <unknown>
E       #4 0x55718c7808b4 <unknown>
E       #5 0x55718c77dc87 <unknown>
E       #6 0x55718c723aca <unknown>
E       #7 0x55718c7247d1 <unknown>
E       #8 0x55718cc59ab9 <unknown>
E       #9 0x55718cc5ca8c <unknown>
E       #10 0x55718cc42d49 <unknown>
E       #11 0x55718cc5d685 <unknown>
E       #12 0x55718cc2a6c3 <unknown>
E       #13 0x55718cc7f7d8 <unknown>
E       #14 0x55718cc7f9b3 <unknown>
E       #15 0x55718cc91a83 <unknown>
E       #16 0x7ff21798baa4 <unknown>
E       #17 0x7ff217a18a64 __clone

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.12/lib/python3.12.../webdriver/support/wait.py:146: TimeoutException

uk_bin_collection.tests.step_defs.test_validate_council::test_scenario_outline[SouthHollandDistrictCouncil]

Flake rate in main: 78.71% (Passed 43 times, Failed 159 times)

Stack Traces | 0.683s run time

self = <Response [403]>, kwargs = {}

    def json(self, **kwargs):
        r"""Returns the json-encoded content of a response, if any.
    
        :param \*\*kwargs: Optional arguments that ``json.loads`` takes.
        :raises requests.exceptions.JSONDecodeError: If the response body does not
            contain valid json.
        """
    
        if not self.encoding and self.content and len(self.content) > 3:
            # No encoding set. JSON RFC 4627 section 3 states we should expect
            # UTF-8, -16 or -32. Detect which one to use; If the detection or
            # decoding fails, fall back to `self.text` (using charset_normalizer to make
            # a best guess).
            encoding = guess_json_utf(self.content)
            if encoding is not None:
                try:
                    return complexjson.loads(self.content.decode(encoding), **kwargs)
                except UnicodeDecodeError:
                    # Wrong UTF codec detected; usually because it's not UTF-8
                    # but some other 8-bit codec.  This is an RFC violation,
                    # and the server didn't bother to tell us what codec *was*
                    # used.
                    pass
                except JSONDecodeError as e:
                    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
    
        try:
>           return complexjson.loads(self.text, **kwargs)

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.12/lib/python3.12....../site-packages/requests/models.py:971: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
....../usr/lib/python3.12/json/__init__.py:346: in loads
    return _default_decoder.decode(s)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <json.decoder.JSONDecoder object at 0x7fac01f4dcd0>, s = '403 Forbidden'
_w = <built-in method match of re.Pattern object at 0x7fac01c1cba0>

    def decode(self, s, _w=WHITESPACE.match):
        """Return the Python representation of ``s`` (a ``str`` instance
        containing a JSON document).
    
        """
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
        end = _w(s, end).end()
        if end != len(s):
>           raise JSONDecodeError("Extra data", s, end)
E           json.decoder.JSONDecodeError: Extra data: line 1 column 5 (char 4)

....../usr/lib/python3.12/json/decoder.py:340: JSONDecodeError

During handling of the above exception, another exception occurred:

fixturefunc = <function scrape_step at 0x7fabebff19e0>
request = <FixtureRequest for <Function test_scenario_outline[SouthHollandDistrictCouncil]>>
kwargs = {'context': <test_validate_council.Context object at 0x7fabeccfd6d0>, 'headless_mode': 'True', 'local_browser': 'False', 'selenium_url': 'http://localhost:4444'}

    def call_fixture_func(
        fixturefunc: _FixtureFunc[FixtureValue], request: FixtureRequest, kwargs
    ) -> FixtureValue:
        if is_generator(fixturefunc):
            fixturefunc = cast(
                Callable[..., Generator[FixtureValue, None, None]], fixturefunc
            )
            generator = fixturefunc(**kwargs)
            try:
                fixture_result = next(generator)
            except StopIteration:
                raise ValueError(f"{request.fixturename} did not yield a value") from None
            finalizer = functools.partial(_teardown_yield_fixture, fixturefunc, generator)
            request.addfinalizer(finalizer)
        else:
            fixturefunc = cast(Callable[..., FixtureValue], fixturefunc)
>           fixture_result = fixturefunc(**kwargs)

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.12/lib/python3.12.../site-packages/_pytest/fixtures.py:898: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../tests/step_defs/test_validate_council.py:101: in scrape_step
    context.parse_result = CollectData.run()
uk_bin_collection/uk_bin_collection/collect_data.py:101: in run
    return self.client_code(
uk_bin_collection/uk_bin_collection/collect_data.py:121: in client_code
    return get_bin_data_class.template_method(address_url, **kwargs)
uk_bin_collection/uk_bin_collection/get_bin_data.py:61: in template_method
    bin_data_dict = self.get_and_parse_data(this_url, **kwargs)
uk_bin_collection/uk_bin_collection/get_bin_data.py:84: in get_and_parse_data
    bin_data_dict = self.parse_data("", url=address_url, **kwargs)
.../uk_bin_collection/councils/SouthHollandDistrictCouncil.py:35: in parse_data
    bin_collection = response.json()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Response [403]>, kwargs = {}

    def json(self, **kwargs):
        r"""Returns the json-encoded content of a response, if any.
    
        :param \*\*kwargs: Optional arguments that ``json.loads`` takes.
        :raises requests.exceptions.JSONDecodeError: If the response body does not
            contain valid json.
        """
    
        if not self.encoding and self.content and len(self.content) > 3:
            # No encoding set. JSON RFC 4627 section 3 states we should expect
            # UTF-8, -16 or -32. Detect which one to use; If the detection or
            # decoding fails, fall back to `self.text` (using charset_normalizer to make
            # a best guess).
            encoding = guess_json_utf(self.content)
            if encoding is not None:
                try:
                    return complexjson.loads(self.content.decode(encoding), **kwargs)
                except UnicodeDecodeError:
                    # Wrong UTF codec detected; usually because it's not UTF-8
                    # but some other 8-bit codec.  This is an RFC violation,
                    # and the server didn't bother to tell us what codec *was*
                    # used.
                    pass
                except JSONDecodeError as e:
                    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
    
        try:
            return complexjson.loads(self.text, **kwargs)
        except JSONDecodeError as e:
            # Catch JSON-related errors and raise as requests.JSONDecodeError
            # This aliases json.JSONDecodeError and simplejson.JSONDecodeError
>           raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
E           requests.exceptions.JSONDecodeError: Extra data: line 1 column 5 (char 4)

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.12/lib/python3.12....../site-packages/requests/models.py:975: JSONDecodeError

uk_bin_collection.tests.step_defs.test_validate_council::test_scenario_outline[SouthLanarkshireCouncil]

Flake rate in main: 40.10% (Passed 115 times, Failed 77 times)

Stack Traces | 1.14s run time

fixturefunc = <function scrape_step at 0x7fabebff19e0>
request = <FixtureRequest for <Function test_scenario_outline[SouthLanarkshireCouncil]>>
kwargs = {'context': <test_validate_council.Context object at 0x7fabeccfd6d0>, 'headless_mode': 'True', 'local_browser': 'False', 'selenium_url': 'http://localhost:4444'}

    def call_fixture_func(
        fixturefunc: _FixtureFunc[FixtureValue], request: FixtureRequest, kwargs
    ) -> FixtureValue:
        if is_generator(fixturefunc):
            fixturefunc = cast(
                Callable[..., Generator[FixtureValue, None, None]], fixturefunc
            )
            generator = fixturefunc(**kwargs)
            try:
                fixture_result = next(generator)
            except StopIteration:
                raise ValueError(f"{request.fixturename} did not yield a value") from None
            finalizer = functools.partial(_teardown_yield_fixture, fixturefunc, generator)
            request.addfinalizer(finalizer)
        else:
            fixturefunc = cast(Callable[..., FixtureValue], fixturefunc)
>           fixture_result = fixturefunc(**kwargs)

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.12/lib/python3.12.../site-packages/_pytest/fixtures.py:898: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../tests/step_defs/test_validate_council.py:101: in scrape_step
    context.parse_result = CollectData.run()
uk_bin_collection/uk_bin_collection/collect_data.py:101: in run
    return self.client_code(
uk_bin_collection/uk_bin_collection/collect_data.py:121: in client_code
    return get_bin_data_class.template_method(address_url, **kwargs)
uk_bin_collection/uk_bin_collection/get_bin_data.py:61: in template_method
    bin_data_dict = self.get_and_parse_data(this_url, **kwargs)
uk_bin_collection/uk_bin_collection/get_bin_data.py:82: in get_and_parse_data
    bin_data_dict = self.parse_data(page, url=address_url, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <SouthLanarkshireCouncil.CouncilClass object at 0x7fabebc43500>
page = <Response [200]>
kwargs = {'council_module_str': 'SouthLanarkshireCouncil', 'dev_mode': False, 'headless': True, 'local_browser': False, ...}
data = {'bins': []}
collection_types = ['non recyclable waste', 'food and garden', 'paper and card', 'glass, cans and plastics']
soup = <!DOCTYPE html>

<html lang="en">
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en"> <![endif]-->
<!--[if IE 7...4933156"></script>
<script src="//www.southlanarkshire.gov..../site/javascript/site.js"></script>
</div></body>
</html>

week_details = <div class="bin-dir-snip"><div class="clearfix">This week's collection<p>Monday 8 December 2025 to Friday 12 December ...plastic bags.</p></h4><img alt="red bin" src="https://www.southlanarkshire.gov.uk/images/bin-red.png"/></li></ul></div>
week_dates = <p>Monday 8 December 2025 to Friday 12 December 2025 </p>
week_collections = [<h4><a href="https://www.southlanarkshire.gov..../bins_and_recycling/1841/bins_-_what_goes_in_them/" title...dy bin - food and garden waste">Burgundy bin - food and garden waste</a>
<p>No liquids, oils or plastic bags.</p></h4>]
results = <re.Match object; span=(0, 49), match='Monday 8 December 2025 to Friday 12 December 2025>

    def parse_data(self, page: str, **kwargs) -> dict:
        data = {"bins": []}
        collection_types = [
            "non recyclable waste",
            "food and garden",
            "paper and card",
            "glass, cans and plastics",
        ]
    
        # Make a BS4 object
        soup = BeautifulSoup(page.text, features="html.parser")
        soup.prettify()
    
        week_details = soup.find("div", {"class": "bin-dir-snip"})
        week_dates = week_details.find("div", {"class": "clearfix"}).find("p")
        week_collections = week_details.find_all_next("h4")
    
        results = re.search(
            "([A-Za-z0-9 ]+) to ([A-Za-z0-9 ]+)", week_dates.get_text().strip()
        )
        if results:
            week_start = datetime.strptime(results.groups()[0], "%A %d %B %Y")
            week_end = datetime.strptime(results.groups()[1], "%A %d %B %Y")
            week_days = (
                week_start + timedelta(days=i)
                for i in range((week_end - week_start).days + 1)
            )
    
            week_collection_types = []
            for week_collection in week_collections:
                week_collection = (
                    week_collection.get_text().strip().lower().replace("-", " ")
                )
                for collection_type in collection_types:
                    if collection_type in week_collection:
                        week_collection_types.append(collection_type)
    
            collection_schedule = (
                soup.find("div", {"class": "serviceDetails"})
                .find("table")
                .find_all_next("tr")
            )
            for day in week_days:
                for row in collection_schedule:
                    schedule_type = row.find("th").get_text().strip()
                    results2 = re.search("([^(]+)", row.find("td").get_text().strip())
>                   schedule_cadence = row.find("td").get_text().strip().split(" ")[1]
E                   IndexError: list index out of range

.../uk_bin_collection/councils/SouthLanarkshireCouncil.py:64: IndexError

uk_bin_collection.tests.step_defs.test_validate_council::test_scenario_outline[WestOxfordshireDistrictCouncil]

Flake rate in main: 67.57% (Passed 72 times, Failed 150 times)

Stack Traces | 98.2s run time

fixturefunc = <function scrape_step at 0x7fabebff19e0>
request = <FixtureRequest for <Function test_scenario_outline[WestOxfordshireDistrictCouncil]>>
kwargs = {'context': <test_validate_council.Context object at 0x7fabeccfd6d0>, 'headless_mode': 'True', 'local_browser': 'False', 'selenium_url': 'http://localhost:4444'}

    def call_fixture_func(
        fixturefunc: _FixtureFunc[FixtureValue], request: FixtureRequest, kwargs
    ) -> FixtureValue:
        if is_generator(fixturefunc):
            fixturefunc = cast(
                Callable[..., Generator[FixtureValue, None, None]], fixturefunc
            )
            generator = fixturefunc(**kwargs)
            try:
                fixture_result = next(generator)
            except StopIteration:
                raise ValueError(f"{request.fixturename} did not yield a value") from None
            finalizer = functools.partial(_teardown_yield_fixture, fixturefunc, generator)
            request.addfinalizer(finalizer)
        else:
            fixturefunc = cast(Callable[..., FixtureValue], fixturefunc)
>           fixture_result = fixturefunc(**kwargs)

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.12/lib/python3.12.../site-packages/_pytest/fixtures.py:898: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../tests/step_defs/test_validate_council.py:101: in scrape_step
    context.parse_result = CollectData.run()
uk_bin_collection/uk_bin_collection/collect_data.py:101: in run
    return self.client_code(
uk_bin_collection/uk_bin_collection/collect_data.py:121: in client_code
    return get_bin_data_class.template_method(address_url, **kwargs)
uk_bin_collection/uk_bin_collection/get_bin_data.py:61: in template_method
    bin_data_dict = self.get_and_parse_data(this_url, **kwargs)
uk_bin_collection/uk_bin_collection/get_bin_data.py:84: in get_and_parse_data
    bin_data_dict = self.parse_data("", url=address_url, **kwargs)
.../uk_bin_collection/councils/WestOxfordshireDistrictCouncil.py:57: in parse_data
    first_found_address = wait.until(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <selenium.webdriver.support.wait.WebDriverWait (session="ce159fe87ef1601d9d9a2699544c5557")>
method = <function element_to_be_clickable.<locals>._predicate at 0x7fabeb55f380>
message = ''

    def until(self, method: Callable[[D], Union[Literal[False], T]], message: str = "") -> T:
        """Wait until the method returns a value that is not False.
    
        Calls the method provided with the driver as an argument until the
        return value does not evaluate to ``False``.
    
        Parameters:
        -----------
        method: callable(WebDriver)
            - A callable object that takes a WebDriver instance as an argument.
    
        message: str
            - Optional message for :exc:`TimeoutException`
    
        Return:
        -------
        object: T
            - The result of the last call to `method`
    
        Raises:
        -------
        TimeoutException
            - If 'method' does not return a truthy value within the WebDriverWait
            object's timeout
    
        Example:
        --------
        >>> from selenium.webdriver.common.by import By
        >>> from selenium.webdriver.support.ui import WebDriverWait
        >>> from selenium.webdriver.support import expected_conditions as EC
    
        # Wait until an element is visible on the page
        >>> wait = WebDriverWait(driver, 10)
        >>> element = wait.until(EC.visibility_of_element_located((By.ID, "exampleId")))
        >>> print(element.text)
        """
        screen = None
        stacktrace = None
    
        end_time = time.monotonic() + self._timeout
        while True:
            try:
                value = method(self._driver)
                if value:
                    return value
            except self._ignored_exceptions as exc:
                screen = getattr(exc, "screen", None)
                stacktrace = getattr(exc, "stacktrace", None)
            if time.monotonic() > end_time:
                break
            time.sleep(self._poll)
>       raise TimeoutException(message, screen, stacktrace)
E       selenium.common.exceptions.TimeoutException: Message: 
E       Stacktrace:
E       #0 0x556414bd1aea <unknown>
E       #1 0x55641461dcdb <unknown>
E       #2 0x5564146706c4 <unknown>
E       #3 0x556414670901 <unknown>
E       #4 0x5564146bf8b4 <unknown>
E       #5 0x5564146bcc87 <unknown>
E       #6 0x556414662aca <unknown>
E       #7 0x5564146637d1 <unknown>
E       #8 0x556414b98ab9 <unknown>
E       #9 0x556414b9ba8c <unknown>
E       #10 0x556414b81d49 <unknown>
E       #11 0x556414b9c685 <unknown>
E       #12 0x556414b696c3 <unknown>
E       #13 0x556414bbe7d8 <unknown>
E       #14 0x556414bbe9b3 <unknown>
E       #15 0x556414bd0a83 <unknown>
E       #16 0x7f312219aaa4 <unknown>
E       #17 0x7f3122227a64 __clone

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.12/lib/python3.12.../webdriver/support/wait.py:146: TimeoutException

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[coderabbitai]

Actionable comments posted: 14

♻️ Duplicate comments (7)

uk_bin_collection/uk_bin_collection/councils/ThurrockCouncil.py (1)

39-40: Same date concern applies to Round B.

These November 2025 dates have the same issue as noted in lines 36-37. Ensure the start dates align with your intended collection schedule.

.github/workflows/wiki.yml (1)

25-25: Consistent Actions upgrade across workflows.

Same v6 upgrade as in other workflow files—refer to the verification request in validate-release-ready.yml.

.github/workflows/hacs_validation.yml (1)

14-14: Consistent Actions upgrade.

Same v6 upgrade—refer to verification in validate-release-ready.yml.

.github/workflows/lint.yml (1)

19-19: Consistent Actions upgrade.

Same v6 upgrade—refer to verification in validate-release-ready.yml.

.github/workflows/codeql-analysis.yml (1)

50-50: Consistent Actions upgrade.

Same v6 upgrade—refer to verification in validate-release-ready.yml.

.github/workflows/bump.yml (1)

22-22: Consistent Actions upgrade.

Same v6 upgrade—refer to verification in validate-release-ready.yml.

.github/workflows/ha_compatibility_test.yml (1)

56-56: Consistent Actions upgrade across both jobs.

Both checkout steps upgraded to v6—refer to verification in validate-release-ready.yml.

Also applies to: 230-230

🧹 Nitpick comments (18)

uk_bin_collection/uk_bin_collection/councils/HaltonBoroughCouncil.py (1)

38-40: User‑agent addition and create_webdriver call look correct; consider fixing Ruff F405 by avoiding import *.

• The new user_agent assignment and the call create_webdriver(web_driver, headless, user_agent, __name__) match the create_webdriver signature and should behave as intended.
• Ruff’s F405 on create_webdriver is due to from uk_bin_collection.uk_bin_collection.common import * (name only guaranteed via star import). To keep linters happy and make dependencies explicit, you can switch to explicit imports and pull in re directly:

import re
from uk_bin_collection.uk_bin_collection.common import create_webdriver

Then remove the import * line. This keeps the new behavior intact while resolving the lint warning and making the dependency on create_webdriver and re explicit.

uk_bin_collection/uk_bin_collection/councils/FarehamBoroughCouncil.py (1)

1-4: Consider explicit imports for clarity.

The static analysis tool flags that re, datetime, and date_format may be undefined due to the star import on line 4. While this likely works correctly as a project-wide pattern, explicit imports improve code clarity and make dependencies obvious.

uk_bin_collection/uk_bin_collection/councils/EdinburghCityCouncil.py (1)

11-12: Consider replacing * import and making datetime imports explicit (to satisfy Ruff F405)

Ruff is flagging datetime(...) here because it’s being pulled in via from uk_bin_collection.uk_bin_collection.common import *, which makes the name resolution ambiguous to static analysis. To make this file self‑contained and silence F405, you could switch to explicit imports, e.g.:

-from uk_bin_collection.uk_bin_collection.common import *
+from datetime import datetime, timedelta
+from uk_bin_collection.uk_bin_collection.common import get_dates_every_x_days

This keeps behaviour the same while avoiding * imports and clarifying where datetime/timedelta come from.

Also applies to: 45-51

uk_bin_collection/uk_bin_collection/councils/ThurrockCouncil.py (1)

1-1: Consider explicit imports to address static analysis warnings.

The star import makes it unclear where datetime and other symbols come from, triggering Ruff warnings (F405). While this appears to be a project-wide pattern, explicit imports improve code clarity and help static analysis tools.

Consider replacing the star import with explicit imports:

-from uk_bin_collection.uk_bin_collection.common import *
+from uk_bin_collection.uk_bin_collection.common import datetime, timedelta, get_dates_every_x_days

uk_bin_collection/uk_bin_collection/councils/ChelmsfordCityCouncil.py (1)

110-112: Consider refactoring exception handling per static analysis hints.

Static analysis suggests abstracting the exception raise to an inner function and avoiding long messages in the exception instantiation. However, the current implementation is clear and functional.

If you want to address the linter hints:

+def _raise_ics_not_found(address: str) -> None:
+    """Helper to raise ValueError when ICS file cannot be found."""
+    raise ValueError(f"Could not find collection ICS file for address: {address}")
+

Then replace the raise statement:

-                            raise ValueError(
-                                f"Could not find collection ICS file for address: {user_paon}"
-                            )
+                            _raise_ics_not_found(user_paon)

uk_bin_collection/uk_bin_collection/councils/KingsLynnandWestNorfolkBC.py (1)

25-28: User-Agent header addition is sensible; consider adding a timeout

Explicit Cookie + realistic User-Agent is a good way to keep the council site happy. Since this scraper runs over the network, it would be safer to add a finite timeout so a slow endpoint doesn’t hang the entire run, e.g.:

-        response = requests.get(URI, headers=headers)
+        response = requests.get(URI, headers=headers, timeout=10)

Tune the timeout to whatever matches your typical end‑to‑end runtime.

Also applies to: 31-31

uk_bin_collection/uk_bin_collection/councils/ArmaghBanbridgeCraigavonCouncil.py (1)

22-24: Headers addition is good; add a timeout to requests.get

Defining a headers dict with a browser‑like user agent and passing it into requests.get should help avoid the site blocking the scraper, but the call still has no timeout. To avoid hangs on network issues, consider:

-        response = requests.get(url, headers=headers)
+        response = requests.get(url, headers=headers, timeout=10)

(or another suitable timeout for your environment).
Based on static analysis hints.

Also applies to: 47-47

uk_bin_collection/uk_bin_collection/councils/LondonBoroughLambeth.py (1)

45-52: Consider case-insensitive matching for "Recycling" check.

The string comparison "Recycling" in Container["Name"] is case-sensitive. If the API returns "recycling" or "RECYCLING", this check would fail and incorrectly classify it as "refuse".

                            if Container["DisplayPhrase"] == "commercial bin":
                                Bin_Type = (
                                    "recycling"
-                                    if "Recycling" in Container["Name"]
+                                    if "recycling" in Container["Name"].lower()
                                    else "refuse"
                                )

uk_bin_collection/uk_bin_collection/councils/WiltshireCouncil.py (1)

97-120: Minor: Avoid shadowing the type builtin.

The loop variable type on line 112 shadows Python's built-in type() function. While functionally correct here, it's a common style issue that can cause confusion.

-                    for type in collection_types:
-
-                        dict_data = {
-                            "type": type,
+                    for coll_type in collection_types:
+                        dict_data = {
+                            "type": coll_type,
                             "collectionDate": collectiondate,
                         }
                         data_bins["bins"].append(dict_data)

uk_bin_collection/uk_bin_collection/councils/RushmoorCouncil.py (1)

41-43: Add error handling for API response.

If the API returns an error or unexpected structure, soup.find("p") or the JSON parsing could fail with an unclear error.

Consider adding validation:

         result = soup.find("p").contents[0]
+        if not result:
+            raise ValueError("Unexpected API response format")
 
         json_data = json.loads(result)["NextCollection"]

uk_bin_collection/uk_bin_collection/councils/FifeCouncil.py (2)

71-71: Excessive hardcoded sleep.

The 10-second sleep will significantly slow down every request. Consider reducing this or using a more targeted wait condition.

-            time.sleep(10)
+            time.sleep(2)  # Allow dropdown options to populate

If the dropdown needs more time to load, consider using an explicit wait for the expected number of options instead.

---

48-50: Cloudflare bypass approach is fragile.

Checking for "Just a moment" in the title is reasonable, but the subsequent 3-second sleep is arbitrary. This may fail intermittently.

Consider adding retry logic or a more robust wait condition if Cloudflare challenges are common.

uk_bin_collection/uk_bin_collection/councils/BlackpoolCouncil.py (2)

1-1: Remove unused import.

The time module is imported but never used.

-import time
-
 import requests

---

69-82: Add defensive check for missing jobsField.

If the API response doesn't contain jobsField, this will raise a KeyError with no context.

+        if "jobsField" not in bin_collection:
+            raise ValueError("Unexpected API response: missing jobsField")
+
         # Loop through each collection in bin_collection
         for collection in bin_collection["jobsField"]:

uk_bin_collection/uk_bin_collection/councils/MertonCouncil.py (2)

89-100: Consider using a session for polling requests.

Creating a new connection for each poll iteration is inefficient. Using a requests.Session() would reuse the TCP connection and improve performance during the polling loop.

+        session = requests.Session()
         for attempt in range(1, self.MAX_POLLING_ATTEMPTS + 1):
-            response = requests.get(url, headers=headers, timeout=10)
+            response = session.get(url, headers=headers, timeout=10)
             soup = BeautifulSoup(response.text, features="html.parser")

---

141-159: Date parsing assumes specific text format.

The parsing extracts date_parts[1] (day) and date_parts[2] (month), assuming the format is always "DayName Day Month". If the council changes the format or includes additional text, this will silently fail. Consider adding logging when date parsing fails to aid debugging.

uk_bin_collection/uk_bin_collection/councils/IsleOfAngleseyCouncil.py (1)

37-43: Potential compatibility issue with JSONDecodeError import.

requests.exceptions.JSONDecodeError was introduced in requests 2.27.0. For broader compatibility, consider catching json.JSONDecodeError directly or using ValueError as a fallback.

+from json import JSONDecodeError
+
 ...
 
         try:
             if not response.json().get("auth-session"):
                 raise ValueError("Failed to obtain session cookie")
-        except requests.exceptions.JSONDecodeError as e:
+        except JSONDecodeError as e:
             raise ValueError("Failed to decode session response as JSON") from e

uk_bin_collection/uk_bin_collection/councils/SouthamptonCityCouncil.py (1)

67-76: Robust calendar guard looks good; consider explicit re import for linting clarity

The narrowed re.search scope plus explicit ValueError if the calendar fragment is missing is a good defensive improvement and will fail fast if the page structure changes.

Ruff’s F405 hint about re coming via the common star import is reasonable; to keep things explicit and linter‑friendly, you can add a direct import at the top:

-import time
-
-import requests
+import time
+import re
+
+import requests

I’d keep the current error message despite TRY003; it’s concise and actionable enough here.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 37c8a80 and 108d4ae.

⛔ Files ignored due to path filters (1)

• poetry.lock is excluded by !**/*.lock

📒 Files selected for processing (42)

• .github/workflows/behave_pull_request.yml (4 hunks)
• .github/workflows/behave_schedule.yml (4 hunks)
• .github/workflows/bump.yml (1 hunks)
• .github/workflows/codeql-analysis.yml (1 hunks)
• .github/workflows/docker-image.yml (1 hunks)
• .github/workflows/ha_compatibility_test.yml (2 hunks)
• .github/workflows/hacs_validation.yml (1 hunks)
• .github/workflows/lint.yml (1 hunks)
• .github/workflows/release.yml (1 hunks)
• .github/workflows/rollback-release.yml (1 hunks)
• .github/workflows/validate-release-ready.yml (1 hunks)
• .github/workflows/wiki.yml (1 hunks)
• pyproject.toml (1 hunks)
• uk_bin_collection/tests/input.json (8 hunks)
• uk_bin_collection/uk_bin_collection/councils/ArgyllandButeCouncil.py (2 hunks)
• uk_bin_collection/uk_bin_collection/councils/ArmaghBanbridgeCraigavonCouncil.py (2 hunks)
• uk_bin_collection/uk_bin_collection/councils/BirminghamCityCouncil.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/BlackpoolCouncil.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/ChelmsfordCityCouncil.py (3 hunks)
• uk_bin_collection/uk_bin_collection/councils/EdinburghCityCouncil.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/FarehamBoroughCouncil.py (3 hunks)
• uk_bin_collection/uk_bin_collection/councils/FifeCouncil.py (2 hunks)
• uk_bin_collection/uk_bin_collection/councils/HaltonBoroughCouncil.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/HarlowCouncil.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/IsleOfAngleseyCouncil.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/KingsLynnandWestNorfolkBC.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/LondonBoroughLambeth.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/MertonCouncil.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/MidSussexDistrictCouncil.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/NorthHertfordshireDistrictCouncil.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/NorthumberlandCouncil.py (2 hunks)
• uk_bin_collection/uk_bin_collection/councils/NorwichCityCouncil.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/RushmoorCouncil.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/SouthGloucestershireCouncil.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/SouthLanarkshireCouncil.py (2 hunks)
• uk_bin_collection/uk_bin_collection/councils/SouthamptonCityCouncil.py (4 hunks)
• uk_bin_collection/uk_bin_collection/councils/ThurrockCouncil.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/WestOxfordshireDistrictCouncil.py (1 hunks)
• uk_bin_collection/uk_bin_collection/councils/WiltshireCouncil.py (3 hunks)
• uk_bin_collection/uk_bin_collection/councils/WinchesterCityCouncil.py (3 hunks)
• uk_bin_collection_api_server/requirements.txt (1 hunks)
• wiki/Councils.md (7 hunks)

🧰 Additional context used

🧬 Code graph analysis (10)

uk_bin_collection/uk_bin_collection/councils/HaltonBoroughCouncil.py (1)

uk_bin_collection/uk_bin_collection/common.py (1)

• create_webdriver (321-360)

uk_bin_collection/uk_bin_collection/councils/MertonCouncil.py (6)

uk_bin_collection/uk_bin_collection/councils/BirminghamCityCouncil.py (2)

• CouncilClass (27-119)
• parse_data (67-119)

uk_bin_collection/uk_bin_collection/councils/RochfordCouncil.py (2)

• CouncilClass (9-59)
• parse_data (16-59)

uk_bin_collection/uk_bin_collection/councils/ChorleyCouncil.py (1)

• CouncilClass (14-148)

uk_bin_collection/uk_bin_collection/councils/KingstonUponThamesCouncil.py (2)

• CouncilClass (20-86)
• parse_data (27-86)

uk_bin_collection/uk_bin_collection/councils/EastleighBoroughCouncil.py (2)

• CouncilClass (11-89)
• parse_data (18-89)

uk_bin_collection/uk_bin_collection/councils/NorthWestLeicestershire.py (2)

• CouncilClass (16-114)
• parse_data (23-114)

uk_bin_collection/uk_bin_collection/councils/FifeCouncil.py (1)

uk_bin_collection/uk_bin_collection/common.py (3)

• check_postcode (36-49)
• check_paon (52-64)
• create_webdriver (321-360)

uk_bin_collection/uk_bin_collection/councils/NorthHertfordshireDistrictCouncil.py (2)

uk_bin_collection/uk_bin_collection/common.py (3)

• check_paon (52-64)
• check_postcode (36-49)
• check_uprn (67-78)

uk_bin_collection/uk_bin_collection/get_bin_data.py (1)

• AbstractGetBinDataClass (43-146)

uk_bin_collection/uk_bin_collection/councils/BlackpoolCouncil.py (3)

uk_bin_collection/uk_bin_collection/councils/NuneatonBedworthBoroughCouncil.py (1)

• get_bin_data (63-945)

uk_bin_collection/uk_bin_collection/get_bin_data.py (1)

• AbstractGetBinDataClass (43-146)

uk_bin_collection/uk_bin_collection/common.py (2)

• check_uprn (67-78)
• check_postcode (36-49)

uk_bin_collection/uk_bin_collection/councils/WiltshireCouncil.py (1)

custom_components/uk_bin_collection/calendar.py (1)

• event (54-63)

uk_bin_collection/uk_bin_collection/councils/IsleOfAngleseyCouncil.py (2)

uk_bin_collection/uk_bin_collection/common.py (2)

• check_postcode (36-49)
• check_uprn (67-78)

uk_bin_collection/uk_bin_collection/get_bin_data.py (1)

• AbstractGetBinDataClass (43-146)

uk_bin_collection/uk_bin_collection/councils/HarlowCouncil.py (2)

uk_bin_collection/uk_bin_collection/get_bin_data.py (1)

• AbstractGetBinDataClass (43-146)

uk_bin_collection/uk_bin_collection/common.py (1)

• check_uprn (67-78)

uk_bin_collection/uk_bin_collection/councils/MidSussexDistrictCouncil.py (1)

uk_bin_collection/uk_bin_collection/common.py (1)

• check_postcode (36-49)

uk_bin_collection/uk_bin_collection/councils/RushmoorCouncil.py (1)

uk_bin_collection/uk_bin_collection/common.py (1)

• check_uprn (67-78)

🪛 Ruff (0.14.7)

uk_bin_collection/uk_bin_collection/councils/ArmaghBanbridgeCraigavonCouncil.py

47-47: Probable use of requests call without timeout

(S113)

uk_bin_collection/uk_bin_collection/councils/HaltonBoroughCouncil.py

40-40: create_webdriver may be undefined, or defined from star imports

(F405)

uk_bin_collection/uk_bin_collection/councils/MertonCouncil.py

34-34: Unused method argument: page

(ARG002)

---

70-70: Avoid specifying long messages outside the exception class

(TRY003)

---

74-74: Avoid specifying long messages outside the exception class

(TRY003)

---

99-99: Create your own exception

(TRY002)

---

99-99: Avoid specifying long messages outside the exception class

(TRY003)

---

105-105: Create your own exception

(TRY002)

---

105-105: Avoid specifying long messages outside the exception class

(TRY003)

uk_bin_collection/uk_bin_collection/councils/SouthamptonCityCouncil.py

68-68: re may be undefined, or defined from star imports

(F405)

---

69-69: re may be undefined, or defined from star imports

(F405)

---

72-74: Avoid specifying long messages outside the exception class

(TRY003)

uk_bin_collection/uk_bin_collection/councils/FifeCouncil.py

27-27: check_postcode may be undefined, or defined from star imports

(F405)

---

28-28: check_paon may be undefined, or defined from star imports

(F405)

---

38-38: create_webdriver may be undefined, or defined from star imports

(F405)

---

66-66: Unused lambda argument: d

(ARG005)

---

83-85: Abstract raise to an inner function

(TRY301)

---

83-85: Avoid specifying long messages outside the exception class

(TRY003)

---

102-104: Abstract raise to an inner function

(TRY301)

---

102-104: Avoid specifying long messages outside the exception class

(TRY003)

---

126-126: date_format may be undefined, or defined from star imports

(F405)

---

131-131: date_format may be undefined, or defined from star imports

(F405)

uk_bin_collection/uk_bin_collection/councils/ThurrockCouncil.py

36-36: datetime may be undefined, or defined from star imports

(F405)

---

37-37: datetime may be undefined, or defined from star imports

(F405)

---

39-39: datetime may be undefined, or defined from star imports

(F405)

---

40-40: datetime may be undefined, or defined from star imports

(F405)

uk_bin_collection/uk_bin_collection/councils/WinchesterCityCouncil.py

99-99: Abstract raise to an inner function

(TRY301)

---

99-99: Avoid specifying long messages outside the exception class

(TRY003)

---

107-107: datetime may be undefined, or defined from star imports

(F405)

---

108-108: datetime may be undefined, or defined from star imports

(F405)

---

124-124: datetime may be undefined, or defined from star imports

(F405)

uk_bin_collection/uk_bin_collection/councils/FarehamBoroughCouncil.py

60-60: re may be undefined, or defined from star imports

(F405)

---

61-61: datetime may be undefined, or defined from star imports

(F405)

---

64-64: date_format may be undefined, or defined from star imports

(F405)

uk_bin_collection/uk_bin_collection/councils/NorthHertfordshireDistrictCouncil.py

50-50: Avoid specifying long messages outside the exception class

(TRY003)

---

53-53: Avoid specifying long messages outside the exception class

(TRY003)

---

63-63: Avoid specifying long messages outside the exception class

(TRY003)

---

66-69: Avoid specifying long messages outside the exception class

(TRY003)

---

74-74: Avoid specifying long messages outside the exception class

(TRY003)

---

78-81: Avoid specifying long messages outside the exception class

(TRY003)

---

101-105: Avoid specifying long messages outside the exception class

(TRY003)

---

109-113: Avoid specifying long messages outside the exception class

(TRY003)

---

137-137: Avoid specifying long messages outside the exception class

(TRY003)

---

140-143: Avoid specifying long messages outside the exception class

(TRY003)

---

149-149: Avoid specifying long messages outside the exception class

(TRY003)

---

158-158: Unused method argument: page

(ARG002)

---

197-199: Avoid specifying long messages outside the exception class

(TRY003)

---

234-236: Avoid specifying long messages outside the exception class

(TRY003)

uk_bin_collection/uk_bin_collection/councils/EdinburghCityCouncil.py

45-45: datetime may be undefined, or defined from star imports

(F405)

---

46-46: datetime may be undefined, or defined from star imports

(F405)

---

47-47: datetime may be undefined, or defined from star imports

(F405)

---

49-49: datetime may be undefined, or defined from star imports

(F405)

---

50-50: datetime may be undefined, or defined from star imports

(F405)

---

51-51: datetime may be undefined, or defined from star imports

(F405)

uk_bin_collection/uk_bin_collection/councils/BirminghamCityCouncil.py

113-113: date_format may be undefined, or defined from star imports

(F405)

uk_bin_collection/uk_bin_collection/councils/BlackpoolCouncil.py

5-5: from uk_bin_collection.uk_bin_collection.common import * used; unable to detect undefined names

(F403)

---

17-17: Unused method argument: page

(ARG002)

---

21-21: check_uprn may be undefined, or defined from star imports

(F405)

---

22-22: check_postcode may be undefined, or defined from star imports

(F405)

---

41-41: Probable use of requests call without timeout

(S113)

---

59-59: Probable use of requests call without timeout

(S113)

---

77-77: datetime may be undefined, or defined from star imports

(F405)

---

80-80: date_format may be undefined, or defined from star imports

(F405)

---

85-85: datetime may be undefined, or defined from star imports

(F405)

---

85-85: date_format may be undefined, or defined from star imports

(F405)

uk_bin_collection/uk_bin_collection/councils/WiltshireCouncil.py

102-102: datetime may be undefined, or defined from star imports

(F405)

---

105-105: date_format may be undefined, or defined from star imports

(F405)

uk_bin_collection/uk_bin_collection/councils/IsleOfAngleseyCouncil.py

39-39: Avoid specifying long messages outside the exception class

(TRY003)

---

41-41: Avoid specifying long messages outside the exception class

(TRY003)

---

67-67: Avoid specifying long messages outside the exception class

(TRY003)

---

70-70: Avoid specifying long messages outside the exception class

(TRY003)

---

96-96: Avoid specifying long messages outside the exception class

(TRY003)

---

114-118: Avoid specifying long messages outside the exception class

(TRY003)

---

140-140: Avoid specifying long messages outside the exception class

(TRY003)

---

142-144: Avoid specifying long messages outside the exception class

(TRY003)

---

178-178: Avoid specifying long messages outside the exception class

(TRY003)

uk_bin_collection/uk_bin_collection/councils/HarlowCouncil.py

6-6: from uk_bin_collection.uk_bin_collection.common import * used; unable to detect undefined names

(F403)

---

18-18: Unused method argument: page

(ARG002)

---

21-21: check_uprn may be undefined, or defined from star imports

(F405)

---

32-32: Probable use of requests call without timeout

(S113)

---

48-48: datetime may be undefined, or defined from star imports

(F405)

---

54-54: date_format may be undefined, or defined from star imports

(F405)

uk_bin_collection/uk_bin_collection/councils/ChelmsfordCityCouncil.py

110-112: Abstract raise to an inner function

(TRY301)

---

110-112: Avoid specifying long messages outside the exception class

(TRY003)

uk_bin_collection/uk_bin_collection/councils/MidSussexDistrictCouncil.py

25-25: check_postcode may be undefined, or defined from star imports

(F405)

---

29-29: requests may be undefined, or defined from star imports

(F405)

---

44-44: Abstract raise to an inner function

(TRY301)

---

44-44: Create your own exception

(TRY002)

---

44-44: Avoid specifying long messages outside the exception class

(TRY003)

---

64-64: Abstract raise to an inner function

(TRY301)

---

64-64: Create your own exception

(TRY002)

---

64-64: Avoid specifying long messages outside the exception class

(TRY003)

---

76-76: Abstract raise to an inner function

(TRY301)

---

76-76: Create your own exception

(TRY002)

---

76-76: Avoid specifying long messages outside the exception class

(TRY003)

---

90-90: datetime may be undefined, or defined from star imports

(F405)

---

93-93: date_format may be undefined, or defined from star imports

(F405)

---

98-98: datetime may be undefined, or defined from star imports

(F405)

---

98-98: date_format may be undefined, or defined from star imports

(F405)

---

101-101: Consider moving this statement to an else block

(TRY300)

uk_bin_collection/uk_bin_collection/councils/SouthLanarkshireCouncil.py

74-74: date_format may be undefined, or defined from star imports

(F405)

---

82-82: date_format may be undefined, or defined from star imports

(F405)

uk_bin_collection/uk_bin_collection/councils/RushmoorCouncil.py

19-19: check_uprn may be undefined, or defined from star imports

(F405)

---

29-29: requests may be undefined, or defined from star imports

(F405)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: Run Integration Tests (3.12, 1.8.4)

🔇 Additional comments (52)

uk_bin_collection/uk_bin_collection/councils/FarehamBoroughCouncil.py (2)

39-39: Verify the field name in the API response.

The code now extracts data from BinCollectionInformation. Ensure this field exists in the API response for the new DomesticBinCollections2025on parameter.

---

24-24: The new API parameter DomesticBinCollections2025on is confirmed active and returns valid bin collection data. The old parameter DomesticBinCollections is now deprecated and no longer returns data. The parameter update is correct and necessary for the scraper to function.

uk_bin_collection/uk_bin_collection/councils/EdinburghCityCouncil.py (1)

44-51: Date roll‑forward keeps schedule logic intact

The updated 2025 base dates for Week 1 and Week 2 still fall on Mondays and preserve the existing 14‑day cadence and relative offsets between refuse, recycling, and glass collections. The control flow and downstream calculations remain consistent with the previous implementation.

uk_bin_collection/uk_bin_collection/councils/ChelmsfordCityCouncil.py (2)

20-34: LGTM! Documentation improvements enhance clarity.

The updated docstring provides better context about the dynamic ICS retrieval and parameter usage.

---

101-113: Add URL resolution for potentially relative ICS hrefs.

The icalevents library does not automatically resolve relative URLs. If the extracted href is relative (e.g., /path/to/calendar.ics), passing it directly to events() at line 124 will fail. Use urllib.parse.urljoin to ensure the URL is absolute:

+from urllib.parse import urljoin

Then update the extraction logic:

                        if a:
-                            ics_url = a["href"]
+                            ics_url = urljoin(calendar_url, a["href"])
                        else:
                            raise ValueError(
                                f"Could not find collection ICS file for address: {user_paon}"
                            )

uk_bin_collection/uk_bin_collection/councils/SouthGloucestershireCouncil.py (1)

51-52: LGTM! More robust falsy check.

Changing from item == "" to not item correctly handles both missing keys (None) and empty strings, preventing potential runtime errors when calling datetime.fromisoformat(item) on line 54.

.github/workflows/validate-release-ready.yml (1)

14-14: actions/checkout v6 upgrade is compatible.

The upgrade introduces breaking changes to persist-credentials behavior (credentials now written to a separate file under $RUNNER_TEMP instead of .git/config) and requires Actions Runner v2.329.0+ for self-hosted runners that use Docker container actions accessing persisted credentials. These changes don't impact standard workflows like this one. The upgrade is safe to proceed.

.github/workflows/docker-image.yml (1)

25-25: Checkout action bump to v6 looks fine

Pure version bump of actions/checkout with no other workflow changes; consistent with the rest of the PR. Just ensure v6 is available in your GitHub Actions environment.

.github/workflows/release.yml (1)

16-16: Release workflow checkout v6 change is safe

Only the checkout step version is updated; rest of the publish flow is untouched, so behavior should remain identical.

.github/workflows/behave_pull_request.yml (1)

17-17: Consistent migration to actions/checkout@v6 across PR workflow

All jobs (setup, unit, parity, integration) now use the same major version of actions/checkout, which keeps behavior consistent across the pipeline. No other logic changed.

Also applies to: 64-64, 95-95, 128-128

uk_bin_collection/uk_bin_collection/councils/ArgyllandButeCouncil.py (1)

1-1: UPRN normalization to 12 digits is a good hardening

Casting user_uprn to str and zfill(12) before feeding it into the <select> value should make the lookup robust to callers passing non‑padded or integer UPRNs. The added datetime import aligns with existing usage below.

Also applies to: 31-31

.github/workflows/rollback-release.yml (1)

23-23: Rollback workflow checkout bump is straightforward

Only the checkout action version changed; rollback logic, GH CLI usage, and PyPI steps remain as before.

pyproject.toml (1)

80-85: Version bump automation now also updates API server requirements

Adding uk_bin_collection_api_server/requirements.txt:uk-bin-collection to version_files will keep the API server’s pinned dependency in sync with the library version, which should reduce missed bumps on releases. TOML array syntax looks valid.

uk_bin_collection_api_server/requirements.txt (1)

1-3: LGTM!

Good consolidation of Connexion extras into a single line, and adding the minimum version constraint for uk-bin-collection ensures API compatibility.

uk_bin_collection/uk_bin_collection/councils/NorthumberlandCouncil.py (1)

43-43: LGTM!

Padding the UPRN to 12 digits ensures compatibility with the address dropdown's expected value format. This aligns with similar patterns in other council scrapers.

uk_bin_collection/uk_bin_collection/councils/NorwichCityCouncil.py (1)

72-91: LGTM!

Good improvements: the extraction now correctly targets ul elements (fixing what appears to be a typo from u1), adds defensive length checking, and properly strips whitespace from extracted text.

uk_bin_collection/uk_bin_collection/councils/BirminghamCityCouncil.py (1)

98-117: LGTM! Cleaner extraction logic.

The simplified parsing with direct extraction from th and td elements is cleaner. The more specific table selector (class="data-table") reduces fragility.

Consider adding defensive checks if the table structure might vary (e.g., empty tbody), though this may be unnecessary if the upstream page structure is stable.

uk_bin_collection/uk_bin_collection/councils/WiltshireCouncil.py (1)

100-108: LGTM on the new extraction approach.

The refactored logic correctly extracts collection dates from the data-cal-date attribute and handles multiple collection types per event by splitting on " and ".

uk_bin_collection/uk_bin_collection/councils/WinchesterCityCouncil.py (4)

19-38: Good documentation addition.

The detailed docstring clearly documents parameters, return value, and exceptions. This improves maintainability.

---

80-96: Resilient CSS selectors using contains-based matching.

Using lambda-based class checks with in operators makes the parsing more resilient to minor CSS class name changes. This is a solid defensive approach.

---

98-105: Defensive validation for the collections container.

The check prevents silent failures when the page structure changes unexpectedly.

---

110-134: Robust card parsing with proper null checks.

The defensive checks for missing h3 and date elements (lines 112-114, 118-120) prevent crashes on malformed cards. The year rollover logic correctly handles collections spanning the year boundary.

uk_bin_collection/uk_bin_collection/councils/WestOxfordshireDistrictCouncil.py (1)

57-63: XPath selector updated for changed UI structure.

The XPath change from dropdown-element-23 to combobox-input-23-1-23 aligns with updates to the council's web page. Ensure this has been tested against the live site.

uk_bin_collection/uk_bin_collection/councils/FifeCouncil.py (3)

73-86: Clean helper function for address matching.

The _best_option function provides clear, case-insensitive matching logic with good fallback behavior.

---

100-128: Table parsing with proper validation.

The parsing correctly validates the table exists, skips the header row, handles missing data gracefully, and extracts colour from image alt text.

---

134-142: Proper driver cleanup in finally block.

The try/except/finally pattern ensures the WebDriver is always closed, preventing resource leaks.

uk_bin_collection/uk_bin_collection/councils/BlackpoolCouncil.py (1)

84-88: Sorting and return logic looks correct.

The bins are properly sorted by collection date before returning.

uk_bin_collection/uk_bin_collection/councils/MertonCouncil.py (1)

68-74: LGTM - UPRN validation is thorough.

Good defensive programming with both presence and numeric-only validation for the UPRN parameter.

uk_bin_collection/uk_bin_collection/councils/HarlowCouncil.py (1)

43-56: LGTM - Collection parsing logic is sound.

The null check on line 47 properly guards against missing elements, and the date parsing correctly formats the output.

uk_bin_collection/uk_bin_collection/councils/MidSussexDistrictCouncil.py (1)

82-95: LGTM - Collection parsing and sorting logic.

The extraction of dates and bin types from the list items, along with the sorting by collection date, follows a clean pattern.

wiki/Councils.md (3)

675-685: LGTM - Blackpool documentation added correctly.

The new Blackpool council entry follows the established format with appropriate parameters and FindMyAddress reference.

---

1949-1958: LGTM - Harlow documentation added correctly.

The new Harlow council entry is consistent with other UPRN-based council entries.

---

2557-2564: LGTM - Mid Sussex documentation updated.

The removal of -w (Selenium webdriver) parameter aligns with the code refactor from Selenium to HTTP-based approach.

uk_bin_collection/uk_bin_collection/councils/NorthHertfordshireDistrictCouncil.py (4)

18-31: Hardcoded API credentials noted - verify this is acceptable.

The Basic auth header is extracted from a public mobile app. While this is documented and the approach is pragmatic, be aware that if the council rotates these credentials, the scraper will break. Consider adding a comment about monitoring for credential changes.

---

83-98: Address matching may miss some edge cases.

The paon matching checks if the house number equals the first word of an address line. This could miss cases like "Flat 1, Building Name" or "1A" vs "1". The error messages appropriately guide users to manually find their UPRN, which is a good fallback.

---

201-231: LGTM - Container processing is robust.

The iteration handles missing containers, empty dates, and invalid date formats gracefully by continuing to the next container. The internal _sort_date approach for sorting is clean.

---

233-247: LGTM - Clean output generation.

Raising an error when no valid bins are found provides clear feedback, and the list comprehension cleanly strips the internal sort key.

uk_bin_collection/uk_bin_collection/councils/IsleOfAngleseyCouncil.py (4)

1-19: LGTM!

Imports and constants are well-organized. The use of named constants for API endpoints and lookup IDs improves maintainability.

---

22-30: LGTM!

The class structure and session initialization are clean. Using requests.Session for connection pooling and cookie persistence is appropriate for this authentication flow.

---

45-70: LGTM!

The lazy session initialization pattern in _run_lookup is well-implemented. The debug logging on unexpected response structure is helpful for troubleshooting. The same JSONDecodeError note from the previous comment applies here (line 66).

---

120-165: LGTM!

The parse_data method has good input validation with clear error messages. Supporting both UPRN and postcode+PAON lookup provides flexibility for users. The UTC-aware date for the API payload is appropriate.

uk_bin_collection/tests/input.json (8)

232-240: LGTM!

BlackpoolCouncil entry is complete with all necessary fields. The UPRN-based approach with FindMyAddress reference follows the established pattern.

---

957-965: LGTM!

FifeCouncil updated to use postcode/house_number with Selenium driver. The wiki_note correctly reflects the new requirements.

---

1144-1151: LGTM!

HarlowCouncil entry is complete and follows the established pattern for UPRN-based lookups.

---

1279-1286: LGTM!

The IsleOfAngleseyCouncil test entry aligns well with the implementation. The skip_get_url: true is appropriate since the scraper uses its own session-based API calls rather than the inherited get_data method.

---

1537-1543: LGTM!

MertonCouncil migration to UPRN-based lookup is cleanly implemented with appropriate skip_get_url flag.

---

1733-1740: LGTM!

NorthHertfordshireDistrictCouncil updated to use the Cloud9 API endpoint with UPRN. The URL and uprn field are consistent.

---

2056-2061: LGTM!

RushmoorCouncil updated to use UPRN-based lookup with appropriate documentation.

---

1572-1579: LGTM!

MidSussexDistrictCouncil entry updated with simplified lookup approach.

uk_bin_collection/uk_bin_collection/councils/SouthamptonCityCouncil.py (3)

19-32: Docstring now accurately reflects UPRN-driven parsing

The updated docstring clearly describes the UPRN input and the returned bins structure, and matches the current implementation (including the fact that page is not used for extraction).

---

56-58: UPRN query parameter change – verify against live endpoint/tests

Switching to params = {"UPRN": user_uprn} is a sensible simplification, but it changes the exact query string the council endpoint sees. Please double-check this against the live form or existing integration tests to confirm the server still returns the correct calendar for the provided UPRN.

---

79-93: Result extraction and date sorting logic remain correct

The loop building bindata["bins"] and the subsequent sort by collectionDate (parsed as %d/%m/%Y) are consistent with the regex (%m/%d/%Y input → normalized DD/MM/YYYY output) and look correct.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Add timeout to initial request.

The initial session.get call also needs a timeout for consistency and reliability.

             r = session.get(
                 URI,
+                timeout=30,
             )

🤖 Prompt for AI Agents

In uk_bin_collection/uk_bin_collection/councils/MidSussexDistrictCouncil.py
around lines 32-35, the initial session.get call lacks a timeout which can hang;
add a timeout parameter to that call (e.g., timeout=10) or use the
module/project-wide DEFAULT_TIMEOUT constant if one exists so the request uses
the same timeout policy as other requests in the file, and ensure tests/usage
reflect the chosen timeout value.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Add timeout to HTTP request.

The requests.get call lacks a timeout, which could cause the scraper to hang indefinitely if the server is unresponsive.

Apply this diff:

         response = requests.get(
-            "https://www.rushmoor.gov.uk/Umbraco/Api/BinLookUpWorkAround/Get?",
+            "https://www.rushmoor.gov.uk/Umbraco/Api/BinLookUpWorkAround/Get",
             params=params,
             headers=headers,
+            timeout=30,
         )

Note: The trailing ? in the URL is also unnecessary since params handles query string construction.

🧰 Tools

🪛 Ruff (0.14.7)

29-29: requests may be undefined, or defined from star imports

(F405)

🤖 Prompt for AI Agents

In uk_bin_collection/uk_bin_collection/councils/RushmoorCouncil.py around lines
29 to 35, the requests.get call should include a timeout to avoid hanging and
the trailing '?' in the URL is unnecessary; update the call to use the base URL
without the '?' and pass a timeout (e.g. timeout=10) to requests.get (and
optionally keep existing params/headers as-is), and ensure you handle non-200
responses appropriately (e.g. response.raise_for_status() or check status_code)
after the call.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# First, find the get_dates_every_x_days function to understand its behavior
rg -n "def get_dates_every_x_days" --type=py -A 20

Repository: robbrad/UKBinCollectionData

Length of output: 1933

---

🏁 Script executed:

# Check the context around lines 36-37 in ThurrockCouncil.py to see how these dates are used
fd "ThurrockCouncil.py" --type f | xargs cat -n | head -100

Repository: robbrad/UKBinCollectionData

Length of output: 3656

---

🏁 Script executed:

# Look for any date filtering logic that might remove past dates
rg -n "filter|datetime\.now\(\)|today\(\)|strptime" uk_bin_collection/uk_bin_collection/councils/ThurrockCouncil.py -C 3

Repository: robbrad/UKBinCollectionData

Length of output: 1022

---

Update start dates to 2026 to avoid including past collection dates in results.

The hard-coded start dates (November 17 and 24, 2025) are now in the past. Since get_dates_every_x_days generates 28 future dates without filtering past dates, the first ~2 weeks of the returned collection dates will be historical. Update both bluebrownstartDate and greengreystartDate to November 2026 to ensure results contain only current and future collection schedules.

🧰 Tools

🪛 Ruff (0.14.7)

36-36: datetime may be undefined, or defined from star imports

(F405)

---

37-37: datetime may be undefined, or defined from star imports

(F405)

🤖 Prompt for AI Agents

In uk_bin_collection/uk_bin_collection/councils/ThurrockCouncil.py around lines
36-37 the hard-coded start dates are set to datetime(2025, 11, 17) and
datetime(2025, 11, 24), which are now in the past; update both
bluebrownstartDate and greengreystartDate to use November 2026 (datetime(2026,
11, 17) and datetime(2026, 11, 24)) so get_dates_every_x_days produces only
current/future collection dates. Ensure the datetime imports are correct and run
tests to verify schedules begin in 2026.

[robbrad]

@coderabbitai generate docstrings

[coderabbitai]

✅ Actions performed

Initiated docstring generation; will generate only if new commits exist.

[coderabbitai]

Note

Docstrings generation - SUCCESS
Generated docstrings for this pull request at #1755