Skip to content

May 2026 Release#1996

Merged
robbrad merged 5 commits into
masterfrom
may-2026-release
May 2, 2026
Merged

May 2026 Release#1996
robbrad merged 5 commits into
masterfrom
may-2026-release

Conversation

@robbrad
Copy link
Copy Markdown
Owner

@robbrad robbrad commented May 2, 2026

May 2026 Release — 63 PRs Consolidated

All open PRs merged into a single release branch with merge conflicts resolved.

Council Fixes (56 PRs)

PR Council(s) Change
#1899 Babergh, MidSuffolk, Rother, Wirral Date parsing, null handling, input.json
#1909 EastSuffolk Fix collection parsing
#1916 Southampton Include food waste collections
#1918 Blaby Add User-Agent header (403 fix)
#1919 Brent Add User-Agent header
#1920 Swale Add user_agent to webdriver
#1921 SouthHams Init fcc_session_token before try
#1922 Oxford Handle multiple editor divs
#1923 ReigateAndBanstead Select populated span
#1924 Shropshire Use full link text for bin type
#1925 Salford Use lxml parser for malformed HTML
#1926 Rushmoor Handle raw JSON response
#1927 Plymouth Fix garden collection
#1929 Crawley Add XML response parsing fallback
#1930 Newham Updated card selectors, date format fallback
#1932 Torridge Parse new relative-date SOAP format
#1933 Peterborough user_agent and container selector
#1934 Horsham user_agent and postcode ID selector
#1935 API server Fix double-encoded JSON string
#1936 NeathPortTalbot user_agent and JS click for overlay
#1937 Bexley Replace Selenium with requests
#1938 Stirling Replace Selenium with Recollect API
#1939 WindsorAndMaidenhead Replace Selenium with requests
#1940 EppingForest Replace Selenium with ArcGIS REST API
#1941 Moray Rewrite for annual calendar
#1942 EastLothian Rewrite for Drupal migration
#1943 Barnet Rewrite for Jadu migration
#1944 NorthNorfolk Rewrite for X-Forms
#1947 63 councils Add realistic user_agent to all Selenium scrapers
#1948 Richmond Rewrite for My Richmond page
#1949 NorthYorkshire Find HTML data dynamically in AJAX
#1950 GreatYarmouth Use current year instead of hardcoded 2025
#1952 Wyre Skip non-bin divs, harden date parsing
#1953 Chichester Rewrite for AchieveForms V7
#1954 StocktonOnTees Update form IDs to V2, dismiss cookie banner
#1955 WestSuffolk Send User-Agent header
#1956 Midlothian Rewrite for MyMidlothian Granicus
#1957 SouthStaffordshire Use objectId query param
#1958 Chorley, Islington, Medway, Hastings, Conwy Reliability sweep
#1971 Tewkesbury Handle new API response format
#1972 Boston Drive chosen.js widget
#1973 Chelmsford Dismiss email-signup modal
#1974 ArmaghBanbridgeCraigavon Full browser User-Agent for WAF
#1975 NorthHertfordshire Migrate to Netcall Liberty Create
#1978 WestSuffolk User-Agent and renamed h4 heading
#1979 EnvironmentFirst, Eastbourne, Lewes Handle restructured page
#1980 MidSussex Migrate to Whitespace portal
#1981 Rotherham Rewrite for Imactivate API
#1983 SouthOxfordshire Session establishment and bin type cleanup
#1984 Chelmsford Replace Selenium with requests
#1985 MidSuffolk Replace sleep with WebDriverWait
#1987 NorthKesteven Make own HTTP request
#1988 Fareham Switch to calendar page
#1989 EastRiding Replace Selenium with JSON API
#1990 Edinburgh Accept real postcode and house number
#1991 Thurrock Accept real postcode and house number
#1993 Broxtowe Fall back to house number text match
#1994 Uttlesford House number prefix match in dropdown
#1995 Stirling Resolve street from postcode for Recollect

Bug Fixes

Dependencies (4 PRs)

PR Change
#1963 Bump pillow 12.1.1 to 12.2.0
#1967 Bump mako 1.3.10 to 1.3.11
#1976 Bump python-dotenv 1.1.0 to 1.2.2
#1977 Bump lxml 5.3.2 to 6.1.0

Issues Resolved (23)

Closes #1555, #1616, #1725, #1768, #1804, #1806, #1889, #1890, #1893, #1902, #1904, #1906, #1908, #1910, #1911, #1914, #1931, #1946, #1951, #1962, #1966, #1970, #1982

Code Review Notes

  • No malicious code found
  • No eval/exec/subprocess/os.system calls added
  • No hardcoded credentials or API keys
  • All 334 council files compile cleanly

Summary by CodeRabbit

  • Bug Fixes
    • Improved address selection logic with enhanced fallback matching for more reliable lookups
    • Strengthened address search capabilities using smarter query generation and reverse-geocoding
    • Implemented case-insensitive address matching to handle formatting variations
    • Enhanced error handling with clearer feedback when addresses cannot be located

InertiaUK and others added 5 commits May 1, 2026 22:22
… dropdown

The scraper selected addresses by UPRN value in the dropdown, which fails
when the resolved UPRN doesn't exactly match one of the council's dropdown
option values. Now tries UPRN match first, then falls back to matching the
dropdown option text by house number prefix.
…ch in dropdown

select_by_visible_text fails when the dropdown text includes the street
name (e.g. "72, Birchanger Lane") but only the house number is provided.
Now tries exact match first, then iterates options matching by house
number prefix.
…umber provided

The Recollect address-suggest API needs a street name to find a specific
property, but callers often only have a house number + postcode. Added
postcodes.io + Nominatim reverse geocoding to resolve the street name
from the postcode coordinates, then builds search queries combining the
house number with the resolved street.
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 2, 2026

📝 Walkthrough

Walkthrough

This PR improves address dropdown matching robustness across three council handlers (Broxtowe, Stirling, Uttlesford) by introducing fallback matching strategies that first attempt exact matches or UPRN-based selection, then resort to case-insensitive prefix matching on house numbers. Stirling additionally adds helper functions and constants for enhanced address query building using postcode-to-coordinate and reverse-geocoding lookups.

Changes

Address Matching Robustness Across Council Handlers

Layer / File(s) Summary
Constants & Helpers
StirlingCouncil.py
Added module-level NOMINATIM_URL and HEADERS constants; introduced _extract_street_from_paon() and _build_queries() to construct ranked search queries via postcode lookup, reverse geocoding, and street extraction.
Dropdown Selection Logic
BroxtoweBoroughCouncil.py, UttlesfordDistrictCouncil.py, StirlingCouncil.py
Updated address matching to wait for dropdown availability, attempt exact/UPRN-based selection, then fallback to case-insensitive prefix matching on normalized house numbers; raises ValueError if no match is found.
Integration & Headers
StirlingCouncil.py
parse_data now uses _build_queries() for address search; replaced per-request headers with shared HEADERS constant for both address-suggest and events API calls.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Poem

🐰 Three councils, one quest—to find the right bin day!
With fallback matching and prefixes at play,
No more exact chaos, just "close enough" grace,
House numbers flexing across every place.
Hops away with improved address cheer! 🥕✨

🚥 Pre-merge checks | ✅ 1 | ❌ 4

❌ Failed checks (1 warning, 3 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check ❓ Inconclusive The title 'May 2026 Release' is generic and vague, not describing specific changes, using a non-descriptive term that lacks meaningful information about the changeset. Consider a more descriptive title that highlights the main changes, such as 'Address matching improvements for Broxtowe, Stirling, and Uttlesford councils' or similar.
Linked Issues check ❓ Inconclusive The PR objectives mention issue #1555 about calendar setup and async_config_entry_first_refresh causing unavailable sensors, but the code changes only address address-matching logic in three council files without addressing the calendar initialization issue. Clarify whether the calendar.py changes referenced in issue #1555 are included in this PR, or if the address-matching fixes are separate from that issue's requirements.
Out of Scope Changes check ❓ Inconclusive The PR changes focus on address-matching improvements for three specific councils (Broxtowe, Stirling, Uttlesford), which align with the stated PR objectives of council-specific fixes, but issue #1555 concerns calendar setup that is not addressed by the shown code changes. Verify that the calendar.py fix mentioned in issue #1555 is included in the full PR, or confirm these address-matching changes are separate deliverables from that issue.
✅ Passed checks (1 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch may-2026-release

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
Review rate limit: 0/1 reviews remaining, refill in 60 minutes.

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 2, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.67%. Comparing base (d661bf1) to head (2f4182e).
⚠️ Report is 7 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1996   +/-   ##
=======================================
  Coverage   86.67%   86.67%           
=======================================
  Files           9        9           
  Lines        1141     1141           
=======================================
  Hits          989      989           
  Misses        152      152           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
uk_bin_collection/uk_bin_collection/councils/BroxtoweBoroughCouncil.py (1)

59-66: 💤 Low value

Consider logging the UPRN selection failure for debugging.

The silent except: pass pattern makes debugging difficult when UPRN matching fails unexpectedly. While the fallback logic is sound, adding a debug log would help diagnose issues.

📝 Suggested improvement
             if user_uprn:
                 try:
                     dropdownSelect.select_by_value("U" + user_uprn)
                     matched = True
-                except Exception:
-                    pass
+                except Exception as e:
+                    # Expected if UPRN not found; fall back to house number match
+                    pass

Or, if logging is available in scope:

except Exception:
    logging.debug(f"UPRN '{user_uprn}' not found in dropdown, falling back to house number match")
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@uk_bin_collection/uk_bin_collection/councils/BroxtoweBoroughCouncil.py`
around lines 59 - 66, The current silent except in the UPRN selection block
(around dropdownSelect.select_by_value("U" + user_uprn") in
BroxtoweBoroughCouncil.py) should log failures for debugging: replace the bare
except with an exception handler that emits a debug-level log including the
user_uprn and the exception details (e.g., logging.debug or
logging.exception/with exc_info=True) and then falls back to the house-number
matching logic while leaving matched=False; ensure you reference
dropdownSelect.select_by_value and user_uprn in the log message so the failure
is traceable.
uk_bin_collection/uk_bin_collection/councils/StirlingCouncil.py (1)

53-68: Add error logging or comments to silent exception handler.

The silent except Exception: pass at lines 67–68 swallows failures from both the postcodes.io API call and Nominatim reverse geocoding, making debugging difficult. While the code includes a proper User-Agent header (satisfying Nominatim's policy requirements), the exception handling lacks visibility into why geocoding fails.

The implementation calls Nominatim at most once per parse_data() invocation, so the 1-request-per-second rate limit is not a practical concern for typical use.

Suggested improvement

Add a comment to clarify intent:

         except Exception:
+            # Geocoding fallback failed; continue with other query strategies
             pass

Or if logging is preferred:

except Exception as e:
    logging.debug(f"Geocoding fallback failed for postcode {postcode}: {e}")
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@uk_bin_collection/uk_bin_collection/councils/StirlingCouncil.py` around lines
53 - 68, The broad except in parse_data() around the postcodes.io and Nominatim
reverse call is swallowing errors; update the except Exception: pass to either
log the failure or document the intent—capture the exception as e and call
logging.debug or logging.error with context including postcode, lat/lng and e
(e.g. refer to rev_resp, NOMINATIM_URL, HEADERS, paon, lat, lng) so failures in
the geocoding fallback are visible, or replace the pass with a clear comment
stating that failures are intentionally ignored.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@uk_bin_collection/uk_bin_collection/councils/BroxtoweBoroughCouncil.py`:
- Around line 59-66: The current silent except in the UPRN selection block
(around dropdownSelect.select_by_value("U" + user_uprn") in
BroxtoweBoroughCouncil.py) should log failures for debugging: replace the bare
except with an exception handler that emits a debug-level log including the
user_uprn and the exception details (e.g., logging.debug or
logging.exception/with exc_info=True) and then falls back to the house-number
matching logic while leaving matched=False; ensure you reference
dropdownSelect.select_by_value and user_uprn in the log message so the failure
is traceable.

In `@uk_bin_collection/uk_bin_collection/councils/StirlingCouncil.py`:
- Around line 53-68: The broad except in parse_data() around the postcodes.io
and Nominatim reverse call is swallowing errors; update the except Exception:
pass to either log the failure or document the intent—capture the exception as e
and call logging.debug or logging.error with context including postcode, lat/lng
and e (e.g. refer to rev_resp, NOMINATIM_URL, HEADERS, paon, lat, lng) so
failures in the geocoding fallback are visible, or replace the pass with a clear
comment stating that failures are intentionally ignored.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 01117015-d562-4ef0-9fbf-0d8558a015d1

📥 Commits

Reviewing files that changed from the base of the PR and between d661bf1 and 2f4182e.

📒 Files selected for processing (3)
  • uk_bin_collection/uk_bin_collection/councils/BroxtoweBoroughCouncil.py
  • uk_bin_collection/uk_bin_collection/councils/StirlingCouncil.py
  • uk_bin_collection/uk_bin_collection/councils/UttlesfordDistrictCouncil.py

@robbrad robbrad merged commit d8e25ac into master May 2, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Calendar setup async_config_entry_first_refresh causing unavailable sensors

2 participants