Skip to content

fix: NorthHertfordshireDistrictCouncil - migrate to Netcall Liberty Create portal#1975

Closed
InertiaUK wants to merge 1 commit into
robbrad:masterfrom
InertiaUK:fix/northherts-netcall-migration
Closed

fix: NorthHertfordshireDistrictCouncil - migrate to Netcall Liberty Create portal#1975
InertiaUK wants to merge 1 commit into
robbrad:masterfrom
InertiaUK:fix/northherts-netcall-migration

Conversation

@InertiaUK
Copy link
Copy Markdown
Contributor

Problem

The old scraper targeted https://apps.cloud9technologies.com/northherts/citizenmobile/mobileapi/{UPRN}/ which now returns HTTP 404. NHDC has moved its bin-collection lookup to a Netcall Liberty Create portal at:

https://waste.nc.north-herts.gov.uk/w/webpage/find-bin-collection-day-input-address

There is no public UPRN shortcut on the new portal — submit URLs carry an auth signature bound to the session-issued webpage_token (direct GET with ?id=<record_id> returns 403 Forbidden). The form is also entirely client-rendered (initial HTML contains no <form>), so a pure-requests implementation would have to replay ~30 Liberty presenter params per call and is extremely fragile. Selenium is the pragmatic path — the old fixture already declared web_driver: http://selenium:4444 anyway.

Fix

Selenium-driven typeahead flow:

  1. driver.get the input form page.
  2. Type postcode into input.relation_path_type_ahead_search.
  3. Wait for div.relation_path_type_ahead_results_holder ul.result_list li[data-id] to render.
  4. Pick the <li> whose text contains the paon (case-insensitive substring match; falls back to the first result if the caller doesn't provide a house-number match).
  5. Click input[type=submit][aria-label='Select address and continue'].
  6. Wait for the show-details result URL, then parse bin-type headings (Cardboard & Paper / Food Waste / Non-Recyclable Waste / Mixed Recycling / Garden Waste / Refuse / Glass) paired with the following Next collection date. Strip ordinal suffixes and parse %A %d %B %Y.

Fixture change

Required — the old fixture was UPRN-only and the new portal doesn't accept UPRN input. Updated to:

"NorthHertfordshireDistrictCouncil": {
    "house_number": "Stewards Flat",
    "postcode": "SG5 1PZ",
    "skip_get_url": true,
    "url": "https://waste.nc.north-herts.gov.uk/w/webpage/find-bin-collection-day-input-address",
    "web_driver": "http://selenium:4444",
    "wiki_name": "North Hertfordshire",
    "wiki_note": "Pass a postcode (with space) and house_number/name. The scraper performs the Liberty Create typeahead lookup and matches house_number as a case-insensitive substring.",
    "LAD24CD": "E07000099"
}

Verification

Live run against the fixture returns 4 bins:

Cardboard & Paper      23/04/2026
Food Waste             23/04/2026
Non-Recyclable Waste   30/04/2026
Mixed Recycling        08/05/2026

…reate portal

NHDC has moved its bin-collection lookup from
  apps.cloud9technologies.com/northherts/citizenmobile/mobileapi/{UPRN}/
(now returns 404) to
  waste.nc.north-herts.gov.uk/w/webpage/find-bin-collection-day-input-address
which is a Netcall Liberty Create portal.

There is no public UPRN shortcut on the new portal - submit URLs carry
an auth signature bound to the session-issued webpage_token (direct
GET with id=<record_id> returns 403). The form is also entirely
client-rendered (the initial HTML contains no <form>), so a requests-
only implementation would have to replay 30+ presenter params and is
extremely fragile. Selenium is the pragmatic path and the old fixture
already declared web_driver: http://selenium:4444 anyway.

New scraper flow:
  1. driver.get the input form page.
  2. Type postcode into input.relation_path_type_ahead_search.
  3. Wait for div.relation_path_type_ahead_results_holder ul.result_list
     li[data-id] to render.
  4. Pick the li whose text contains the paon (case-insensitive
     substring match; falls back to the first result if the caller
     doesn't provide a house-number match).
  5. Click input[type=submit][aria-label='Select address and continue'].
  6. Wait for the show-details result URL, then parse bin-type headings
     (Cardboard & Paper / Food Waste / Non-Recyclable Waste / Mixed
     Recycling / Garden Waste / Refuse / Glass) paired with the
     following 'Next collection' date. Strip ordinal suffixes and
     parse '%A %d %B %Y'.

Fixture updated:
  house_number='Stewards Flat', postcode='SG5 1PZ'.
Verified live: returns 4 bins (Cardboard & Paper 23/04/2026,
Food Waste 23/04/2026, Non-Recyclable Waste 30/04/2026,
Mixed Recycling 08/05/2026).
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Apr 20, 2026

Warning

Rate limit exceeded

@InertiaUK has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 52 minutes and 38 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 52 minutes and 38 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: de1e90c1-76f8-4e16-beb8-4a9913f91218

📥 Commits

Reviewing files that changed from the base of the PR and between 60bd3cc and a7b4df2.

📒 Files selected for processing (2)
  • uk_bin_collection/tests/input.json
  • uk_bin_collection/uk_bin_collection/councils/NorthHertfordshireDistrictCouncil.py
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@robbrad robbrad mentioned this pull request May 1, 2026
@robbrad
Copy link
Copy Markdown
Owner

robbrad commented May 1, 2026

Included in May 2026 Release PR #1992. Closing.

@robbrad robbrad closed this May 1, 2026
@robbrad robbrad mentioned this pull request May 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants