POST requests to wfs server failing

[ateucher]

FWIW I am experiencing the same errors using the previous and latest package versions (0.4.1 and 0.5.0).

Looks to only be the queries to the WFS server.

bcdata::bcdc_get_record("78ec5279-4534-49a1-97e8-9d315936f08b")
B.C. Data Catalogue Record: School Districts of BC
Name: school-districts-of-bc (ID:
 78ec5279-4534-49a1-97e8-9d315936f08b)
Permalink:
 https://catalogue.data.gov.bc.ca/dataset/78ec5279-4534-49a1-97e8-9d315936f08b
Licence: Open Government Licence - British Columbia
Description: The School Districts dataset contains the
 spatial representation (polygon) of the current extent of the
 administrative areas defined under section 176(1) of the School Act for
 the purposes of preservation and promotion of the fundamental principle
 of local autonomy and control of public education at the public and
 governmental levels through locally elected school boards.
Available Resources (1):
 1. WMS getCapabilities request (wms)
Access the full 'Resources' data frame using:
 bcdc_tidy_resources('78ec5279-4534-49a1-97e8-9d315936f08b')
Query and filter this data using:
 bcdc_query_geodata('78ec5279-4534-49a1-97e8-9d315936f08b')

bcdc_query_geodata('78ec5279-4534-49a1-97e8-9d315936f08b')
Error: There was an issue sending this WFS request
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
Request:
  URL: https://openmaps.gov.bc.ca/geo/pub/wfs
  POST fields:
    SERVICE=WFS&VERSION=2.0.0&REQUEST=DescribeFeatureType&typeNames=WHSE_TANTALIS.TA_SCHOOL_DISTRICTS_SVW
  Content-Type: application/x-www-form-urlencoded
  Accept-Encoding: gzip, deflate
  Accept: application/json, text/xml, application/xml, */*
  User-Agent: https://github.com/bcgov/bcdata
Response:
  status: HTTP/1.1 500 Internal Server Error
  content-type: application/json; charset=utf-8
  transfer-encoding: chunked
  vary: Origin
  vary: Access-Control-Request-Method
  vary: Access-Control-Request-Headers
  date: Wed, 18 Dec 2024 20:04:44 GMT
  server: uvicorn
  x-kong-upstream-latency: 3
  x-kong-proxy-late
 

Originally posted by @achubaty in #347

[ateucher]

This is a WFS issue: replicated using curl:

curl -v -d "SERVICE=WFS&VERSION=2.0.0&REQUEST=DescribeFeatureType&typeNames=WHSE_LAND_AND_NATURAL_RESOURCE.PROT_HISTORICAL_FIRE_POLYS_SP" https://openmaps.gov.bc.ca/geo/pub/wfs

* Host openmaps.gov.bc.ca:443 was resolved.
* IPv6: (none)
* IPv4: 142.34.229.4
*   Trying 142.34.229.4:443...
* Connected to openmaps.gov.bc.ca (142.34.229.4) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=CA; ST=British Columbia; L=Victoria; O=Government of the Province of British Columbia; CN=*.data.gov.bc.ca
*  start date: Oct  1 15:16:12 2024 GMT
*  expire date: Oct 16 15:16:11 2025 GMT
*  subjectAltName: host "openmaps.gov.bc.ca" matched cert's "openmaps.gov.bc.ca"
*  issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2012 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1K
*  SSL certificate verify ok.
* using HTTP/1.x
> POST /geo/pub/wfs HTTP/1.1
> Host: openmaps.gov.bc.ca
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 124
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 124 bytes
< HTTP/1.1 500 Internal Server Error
< content-type: text/plain; charset=utf-8
< content-length: 21
< vary: Origin
< vary: Access-Control-Request-Method
< vary: Access-Control-Request-Headers
< date: Wed, 18 Dec 2024 20:12:01 GMT
< server: uvicorn
< x-kong-upstream-latency: 2
< x-kong-proxy-latency: 1
<
* Connection #0 to host openmaps.gov.bc.ca left intact
Internal Server Error%

[smnorris]

Can confirm no issues with GET requests made by Python bcdata.

$ bcdata dump -v WHSE_TANTALIS.TA_SCHOOL_DISTRICTS_SVW > t.geojson
2024-12-18 12:09:30,389:INFO:bcdata.wfs: Total features requested: 59
2024-12-18 12:09:40,982:INFO:bcdata.wfs: https://openmaps.gov.bc.ca/geo/pub/wfs?service=WFS&version=2.0.0&request=GetFeature&typeName=WHSE_TANTALIS.TA_SCHOOL_DISTRICTS_SVW&outputFormat=json&SRSNAME=EPSG%3A4326&count=59

[ateucher]

Thanks @smnorris - do you use POST requests in python bcdata anywhere?

[ateucher]

Confirms working with GET via curl:

curl -v "https://openmaps.gov.bc.ca/geo/pub/wfs?SERVICE=WFS&VERSION=2.0.0&REQUEST=DescribeFeatureType&typeNames=WHSE_LAND_AND_NATURAL_RESOURCE.PROT_HISTORICAL_FIRE_POLYS_SP"

* Host openmaps.gov.bc.ca:443 was resolved.
* IPv6: (none)
* IPv4: 142.34.229.4
*   Trying 142.34.229.4:443...
* Connected to openmaps.gov.bc.ca (142.34.229.4) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=CA; ST=British Columbia; L=Victoria; O=Government of the Province of British Columbia; CN=*.data.gov.bc.ca
*  start date: Oct  1 15:16:12 2024 GMT
*  expire date: Oct 16 15:16:11 2025 GMT
*  subjectAltName: host "openmaps.gov.bc.ca" matched cert's "openmaps.gov.bc.ca"
*  issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2012 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1K
*  SSL certificate verify ok.
* using HTTP/1.x
> GET /geo/pub/wfs?SERVICE=WFS&VERSION=2.0.0&REQUEST=DescribeFeatureType&typeNames=WHSE_LAND_AND_NATURAL_RESOURCE.PROT_HISTORICAL_FIRE_POLYS_SP HTTP/1.1
> Host: openmaps.gov.bc.ca
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< content-type: application/gml+xml; version=3.2
< transfer-encoding: chunked
< vary: Origin
< vary: Access-Control-Request-Method
< vary: Access-Control-Request-Headers
< content-disposition: inline; filename=pub-WHSE_LAND_AND_NATURAL_RESOURCE.PROT_HISTORICAL_FIRE_POLYS_SP.xsd
< date: Wed, 18 Dec 2024 20:14:39 GMT
< x-kong-upstream-latency: 11
< x-kong-proxy-latency: 0
<
<?xml version="1.0" encoding="UTF-8"?><xsd:schema xmlns:gml="http://www.opengis.net/gml/3.2" xmlns:pub="http://delivery.openmaps.gov.bc.ca/geo/" xmlns:wfs="http://www.opengis.net/wfs/2.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" targetNamespace="http://delivery.openmaps.gov.bc.ca/geo/">
  <xsd:import namespace="http://www.opengis.net/gml/3.2" schemaLocation="https://openmaps.gov.bc.ca/geo/schemas/gml/3.2.1/gml.xsd"/>
  <xsd:complexType name="WHSE_LAND_AND_NATURAL_RESOURCE.PROT_HISTORICAL_FIRE_POLYS_SPType">
    <xsd:complexContent>
      <xsd:extension base="gml:AbstractFeatureType">
        <xsd:sequence>
          <xsd:element maxOccurs="1" minOccurs="1" name="FIRE_NUMBER" nillable="false" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="VERSION_NUMBER" nillable="true" type="xsd:decimal"/>
          <xsd:element maxOccurs="1" minOccurs="1" name="FIRE_YEAR" nillable="false" type="xsd:decimal"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="FIRE_CAUSE" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="FIRE_LABEL" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="FIRE_SIZE_HECTARES" nillable="true" type="xsd:decimal"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="SOURCE" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="GPS_TRACK_DATE" nillable="true" type="xsd:date"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="LOAD_DATE" nillable="true" type="xsd:date"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="FIRE_DATE" nillable="true" type="xsd:date"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="CREATION_METHOD" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="FEATURE_CODE" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="SHAPE" nillable="true" type="gml:GeometryPropertyType"/>
          <xsd:element maxOccurs="1" minOccurs="1" name="OBJECTID" nillable="false" type="xsd:decimal"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="SE_ANNO_CAD_DATA" nillable="true" type="xsd:hexBinary"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="FEATURE_AREA_SQM" nillable="true" type="xsd:decimal"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="FEATURE_LENGTH_M" nillable="true" type="xsd:decimal"/>
        </xsd:sequence>
      </xsd:extension>
    </xsd:complexContent>
  </xsd:complexType>
  <xsd:element name="WHSE_LAND_AND_NATURAL_RESOURCE.PROT_HISTORICAL_FIRE_POLYS_SP" substitutionGroup="gml:AbstractFeature" type="pub:WHSE_LAND_AND_NATURAL_RESOURCE.PROT_HISTORICAL_FIRE_POLYS_SPType"/>
</xsd:schema>
* Connection #0 to host openmaps.gov.bc.ca left intact

[smnorris]

I don't think so - all tests pass.

[ateucher]

I've opened a ticket with DSS 🤞

[ateucher]

DSS has fixed the service, POST requests now working again, verified in bcdata and on the command line

[achubaty]

thank you for the quick fix!

[achubaty]

@ateucher the issue persists for some requests but not others:

> bcdata::bcdc_get_data("0bc73892-e41f-41d0-8d8e-828c16139337") ## works

> bcdata::bcdc_get_data("61044e1a-cd80-4ed6-9f95-907262b9910f") ## fails
Error: There was an issue sending this WFS request
2 paginated requests issued

> bcdata::bcdc_get_data("11277e35-d8be-47e4-bb1f-c38e393179c6") ## fails with more info
Error: There was an issue sending this WFS request
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
Request:
  URL: https://openmaps.gov.bc.ca/geo/pub/wfs
  POST fields:
    SERVICE=WFS&VERSION=2.0.0&REQUEST=DescribeFeatureType&typeNames=WHSE_LAND_USE_PLANNING.RMP_LANDSCAPE_UNIT_SVW
  Content-Type: application/x-www-form-urlencoded
  Accept-Encoding: gzip, deflate
  Accept: application/json, text/xml, application/xml, */*
  User-Agent: https://github.com/bcgov/bcdata
Response:
  status: HTTP/1.1 500 Internal Server Error
  content-type: application/json
  transfer-encoding: chunked
  vary: Origin
  vary: Access-Control-Request-Method
  vary: Access-Control-Request-Headers
  x-kong-upstream-latency: 30002
  x-kong-proxy-latency: 0

[stephhazlitt]

@achubaty I cannot reproduce the error

bcdata::bcdc_query_geodata("61044e1a-cd80-4ed6-9f95-907262b9910f")
#> Querying 'natural-disturbance-type-map' record
#> • Using collect() on this object will return 15666 features and 8 fields
#> • Accessing this record requires pagination and will make 2 separate
#> • requests to the WFS. See ?bcdc_options
#> • At most six rows of the record are printed here
#> ────────────────────────────────────────────────────────────────────────────────
#> Simple feature collection with 6 features and 8 fields
#> Geometry type: POLYGON
#> Dimension:     XY
#> Bounding box:  xmin: 902336.9 ymin: 995587.5 xmax: 953437.5 ymax: 1255038
#> Projected CRS: NAD83 / BC Albers
#> # A tibble: 6 × 9
#>   id            FEATURE_CLASS_SKEY NATURAL_DISTURBANCE_…¹ NATURAL_DISTURBANCE_…²
#>   <chr>                      <int> <chr>                  <chr>                 
#> 1 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> 2 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> 3 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> 4 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> 5 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> 6 WHSE_FOREST_…                435 NDT2                   Ecosystems with infre…
#> # ℹ abbreviated names: ¹​NATURAL_DISTURBANCE_TYPE_CODE,
#> #   ²​NATURAL_DISTURBANCE_TYPE_DESC
#> # ℹ 5 more variables: FEATURE_AREA_SQM <dbl>, FEATURE_LENGTH_M <dbl>,
#> #   OBJECTID <int>, SE_ANNO_CAD_DATA <chr>, geometry <POLYGON [m]>

> session_info()
─ Session info ─────────────────────────────────────────
 setting  value
 version  R version 4.4.2 (2024-10-31)
 os       macOS Sonoma 14.4.1
 system   aarch64, darwin20
 ui       RStudio
 language (EN)
 collate  en_US.UTF-8
 ctype    en_US.UTF-8
 tz       America/Vancouver
 date     2024-12-19
 rstudio  2024.12.0+467 Kousa Dogwood (desktop)
 pandoc   3.2 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)

─ Packages ─────────────────────────────────────────────
 package     * version  date (UTC) lib source
 bcdata      * 0.5.0    2024-12-12 [1] CRAN (R 4.4.1)