http-resource

A Concourse resource for interacting with HTTP REST APIs.

Table of Contents

• Source Configuration
• Behavior
  • check: Check for new resource version
  • in: Invoke the url, capturing the response
  • out: Invoke the url, (optionally) sending a payload body
• Examples
  • get a resource
  • get and put
  • get ignore some versions
• Version Strategies

Source Configuration

• url: Required. The HTTP URL, e.g. "https://some-server:8080/some/endpoint".

• username: Optional. Username for accessing an authenticated url.

• password: Optional. Password for accessing an authenticated url.

• insecure: Optional. Set to true to ignore SSL errors. Defaults to false.

• method: Optional. HTTP method to use. When not specified here, this defaults to GET for in and POST for out. Can also be overridden as a param to each get or put step.

• headers: Optional. Map of HTTP headers to include in the invocation. Defaults to empty (none). Additional headers can be specified as a params to each get or put step.

  source:
    headers:
      Accept: application/json

• file: Optional. File containing content to be sent as the body of the request. Cannot be specified at the same time as text. Can be overridden as a param to each get or put step.

• text: Optional. Inline text to be sent as the body of the request. Cannot be specified at the same time as file. Can be overridden as a param to each get or put step.

• data_binary: Optional. Whether to treat the file data as binary (rather than plain text). This has implications on the mode used in the put operation only. Can be overridden as a param to each put step. Default is false.

• build_metadata: Optional. List of component(s) to perform build metadata variable substitution on. Can be overridden as a param to each get or put step. By default, no substitution is performed on any components. Valid values include:

  • headers: Substitute any/all variables used in the headers.

    source:
      build_metadata: [headers]
      headers:
        X-Build-Logs: $ATC_EXTERNAL_URL/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME

  • body: Substitute any/all variables used in file or text content.

    source:
      build_metadata: [body]
      text: |
        See build logs at $ATC_EXTERNAL_URL/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME.

    Overriding in a step

    Configuring the build_metadata in a step completely overrides the source config. For example, the pipeline below will perform token replacements on the headers and body (if configured with one) during the check and get, but on neither during the put step.

    resources:
      - name: my-http-resource
        type: http-resource
        source:
          build_metadata: [headers,body]
    
    jobs:
      - name: post-something
        plan:
          - put: my-http-resource
            params:
              build_metadata: []

    Binary content is never attempted

    Substitution is also never performed on a body whose content is flagged as data_binary (regardless of what build_metadata says). For example, the put below will not attempt token replacements on the body since data_binary is true (but will still replace any tokens in the headers).

    resources:
      - name: my-http-resource
        type: http-resource
        source:
          build_metadata: [headers,body]
    
    jobs:
      - name: post-something
        plan:
          - put: my-http-resource
            params:
              data_binary: true

• version: Optional. How to determine the resource’s version (see Version Strategies). Default is to compute a hash digest of the response body (without the headers).

  • jq: A jq expression to select a value from the response body as the version.

    source:
      version:
        jq: .some.path.in.the.response.body

  • header: A response header to use as the version.

    source:
      version:
        header: Last-Modified

  • hash: Compute a digest from a particular component of the response. Options are:

    • headers - Compute the digest from the response headers only.

    • body - Compute the digest from the response body only.

    • full - Compute the digest from the response headers and body together.

      source:
        version:
          hash: body

  • default: The default version strategy to fall back on if none of the above are configured or cannot yield a non-empty value. Options are:

    • hash - (Default) Compute a hash digest of the response body.

    • none - Don’t emit any version. Concourse will ignore the check result.

• out_only: Optional. Disables the check and in operations, turning them into no-ops (including the implicit get after each put). Relevant for scenarios where you are only using put operations, or for any other reason you do not want to regularly check the endpoint. Default is false.

• sensitive: Optional. If true, the responses from the endpoint will be considered sensitive and not show up in the logs or Concourse UI. Can be overridden as a param to each get or put step. Default is false.

Behavior

check: Check for new resource version

Invokes the url and determines the current version according to the source version configuration.

ℹ️	No-op if source param out_only is true.

in: Invoke the url, capturing the response

Invokes the url and:

• writes the response headers to a file named headers.

• writes the response body to a file named body.

ℹ️	No-op if source param out_only is true.

Parameters

• strict: Optional. Whether to strictly assert the version retrieved matches the version requested (from check). Defaults to false. If set to true and the versions to do not match, the step will fail.

  💡	Not all endpoints may be able to provide an idempotent version, this configuration lets you decide how you want to handle those scenarios.

out: Invoke the url, (optionally) sending a payload body

General purpose invocation of the url, optionally sending a request body.

  plan:
    - put: my-http-resource
      params:
        build_metadata: [body]
        text: |
          The build had a result. Check it out at:
          $ATC_EXTERNAL_URL/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME
          or at:
          $ATC_EXTERNAL_URL/builds/$BUILD_ID

Examples

get a resource

Issue GET requests to https://httpbin.org/get, and display the response headers and body we get back.

resource_types:
  - name: http-resource
    type: docker-image
    source:
      repository: jgriff/http-resource

resources:
  - name: http-bin
    type: http-resource
    source:
      url: https://httpbin.org/get

jobs:
  - name: get-something
    plan:
      - get: http-bin
        trigger: true
      - task: take-a-look
        config:
          platform: linux
          image_resource:
            type: registry-image
            source: { repository: busybox }
          inputs:
            - name: http-bin
          run:
            path: cat
            args: ["http-bin/headers", "http-bin/body"]

get and put

GET a file, and POST it to another endpoint.

resource_types:
  - name: http-resource
    type: docker-image
    source:
      repository: jgriff/http-resource

resources:
  - name: http-bin-get
    type: http-resource
    source:
      url: https://httpbin.org/get
  - name: http-bin-post
    type: http-resource
    source:
      url: https://httpbin.org/post
      out_only: true                  (2)

jobs:
  - name: post-something
    plan:
      - get: http-bin-get
        trigger: true
      - put: http-bin-post
        params:
          file: http-bin-get/body     (1)

1. post the file content that was retrieved in the get step.

2. disable the implicit get after a put (since issuing a GET to https://httpbin.org/post returns a 405 METHOD NOT ALLOWED and will fail our pipeline).

get ignore some versions

In some scenarios, you may want to version on a response property or header that may not always be returned.

The default behavior for this would be to fallback to generating a hash of the response payload. However, if you would rather simply skip those missing versions all together, you can configure the default to none. This will cause check to omit that version.

For example, if we want to version only on responses that contain the structure:

{
  "usually": {
    "present": {
      "version": "some-version-value"
    }
  }
}

Then we can configure our pipeline as:

resource_types:
  - name: http-resource
    type: docker-image
    source:
      repository: jgriff/http-resource

resources:
  - name: volatile-endpoint
    type: http-resource
    source:
      url: https://someplace.io/anything
      version:
        jq: .usually.present.version  (1)
        default: none                 (2)

jobs:
  - name: get-good-version
    plan:
      - get: volatile-endpoint
        trigger: true
        params:
          strict: true                (3)

1. for versions we want, this attribute will be present in the response body.

2. ignores any response without our desired jq path

3. ensure we only process resource versions that strictly match our version requirements.

This also works nicely in fallback strategies.

    source:
      url: https://someplace.io/anything
      version:
        jq: .usually.present.version  (1)
        header: Might-Exist           (2)
        default: none                 (3)

1. Try a jq query first.

2. If that doesn’t match, check for a response header.

3. If neither of those match, then ignore the version.

For more details, see Version Strategies.

Version Strategies

By default, a hash digest of the response body is used as the version of the resource.

However, you can configure any/all of the version strategies together and they will be attempted in the following order:

1. jq

2. header

3. hash

4. default

The first one to yield a non-empty value will be used as the version.

If none of them can produce a non-empty string, then the configured default strategy is used (which defaults to a hash of the response body).

For example, suppose our endpoint returns the following response:

HTTP/1.1 200 OK
Content-Type: application/json
Some-Header: some-header-value
Version: 1

{
  "some": "response",
  "version": "abc-123"
}

The table below lists various examples for determining the version from this endpoint.

Table 1. Version Examples

Source Config	Yields
source: version: jq: .version	"abc-123"
source: version: header: Version	1
source: # no version config or source: version: hash: body or source: version: default: hash	(hash of response body)
source: version: hash: headers	(hash of response headers)
source: version: hash: full	(hash of response headers + body)

Table 2. Version Examples - Fallback Scenarios

Source Config	Yields
source: version: jq: .version # value header: Version # not tried	"abc-123"
source: version: jq: .does.not.exist # no value header: Version # value	1
source: version: jq: .does.not.exist # no value header: Does-Not-Exist # no value # none match, defaults to hash or source: version: jq: .does.not.exist # no value header: Does-Not-Exist # no value hash: body # value or source: version: jq: .does.not.exist # no value header: Does-Not-Exist # no value default: hash # default to hash	(hash of response body)
source: version: jq: .does.not.exist # no value header: Does-Not-Exist # no value default: none # default to no version	Yields no versions. Concourse will ignore the result of check.

⚠️	Configuring the version with only a default of none will never yield any version from check. source: version: default: none