Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
1229abc
Add diffstat case for deleted file
May 19, 2021
2c3c7d1
Remove unused import
May 19, 2021
36fb801
Bump package to minor release
May 19, 2021
f5a825d
Remove unused import
May 19, 2021
3038c00
Restrict Deno permissions by removing -A flag
May 20, 2021
03db417
Merge pull request #11 from githubocto/diffstat-fix
irealva May 21, 2021
0a33cd6
Strip http_url of secrets, add optional 'mask' param
May 24, 2021
cfb2cb1
Merge main
May 26, 2021
6283443
Merge pull request #18 from githubocto/deno-permissions
irealva May 26, 2021
ba146ea
Add boolean string to completely mask source string
May 26, 2021
5c3a038
Fix description of mask in action.yml
May 26, 2021
5ed3a51
Document new mask param
May 26, 2021
471ce76
Merge main
May 26, 2021
eeddd09
Merge pull request #26 from githubocto/secrets-mask
irealva May 26, 2021
de8614e
Bump major package because of Deno permission restrictions
May 26, 2021
e83e6ea
Fix action version in readme
May 26, 2021
a480644
feat: expose typeorm config object (#32)
mattrothenberg May 27, 2021
5d978df
docs: bump version number
May 27, 2021
ff73f51
chore(README): updated readme to include authorization header
4lch4 May 21, 2021
73580df
Only pass authorization header if value is not undefined
May 28, 2021
daa0b88
Add authorization yaml parameter
May 28, 2021
0b5a9cd
Merge main
May 28, 2021
157cd81
Log header
May 28, 2021
d0e6451
Remove authorization log
May 28, 2021
93da886
Add correct required param to authorization input
May 28, 2021
e59f125
Merge pull request #33 from githubocto/ia/authorization-header
irealva May 28, 2021
a49c8cd
Bump minor version with feat authorization header
May 28, 2021
b86f999
Update docs to use Deno 1.10.x as 1.11
GerryWilko Jun 9, 2021
2919de1
Merge pull request #46 from GerryWilko/patch-1
Jun 11, 2021
06b5f4b
fix: turn off color output for deno
Jul 12, 2021
1e23cb6
Merge pull request #51 from githubocto/ia/deno-color
irealva Jul 12, 2021
77ea764
bump version patch
Jul 12, 2021
9a17f4e
feat: add config option for Axios configuration
Jul 21, 2021
40dfcf4
fix: update readme
Jul 21, 2021
416fca1
Merge pull request #55 from githubocto/mr/ia/expose-axios-config
irealva Jul 22, 2021
fc34373
bump minor version
Jul 22, 2021
858c891
Create repo dispatch action
irealva Aug 26, 2021
dacf5f7
use node16, bump sqlite (#86)
mattrothenberg Jun 7, 2023
cea5ee4
version bump to 3.4.0
mattrothenberg Jun 7, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .github/workflows/dispatch.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
name: Repo Events Repository Dispatch

on:
- issues
- issue_comment
- pull_request

jobs:
preflight-job:
name: Dispatch
runs-on: ubuntu-latest
steps:
- name: Print Outputs
env:
outputs: ${{ toJSON(github) }}
run: |
echo outputs: $outputs
- name: Repository Dispatch
uses: peter-evans/repository-dispatch@v1
with:
token: ${{ secrets.PAT }}
repository: githubocto/next-devex-workflows # repo to send event to
event-type: repoevents # name of the custom event
client-payload: '{"event": ${{ toJSON(github) }}}'
75 changes: 67 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Check out our [example repositories](https://github.com/githubocto?q=flat-demo&t

### Option 1: Flat Editor VSCode Extension

The easiest way to get a Flat Data action up and running is with the accompanying [Flat Editor VSCode Extension](https://marketplace.visualstudio.com/items?itemName=GitHubOCTO.flat) which helps you author Flat yml files.
The easiest way to get a Flat Data action up and running is with the accompanying [Flat Editor VSCode Extension](https://marketplace.visualstudio.com/items?itemName=GitHubOCTO.flat) which helps you author Flat yml files.

To use it, [install the extension](https://marketplace.visualstudio.com/items?itemName=GitHubOCTO.flat) and then invoke `Flat Editor` from the command palette within VSCode (Mac: ⌘⇧P, Others:ctrl-shift-P).

Expand All @@ -56,13 +56,13 @@ jobs:
- name: Setup deno
uses: denoland/setup-deno@main
with:
deno-version: v1.x
deno-version: v1.10.x
# Check out the repository so it can read the files inside of it and do other operations
- name: Check out repo
uses: actions/checkout@v2
# The Flat Action step. We fetch the data in the http_url and save it as downloaded_filename
- name: Fetch data
uses: githubocto/flat@v2
uses: githubocto/flat@v3
with:
http_url: # THE URL YOU WISH TO FETCH GOES HERE
downloaded_filename: # The http_url gets saved and renamed in our repository. Example: data.json, data.csv, image.png
Expand All @@ -85,20 +85,68 @@ These two modes are exclusive; you cannot mix settings for these two in one Flat

#### `http_url`

A URL from which to fetch data. Specifying this input puts Flat into `http` mode.
A URL from which to fetch data. Specifying this input puts Flat into `http` mode.

This can be any endpoint: a json, csv, png, zip, xlsx, etc.

#### `authorization` (optional)

A string used for authorizing the HTTP request. The value of this field is passed in as a header w/ the `authorization` key.

For example, if this field is set to `Bearer abc123` then the following header is sent with each request:

```json
{
"Authorization": "Bearer abc123"
}
```

#### `axios_config` (optional)

Under the hood, the `http` backend uses [Axios](https://github.com/axios/axios) for data fetching. By default, Flat assumes you're interested in using the `GET` method to fetch data, but if you'd like to `POST` (e.g., sending a GraphQL query), the `axios_config` option allows you to override this behavior.

Specifically, the `axios_config` parameter should reflect a relative path to a `.json` file in your repository. This JSON file should mirror the shape of [Axios' request config parameters](https://github.com/axios/axios#request-config), with a few notable exceptions.

- `url` and `baseURL` will both be ignored, as the `http_url` specified above will take precedence.
- `headers` will be merged in with the authorization header described by the `authorization` parameter above. Please do not put secret keys here, as they will be stored in plain text!
- All `function` parameters will be ignored (e.g., `transformRequest`).
- The response type is always set to `responseType: 'stream'` in the background.

An example `axios_config` might look thusly if you were interested in hitting GitHub's GraphQL API ([here is a demo](https://github.com/githubocto/flat-demo-graphql)) 👇

```json
{
"method": "post",
"data": {
"query": "query { repository(owner:\"octocat\", name:\"Hello-World\") { issues(last:20, states:CLOSED) { edges { node { title url labels(first:5) { edges { node { name } } } } } } } }"
}
}
```

We advise escaping double quotes like `\"` in your JSON file.

#### `downloaded_filename`

The name of the file to store data fetched by Flat.
The name of the file to store data fetched by Flat.

In `http` mode this can be anything. This can be any endpoint: a json, csv, txt, png, zip, xlsx, etc. file

#### `postprocess` (optional)

A path to a local Deno javascript or typescript file for postprocessing the `downloaded_filename` file. Read more in the ["Postprocessing section"](https://github.com/githubocto/flat#postprocessing).

#### `mask` (optional)

If your `http_url` string contains secrets, you can choose to mask it from the commit message. You have two options:

**Option 1**: use a string boolean

`mask: true # removes the source entirely from the commit message, defaults to false`

**Option 2**: use a string array with each secret to mask

`mask: '["${{ secrets.SECRET1 }}", "${{ secrets.SECRET2 }}"]'`

### SQL Mode

#### `sql_connstring`
Expand All @@ -115,19 +163,30 @@ A URI-style database connection string. Flat will use this connection string to
>
> If you're using the [flat-vscode extension](https://github.com/githubocto/flat-vscode), this is handled for you.


#### `sql_queryfile`

The pathname of the file containing the SQL query that will be issued to the database. Defaults to `.github/workflows/query.sql`. This path is relative to the root of your repo.

#### `downloaded_filename`

The name of the file to store data fetched by Flat.
The name of the file to store data fetched by Flat.

In `sql` mode this should be one of `csv` or `json`. SQL query results will be serialized to disk in the specified format.

> ⚠️ While the JSON is not pretty-printed, CSV is often a more efficient serialization for tabular data.

#### `typeorm_config` (optional)

A JSON string representing a configuration passed to [TypeORMs createConnection function](https://orkhan.gitbook.io/typeorm/docs/connection-api#main-api).

A common use case for this value is connecting your [Flat action to a Heroku database](https://github.com/typeorm/typeorm/issues/278).

For instance, you can pass the following configuration string to your Flat action in order to connect to a Heroku Postgres database.

```yaml
typeorm_config: '{"ssl":true,"extra":{"ssl":{"rejectUnauthorized":false}}}'
```

#### `postprocess` (optional)

A path to a local Deno javascript or typescript file for postprocessing the `downloaded_filename` file. Read more in the ["Postprocessing section"](https://github.com/githubocto/flat#postprocessing).
Expand All @@ -142,7 +201,7 @@ A signed number describing the number of bytes that changed in this run. If the

You can add a `postprocess` input in the Action which is path to a [deno](https://deno.land) Javascript or Typescript script that will be invoked to postprocess your data after it is fetched. This path is relative to the root of your repo.

The script can use either `Deno.args[0]` or the name of the `downloaded_filename` to access the file fetched by Flat Data.
The script can use either `Deno.args[0]` or the name of the `downloaded_filename` to access the file fetched by Flat Data.

```ts
import { readJSON, writeJSON } from 'https://deno.land/x/flat/mod.ts'
Expand Down
11 changes: 10 additions & 1 deletion action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@ name: 'Flat Data'
description: 'The GitHub action which powers data fetching for Flat'
author: 'GitHub OCTO'
inputs:
axios_config:
description: 'A path (relative to the root of your repo) of a JSON file containing a subset of Axios configuration'
required: false
downloaded_filename:
description: 'The filename to use for writing data.'
required: false
Expand All @@ -11,21 +14,27 @@ inputs:
authorization:
description: 'A string to send as a header for authorizing API requests.'
required: false
mask:
description: 'A string array of secrets to strip from the http_url or a string boolean'
required: false
sql_connstring:
description: 'A connection string for making a SQL query.'
required: false
sql_queryfile:
description: 'A path (relative to the root of your repo) of a SQL query file to execute for fetching data.'
required: false
default: '.github/workflows/query.sql'
typeorm_config:
description: 'A JSON string representing a configuration passed to TypeORMs createConnection function'
required: false
postprocess:
description: 'A path (relative to the root of your repo) or a URL to a deno postprocessing script.'
required: false
outputs:
delta_bytes: # size of changed data
description: The overall number of bytes changed in the output data this run (current size - previous size)
runs:
using: 'node12'
using: 'node16'
main: 'dist/index.js'
post: 'dist/post/index.js'
branding:
Expand Down
Loading