Skip to content

Create JSON multi-sheet extractor plugin#690

Closed
amarbakir wants to merge 1 commit intoFlatFilers:mainfrom
amarbakir:feat/json-multisheet-extractor
Closed

Create JSON multi-sheet extractor plugin#690
amarbakir wants to merge 1 commit intoFlatFilers:mainfrom
amarbakir:feat/json-multisheet-extractor

Conversation

@amarbakir
Copy link

Please explain how to summarize this PR for the Changelog:

  • Initialize JSON multi-sheet extractor plugin
  • Refactor out shared code from plugins/json-extractor into utils/extractor

Tell code reviewer how and what to test:

The plugins/json-extractor and plugins/json-multisheet-extractor plugins. Existing behavior has been kept

Requirements:

  • Must allow users to upload a single JSON file which results in data correctly uploading to the corresponding sheets
  • Should be able to determine sheet to data relationships through top level JSON keys. E.g.
{
	"contacts": [ // sheet 1
		{ // record 1
			"firstName": "John", // header 1, value
            		"lastName": "Doe", // header 2, value
            		"email": "john.doe@example.com"
		},
		{ // record 2
			"firstName": "Jane", // header 1, value
        			"lastName": "Smith",
        			"email": "jane.smith@example.com"
		},
	],
	"orders": [ // sheet 2
		{ // record 1
			"id": "ORD123456", // header 1, value
            		"amount": 250 // header 2, value
		},
	]
}

Resulting UI somewhat supports this, but a nice next step would be to allow for a multisheet import in one step through the UI, or at the very least not require one to go back to the files list to reimport the second sheet (load data from sheet contacts -> import data to contacts & load data from sheet order -> import data to order all in one step):
Screenshot 2024-10-30 at 6 32 43 PM

@amarbakir
Copy link
Author

Btw this might be out of place but worth noting here since it's related to tesing/validating this effort

I'm pretty confident JSON extractor release 0.9.0 broke previous behavior. So if you have a working app using 0.8.2 and update to 0.9.0, the 0.9.0 JSON extractor doesn't work for both a valid, populated JSON array as well as an empty array (both cases are handled elegantly in 0.8.2). Both fail with this error

Event Received: file:created
TypeError: Cannot read properties of undefined (reading 'get')
    at Object.callback (/Users/amarbakir/dev/FlatfileApp/.flatfile/develop.js:96647:46)
    at exports.Client.trigger (/Users/amarbakir/dev/FlatfileApp/node_modules/@flatfile/listener/dist/index.cjs:1:6431)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async exports.Client.dispatchEvent (/Users/amarbakir/dev/FlatfileApp/node_modules/@flatfile/listener/dist/index.cjs:1:6214)
    at async Object.message (/Users/amarbakir/dev/FlatfileApp/node_modules/@flatfile/listener-driver-pubsub/dist/index.js:108:17)

Given it was a minor version update, I would expect no breaking changes. Let me know if you'd like any further details on this or help debugging/fixing

@carlbrugger
Copy link
Contributor

@amarbakir thanks for the PR! Will take a look.

On the @flatfile/plugins-json-extractor@0.9.0 issue, it should be resolved with @flatfile/util-extractor@2.2.1. Try deleting your node_modules and reinstalling.

@amarbakir
Copy link
Author

amarbakir commented Nov 1, 2024

@carlbrugger moving this over to #693

We will actually address the internal comments and AI bot comments before opening a PR back into this repo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants