[Discussion] Electron app features, data persistence, version control

[subdavis]

It is my intention to have the desktop client be significantly simpler than the web one.

App Routes

The application routes

• Dataset Viewer - same as web
• Settings - not sure what will got here, but it will inevitably have some desktop-specific config
• Recently opened - A flat list of all the datasets that have been opened, plus an Open Folder button.
• Jobs (future) - a similar panel for managing running local jobs.

Data

The desktop app will be conceptually different from the web in terms of what you "Open".

• You can open a directory containing images (Same as web)
• You can open a single video (Same as web)
• You can open a directory containing a single video (different. In web, this would upload a bunch of videos at the same time)
• You cannot open a multi-selected group of images (different. On desktop, it doesn't make sense to specify the specific images you're opening, because then you could open different subsets of images in the same folder, and we have to keep track of that in a way that isn't intuitive and might be bad for users. I think it's a reasonable requirement that, if you want to do this, you should be forced to copy/paste your images into another folder.)

Test Data / Training Data

How do we help users to organize between different groups of annotations belonging to the same dataset. Users may have their ground truth, plus they might want to try 3-4 different pipelines and see how they compare. Web doesn't currently deal with this problem, which I think is a problem. The desktop needs to handle it.

Do we need to support multiple annotation sets for a single dataset, each with its own history log?

This is actually exactly the problem that git solves, where each pipeline run creates a new branch, and each edit creates a commit. We could really spiral out of control by talking about https://dvc.org/ (which is EXACTLY the tool for this problem, but would cost many innovation tokens)

Recent Dataset persistence

What does it mean to keep track of a recent dataset?

• Just persist a directory path (in localStorage, for example), and "discover" the contents every time you open a dataset
  • Allows you to put new images in a directory at the "end"
  • What if the data changes? Now all your annotations are borked?
• Persist a bunch of data about the dataset, such as the images that were loaded.
  • What if you want to change this later?

Track persistence

Where should track data live? In what format?

• Store JSON...
  • Alongside the dataset data on disk.
  • In dedicated application space, like ~/.VIAMEdata/<dataset_identifier>/tracks.json
  • In some embeddable datastore like sqlite or https://github.com/typicode/lowdb ()
  • In some electron-embedded persistence layer, like localstorage (pls not this)
  • Some priority order of multiple of these. i.e. "Try to load from ~/.VIAMEdata first, then try the dataset folder, then open an empty project.
• Store CSV instead of JSON, and do serialization/deserialization every time.
  • As an engineer, I don't want to do this because the JSON format is much more appropriate for us.
  • But for the sake of users, I can understand how having to go through the "Export" process with a desktop app adds burden
  • However, I think this will be familiar for users. Photoshop, for example, has bespoke project file formats, and you have to do an export to get the actual "rendered" version of your data. Same idea.

I'm leaning toward the priority order of JSON from a dedicated folder, then project folder.

Advanced/Future

• Perhaps it'd be good to support some kind of "Project file" that viame recognizes that contains a manifest of the images or video to open, plus metadata like fps and whatever else.
• it'd be good to have some kind of single-file manifest that can tell you where to find data, tracks, and metadata all in one.

[jjnesbitt]

Track persistence

Where should track data live? In what format?

• Store JSON...

  • Alongside the dataset data on disk.
  • In dedicated application space, like ~/.VIAMEdata/<dataset_identifier>/tracks.json
  • In some embeddable datastore like sqlite or typicode/lowdb ()
  • In some electron-embedded persistence layer, like localstorage (pls not this)
  • Some priority order of multiple of these. i.e. "Try to load from ~/.VIAMEdata first, then try the dataset folder, then open an empty project.

• Store CSV instead of JSON, and do serialization/deserialization every time.

  • As an engineer, I don't want to do this because the JSON format is much more appropriate for us.
  • But for the sake of users, I can understand how having to go through the "Export" process with a desktop app adds burden
  • However, I think this will be familiar for users. Photoshop, for example, has bespoke project file formats, and you have to do an export to get the actual "rendered" version of your data. Same idea.

I'm leaning toward the priority order of JSON from a dedicated folder, then project folder.

I think we should definitely be storing it as JSON, and allowing for export as needed. As far as where, I'm actually kind of a fan of your second option (detections in a dedicated space, separate from dataset imagery). In the web version, the detections and data are kind of mixed together, and it's sometimes confusing. Storing these in a database wouldn't be a bad idea either, but I'm not sure if the added complexity and edge cases would be worth it.

This makes me think of another situation, where someone is importing their dataset for the first time, and the detections are stored in a location that we're unaware of. We should still load the dataset, but show a warning that we can't find their detections, and give them an option to specify the location for us, at which point we can serialize it into JSON if necessary, and store it where it needs to go.

Advanced/Future

• Perhaps it'd be good to support some kind of "Project file" that viame recognizes that contains a manifest of the images or video to open, plus metadata like fps and whatever else.
• it'd be good to have some kind of single-file manifest that can tell you where to find data, tracks, and metadata all in one.

Is there a proposed way to store metadata? Would it live in the same directory as the option you mentioned above?

[BryonLewis]

I would say we store the most recent annotations always in JSON.

I think it would nice to keep the detections in folders alongside the data because they are tied together. It's the clearest way for a user to have the most control over their data for manipulating it without having to create interfaces specific to those actions.

I think something missing that may be nice is the ability to just point to a specific CSV or JSON and load it on the current dataset. (We would need some checking to make sure frames match). This would allow the user the most control of how they want to arrange their data and how they want to view things on a dataset.

I have absolutely no clue if the below makes any sense. It's Friday and I might not be thinking clearly.

I think we have 4 types of data associated with a dataset (disregarding training for now):

• Current JSON data used for visulization
• Backup JSON data after editing or modification
• Imported CSVs loaded from another location
• ResultingPipeline CSVs

So 4 folders (folders because they default sorted to top when in a directory)

• currentAnnotations (JSON) - I don't like the JSON file sitting there top level makes it harder to browse for
• backups (JSON) - timestamped json backup files when editing or replacing currentAnnotations
• importedData (CSV) - retains imported name when imported, easy to reference again when needed
• pipelineResults (CSV) - no more temp names, name includes pipeline and timestamp can go back to when needed

To put more simply:

• User loads application
• Imports folder with list of images and a CSV groundTruth
  • -> CSV is converted to JSON, JSON is in currentAnnotations, CSV placed in importedData
• Runs Pipeline
  • -> output creates a JSON file in the 'currentAnnotation' folder within the image folder of the images also creates an immediate backup in 'backup' folder, Pipeline CSV is placed in pipelineResults folder.
• User edits some stuff on the annotations and saves
  • -> takes current JSON moves to backup and replaces with new JSON
• User opens data set and decides to use CSV from another machine or data source for visualization
  • -> CSV is placed in importedCSV and the currentAnnotation is moved to backup and replaced with a JSON one generated from the imported CSV
• User wants to go back to an older version
  • -> Just open the backup folder and import the JSON from backup
• User wants to compare specific Pipelines ran on the same data
  • -> start a new instance and manually import the CSV from the pipeline folders

The above doesn't account for nastiness in conflicts on imported, or data that is imported multiple times with the same filename.

Now this is out there and something I've done for the WAMI viewer (easier, single types), but having the ability to merge two sets of annotations for visualization could be neat. The option to either overwrite or merge data on import so you can visualize the differences between pipelines by types

[mattdawkins]

First pass notes -

Recent Dataset persistence

Maybe on the dataset frontpage instead of a "recent data" section, the datasets just stay in the "Data" viewer until they're not present on disk anymore or modified (greyed out) or removed by the user (X) then there's a similar view to the Web GUI alongside an "open / import data from disk" option? Maybe this the same as what you're thinking.

Where data should be stored (csv vs json vs database), Track persistence

It would be great if this were a setting down the line. Store as .csv, store as .json (your json will be meaningless to all people who either expect a viame .csv [external] or coco .json [jon/other internal]). Also store alongside imagery or store in a local "VIAME-Web" folder. Short term should probably go with the easiest option as we just need something that can iterate from there. For most external people some would prefer a csv or a database depending on the party unless large speed costs, some internal json. If internal VIAME-Web folder as json just need option to export it to csv somewhere, similar to web.

You can open a single video (Same as web)

Ideally we'd want to minimize conversions of video (if possible) e.g. use our python kwiver video reader instead of converting to mp4 to get frames

Settings - not sure what will got here, but it will inevitably have some desktop-specific config

Down the line there will likely be lots of settings besides just attributes which we still want. Ability to export/import different class labels could be here e.g. the full category list. Typically for the desktop it's a single user, with them using it for their particular problem so it's more okay these things being global unlike the web version. Other things maybe - default export formats (VIAME-csv, json...), maximum train time and other train settings, etc... though some of the latter might be good to have in settings in the train panel instead. Also want that in web.

Jobs (future) - a similar panel for managing running local jobs.

Yeah we'd still want to track them and have more things than the web version supports (showing percent status bars? though that would be something good in either)

[subdavis]

This issue has mostly been implemented or tracked in other issues.