Task file storage

[shfattig]

Enabling direct manipulation of taskwarrior .data files and external file R/W permissions to other apps so that third-party file sync services (i.e. Syncthing) can provide two-way synchronization of tasks, in lieu of a taskserver

[shfattig]

Feature requested in #145

[shfattig]

@bgregos continuing from issue...

I'm glad to hear you're going to be doing this!

1. I'm fine with ignoring `completed.data` and `deleted.data` as long as it doesn't cause an issue with Taskwarrior. I think you may need to implement them so the other Taskwarrior client knows that those tasks have been completed. I'm not sure what to do with `undo.data` since Foreground doesn't support undo - we'd need some more investigation to see if we need to handle this.

2. The app basically implements a full taskwarrior sync client without using any existing Taskwarrior binaries or code. Changing the underlying storage mechanism shouldn't affect sync, since Forground loads your task data into memory, works on it, and handles syncing from the copy of the data in memory.
   
   1. About integrating the Taskwarrior binary - this would be a (very) big lift since Foreground itself implements its own Taskwarrior client and we'd be effectively rewriting Foreground's entire data layer. I'm not opposed to this in the long run but it's probably outside the scope of this issue. Feel free to file a new issue if you'd like to explore this deeper.

3. The problem here is that Taskwarrior's internal data format isn't JSON, so you'll need to convert everything into Taskwarrior's data format, including all fields. Taking a shortcut and only supporting fields is likely to cause issues in my opinion.

4. Yes, my issue with defaulting to external/shared storage is that user data would be visible to all apps by default, which is a major security issue in my eyes. We would want to keep it private by default, using either the existing SharedPreferences implementation or using internal app-specific storage. I'd be fine keeping the SharedPreferences as the default storage mechanism and using external/shared storage if the user requests it. Ditching SharedPreferences entirely might be a bit easier, since you wouldn't need to juggle the different formats of SharedPreferences (json) and external/shared (Taskwarrior format) and you could use Taskwarrior format for both internal (private) and external (shared) storage.

Opening a WIP PR sounds good to me!

Yeah the simple answer to handling completed and deleted is just to remove the pending task and naively append to the relevant .data file (if appending is what taskwarrior does, I'll have to check that. Also may not matter). We also just may not be able to perform undos. Some test cases would help to determine whether that breeds undefined behavior.

Regarding taskwarrior's file format, it may not be valid JSON but it's definitely a close cousin! I'm also noticing that completed.data and pending.data follow the same format, but backlog.data and undo.data or something different.

I hear you on reasons to ditch SharedPreferences entirely. Judging by your comments in the code and README it sounds like you intended to do this at some point anyway. It also looks like you intended to move to a Room database though. You're ok not going that route?

[bgregos]

Serializing to JSON and sticking it in SharedPreferences is a shortcut I took earlier in development and I've always wanted to change it. Going with a file-based implementation instead of Room is fine, no sense in having both a file based implementation and a Room one since our datasets are small.

[bgregos]

You can remove the comments about the Room implementation so we don't confuse future contributors 🙂

[shfattig]

Alright, I've been able to do a little digging on this one.

So the taskwarrior backend files are json-like text files, but are not json. However, there IS a defined json standard for importing and exporting tasks - https://github.com/GothenburgBitFactory/taskwarrior/blob/develop/docs/rfcs/task.md

There is also some pretty great 3rd party app documentation, which actually states that the standard for integrating 3rd party apps should be to task export json data from taskwarrior. It also explicitly forbids trying to read the .data files - Don't attempt to parse the pending.data file. - and gives good reasons for doing so.

SO, this is helpful direction. My proposed path forward is to align the json format that Foreground uses to match Taskwarrior's json format. This should allow us to retain current functionality with actually minimal modifications!

Then, for those who want to to connect Foreground with their desktop TW through SyncThing, we just have to write two taskwarrior hook scripts, on-launch.sh & on-exit.sh, which automatically import and export, respectively, the json file which is tracked by SyncThing. I've done a little testing of this interaction locally and I think it will work well. Requiring these extra scripts is less clean than ideal, but I think it's reasonable since anyone using SyncThing is likely comfortable enough with doing something like that.

This would still necessitate the changes to the file storage mechanism.

[martin-braun]

Hi @shfattig,

thanks for working on this issue, it means a lot to me.

Alright, I've been able to do a little digging on this one.

So the taskwarrior backend files are json-like text files, but are not json. However, there IS a defined json standard for importing and exporting tasks - GothenburgBitFactory/taskwarrior@develop/docs/rfcs/task.md

Thanks for sharing this, https://github.com/GothenburgBitFactory/taskwarrior/blob/develop/doc/devel/rfcs/task.md is the new link for record.

There is also some pretty great 3rd party app documentation, which actually states that the standard for integrating 3rd party apps should be to task export json data from taskwarrior. It also explicitly forbids trying to read the .data files - Don't attempt to parse the pending.data file. - and gives good reasons for doing so.

Great findings, so we should comply with these guidelines indeed and just ensure interoperability between Foreground and Taskwarrior's exported format instead.

Then, for those who want to to connect Foreground with their desktop TW through SyncThing, we just have to write two taskwarrior hook scripts, on-launch.sh & on-exit.sh, which automatically import and export, respectively, the json file which is tracked by SyncThing.

Well ...

My hook attempt

I've done a little testing of this interaction locally and I think it will work well.

So did I, but I ended up with an endless loop. on-launch cannot contain task export, because it will also trigger on-launch, same story with on-exit. Using on-add on on-modify hooks won't work either, because any task export will not contain the recent change, as it will be written to the database after the hook executed.

I guess you could build a complex on-add and on-modify to store the new task in an own format and redirect (portions) of the stdin to the stdout so taskwarrior keeps working with those hooks. Then you also have to alias task to parse all those clear text files back into the system before doing any operation.

This is terrible and slows Taskwarrior down, imho. Also it is only half of the story, because now Foreground need to two way sync our own format.

I'm against hooks

Instead of listening to on-launch and on-exit, which obviously can't contain any task import / task export, I've written a function to wrap Taskwarrior to do the import / export accordingly:

export $TASK_DATA_HOME="$HOME/sync/taskwarrior" # can also be referenced in .taskrc on data.location
task() { command task import >/dev/null 2>&1 < "$TASK_DATA_HOME/tasks.json" && command task "$@" && command task export > "$TASK_DATA_HOME/tasks.json"; }

This seems to work, I now have a consistent file at ~/sync/taskwarrior/tasks.json which Foreground could manage. Unfortunately, it's still slowing down Taskwarrior a little, but I can live with that for the benefit of synchronization over Syncthing. We should document this function (maybe with more improvements to it) once the feature lands in Foreground.

This is just part of the story

Foreground's synchronization with such a file needs to be deeply integrated. On any active or passive change, the file needs to be taken into consideration:

• File was changed from Syncthing: Refresh the view, so the app needs to have a FileWatcher if Android offers something like that, or poll for changes
• Any action taken from within the app: The file needs to be written out again

Conflict resolution

Is not required as conflict resolution will be handled by Syncthing. It's not a smart one, though, the user will end up with two files. It's not worth to invest into this section further, as Syncthing is just one of many synchronization solutions. If the user decides to sync with the local Android file system, he/she needs to be aware that Foreground cannot handle conflicts, this work is shifted to the synchronization software.

Personal note

I'd love to hear from you. Did you happen to have some time to work on the app to synchronize using Taskwarriors JSON export format?

Thanks for working on this.

[shfattig]

Hey @martin-braun ! Thanks for your message! I'm glad you've started some work on this! I haven't made much more progress since my previous research, but I might be getting some time to invest back into it!

I ran into the same loop issue you have with the hooks. A fix would be to set a flag that checks whether you're already running the hook, something like this:

#!/bin/bash

json_file=/SyncThing/path/to/json/json_data.json 
if [[ -f $json_file ]]; then
  if [[ -z $_reading_json ]]; then
    export _reading_json=true
    task import $json_file
  fi
  unset $_reading_json
fi

I can put that up in this pull request

With regard to SyncThing, I think we will have to assume that changes are made on each node w/ significant time separation. SyncThing isn't really built to handle concurrent changes and conflicts. But if there are better realtime synchronization mechanisms I'd be glad to hear!

[martin-braun]

@shfattig Thanks for sharing the hook solution. After all though, it made sense for me to stick to my method, because I have to use this method for taskwarrior-tui with an introduced lock file, but this is an unrelated problem.

I agree on your stance about Syncthing, that's exactly my conclusion as well. Still I want to emphasize that pulling of file changes on Android's size should happen when the app comes into foreground and not just when it's started, in case it wasn't clear.

I haven't made much more progress since my previous research, but I might be getting some time to invest back into it!

No pressure my friend, just to mind, there is nothing similar available as of now. I guess I can edit my tasks.json with DroidVim, but it's very painful to do so.

I also looked around maybe finding a JSON GUI editor for Android to make it more pleasing to work with that file, but there is nothing on the FOSS side and on top of that, Taskwarrior's JSON format has some additional rules that might not be followed by other apps.

[shfattig]

I agree on your stance about Syncthing, that's exactly my conclusion as well. Still I want to emphasize that pulling of file changes on Android's size should happen when the app comes into foreground and not just when it's started, in case it wasn't clear.

I do agree, the app should refresh on opening, coming into foreground, and likely every few minutes while it's open. I don't know the API for that as I'm very new to mobile development, but I'm certain it exists!

there is nothing similar available as of now

Indeed. Foreground is one of a kind!
Do you have any expertise that you'd like to share for this task? We can break it into subtasks if you're willing to devote some time to it.

[martin-braun]

@shfattig As of now, the need for it is not strong enough to devot any time for it on my end (yet), since I have to focus on things that pay the bills with dead lines keeping the pressure up. Over time the priority shall rise though. I have experience in Android development to a point that I would feel confident to add a feature to an existing app, however I don't think I can deliver a state-of-the-art implementation, especially since I have no experience with Kotlin so far (which isn't a dealbreaker tbh).

[bgregos]

If anyone working on this has questions, feel free to give me a ping. I'm happy to walk you through the app structure, Kotlin, or anything else.

…

On Sat, Aug 26, 2023, 10:11 AM Martin Braun ***@***.***> wrote: @shfattig <https://github.com/shfattig> As of now, the need for it is not strong enough to devot any time for it on my end (yet), since I have to focus on things that pay the bills with dead lines keeping the pressure up. Over time the priority shall rise though. I have experience in Android development to a point that I would feel confident to add a feature to an existing app, however I don't think I can deliver a state-of-the-art implementation, especially since I have not experience with Kotlin so far (which isn't a dealbreaker tbh). — Reply to this email directly, view it on GitHub <#157 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABM2MFIY7BMYNUPC6CQN32LXXH7XVANCNFSM6AAAAAAWHWEYF4> . You are receiving this because you were mentioned.Message ID: ***@***.***>