libdjinterop Improvements : Dropbox streaming track ID missing

[audministrator]

Hi mr_smidge,

I am still very grateful that you developed the Engine OS Export feature in MIXXX !!

After the release yesterday in MIXXX 2.5.1 I tested it and all is working perfect ... as long as you export to a USB device.
In this case the controller can find the tracks easily.

But if you have the tracks exported to Dropbox (which is supported by Engine OS out of the box)
it does work because it can't find the tracks obviously...

After analysing the database content and comparing it seems that Engine OS needs the dropbox track id's like this

The problem is that this can only be obtained using an the Dropbox API call....
Not so easy to implement I guess ?

Next I can do a database update AFTER the export but I have a problem again because when you export the tracks from MIXXX.
The file names are not the same anymore ... as they are prepended with a strange number sequence ?

So I don't find a link anymore between the original file name and the exported file name :-(

So I am stuck for the moment.

Can you give me some tips on how to proceed with this ?
What was actually the reason for this number sequence ?

[audministrator]

The solution might be simpeler than initially perceived....

Once a file is saved and synced to a Dropbox it writes the Unique ID to the local Dropbox home.db
Which is as well an SQLite db 😃

So what is needed is after the export to a lookup in the Dropbox database in the table sfj_resources and instert in the export-mixxx m.db tracks table

This way there is no need for an API call what so ever...

To be continued ...

[audministrator]

Update ...

I tested getting the tracks from Dropboxx by adding the corresponding missing data in the 2 columns :

streamingSource = Dropbox
uri = streaming://Dropbox/Track/id% + File Unique ID

TEST :
Next after updating the Engine OS cloud database on the USB stick.
"G:\Engine Library\Cloud\Dropbox\Engine Library\Database2\m.db"

I started the controller connected to the Engines OS Dropbox streaming source ...
And I was able to play the tracks that where hosted in the cloud b 😃

The challenge is to get the online File Unique ID's of the Tracks in Dropbox.

There are 2 options :

1. Using the web services API of dropbox
2. Or get it from the local dropbox database on your system that contrain all data from the synced files to that cloud

Bottom line is that everything works as expected and tested by just added 2 simple missing data elements in 2 colums.

[mr-smidge]

Hi @audministrator , thank you for the kind words!

To answer your first question, the number sequence prefix on filenames is simply Mixxx's internal id for that track. It means that a track in dir1/mysong.mp3 and dir2/mysong.mp3 can be exported to the same mixxx-export dir without a filename clash.

I have not tested any Engine devices involving DropBox before... so this is new territory for me. But I'm pleased to hear you got things working by modifying the streamingSource and uri columns.

I think that if we were to add Dropbox support (or more general streaming track support) officially. the changes would all be in Mixxx rather than libdjinterop... so if you're asking for a feature request it should probably go in that project 😄.

[audministrator]

Hi Adam,

Thanks for the feedback.

I understand that you don't want to add this functionality.
The functionality is maybe not for all engine os users.
I can live with this :slight_smile:

Are you still planning to release the export playlists some day ?

[mr-smidge]

I understand that you don't want to add this functionality.

Don't get me wrong - I'd be happy to have this functionality in mixxx... But I think the changes would be in mixxx rather than libdjinterop. I might be mistaken though!

Are you still planning to release the export playlists some day ?

Working on it now, in fact!

[audministrator]

Great news looking forward to this feature 😄

[audministrator]

I doubt that mixxx will implement the streaming services...
Because it is has limited use if you are not on a Denon / Numark standalone controller that runs on Engine OS.

The beauty of your solution is that we do t have to use the Engine OS Desktop as a software.

This part of software is really crappy Built.
And it avoids double work in preparation of a set.
Because it was already prepared in MIXXX.

[audministrator]

Hi Adam

Some update on further testing I did ...

All seem to work well if I added the dropbox Track ID (getting from the dropbox DB)
BUT when I choose Dropbox as a source in Engine OS the tracks are there, but all the performance meta data have gone 😦

Like hot-cues, loops etc.

This is not the case when I export to USB all remains in tact ...

That is a bummer ! Because this means all the prep work of the set is gone 😕

I noticed that Engine OS uses a different database if downloaded from cloud services like dropbox.
On the USB drive when the track is downloaded it stored the info in cloud.db instead of m.db

Needs forther investigation to see if I find why this is happening and how to fix it.
It all looked very promissing ... but not completely

[mr-smidge]

Hi @audministrator , sorry for the slow reply to your last post.

I have not yet experimented with cloud services on Engine devices, but if you already have a database with tracks that are played from Dropbox, would you be willing to zip up your Engine Library directory and perhaps post it here? Just the DB, not any actual tracks, for copyright reasons of course!

I could use it to help identify the work that might need to be done.

(Thoughts: there are a number of database files in a full-fat Engine Library directory... m.db, hm.db, itm.db, rbm.db, sm.db, stm.db, trm.db... full support for streaming files might involve adding a mechanism to libdjinterop to fully support these databases too.)

[audministrator]

Hi Adam, no worries...

I will check in the weekend to see that I can bring in some relevant information while testing.
Appreciate a lot the work done already !

I am also willing to test the export playlist if usefull (I mentioned that yesterday in the PR) but I am not sure whcih version of the Mixxx 2.6 already has this beta feature ?

Rgds
Patrick

[audministrator]

Hi Adam,

I will try to explain the dropbox official Engine OS approach and next a way to do the Mixxx approach...

Denon desktop to Dropbox

On Dropbox after exporting it goes to the standard folder structure Database2 and stores the information in the m.db.

In the track table you can see 2 fields filled streamingsources and uri

The tracks are is a different folder (which I do not like and also is not needed !!)

As you can it publishes all trackes in it's own folder ??

USB Drive
After you connect your Engine OS controller to access the sreaming serivice on Dropbox, and download tracks...
You can see the following folder structure and databases on your USB Drive.

There is the official Engine Library the where to export goes to.
(The Mixxx-export is yours export main folder from Mixxx)

In the Engine Libr there is a Cloud folder
Full Path is D:\Engine Library\Cloud\Dropbox\Engine Library\Database2

In the Database2 these are the databases, that contain the tracks from dropbox

The m.db contains the track information.
It fills up the URI field content as below.

It should start with streaming:// and next the track locations on dropbox

Next I will create a separate post for the MIXXX approack.

[audministrator]

Mixxx Export to Dropbox

I tried the latest beta v17 that included also the export of playlists 😄
But I ran into a few issues doing the export ?

I could not use dropbox main folder because it clashes with the existing Engine OS export main folder Engine Library

So this needs to be taken care if in the logic that if the filder exists don't create one.

So I chose to export to mixxx-export

The tracks ended up in mixxx-export\mixxx-export for the time being

In your m.db the track paths are stored as relative paths

In the track table the fields streamingSource and uri are empty are empty now.

Should have :
streamingSource = "Dropbox"
uri = track path URL encoded format

I see that Engine OS syncs the cloud database in dropbox to the local USB drive into the CLOUD folder when you donwload a track form the cloud services.

If you need more information please let me know.

PS :
I had some issues with the new playlist exports but I will create a PR in MIXXX for this.

[audministrator]

Conclusion of the test

If we look back at the USB data after I connected to dropbox using Engine OS

You can see that Engine OS downloads for each track in the deck the MP3 file and assignes a unique file name to it.
These 2 tracks where in a random folder in Dropbox.
It doesn't seem to matter where the tracks are we can access all, but of couse we miss the performance data (hotcues / etc)
So far so good.

It creates again a Cloud folder ones connected to Dropbox (which I had deleted from the USB before the test)
So far so good.

It has synced the database form MIXXX export to the local Cloud database on the USB drive
So far so good.

But for the MIXXX export it does not recognize the file location as well as the playlist content.
Because of what I mentioned in the previous post missing streaming data

I guess that I the 2 fields are filled up properly,and the relative paths are full paths, all will be solved !

streamingSource = "Dropbox"
uri = track path URL encoded format

Engine OS will take care of the local syncing in the performance database of the controller.

I will add a few pictures in a seperate post to show you how it looks on the controller.