0016901: Sync: Add <Track ID> mask in order to avoid duplicates on sync - MantisBT

ID	Project	Category	View Status	Date Submitted	Last Update

0016901	MMW 5	Sync	public	2020-08-22 21:47	2023-05-18 18:28

Reporter	peke	Assigned To
Priority	high	Severity	major	Reproducibility	N/A
Status	closed	Resolution	reopened
Target Version	5.2

Summary	0016901: Sync: Add <Track ID> mask in order to avoid duplicates on sync
Description	Even initially it was raised due the issues with special characters it is also useful for users that would like to sync playlists that can contain same Tracks but use lowering filename length and have Unique filename, but avoid syncing duplicates (Sony devices are known to make issues) By adding <Track ID> or <Library ID> it will serve several purposes: a) Unique ID for Sync (like iTunes do) b) No special Chars in filenames (due the incompatibility) c) Smaller Temp Storage/Sync DB (For WiFi and USB Sync #14521) Example: \Music\$Left(<Artist>,1>)\<Track ID> would solve many user issues EDIT: As requested This bug is raised in order to address 3 issues that we are solving 1. Long Filenames by taking approach similar like Apple forced strict pat/filename structure on their devices. 2. Make Scans and pairing more easily once related bugs are addressed and unique ID is introduced eg. as proposed in 0016901:0059473 3. By Unique ID and solving related bug of OTG it will prevent wrong pairing and allow moving OTG drives across MMW/MM(MAC) instances without allocating duplicates
Additional Information	https://www.mediamonkey.com/forum/viewtopic.php?f=12&t=97215
Tags	No tags attached.

Fixed in build

Ludek 2020-08-25 07:46 developer ~0059466	I don't like that adding <Track ID> as mask, this is internal ID in MM database and is unique only to a single instance of MM. i.e. when user deletes a track from database and re-scans then it gets different <Track ID> I think that users already can use something like: $Len(<Title>) -- <length> It is mostly unique and if not then MM auto-adds numbered suffix (without creating duplicates like in case of <AutoNumber>) Note that for <AutoNumber> it creates duplicates intentionally, see issue 0006343 for details

peke 2020-08-25 10:43 developer ~0059468	As noted offline: "Adding <Track ID> is intentional behavior for a single MM instance. If user deletes+rescan files on device needs to be paired again or resynced. Currently there is no way to sync with multiple MM on single device (eg. work/home) without many many issues. Your proposal complicates even further as if you lower title to 6-8 chars on 10k tracks you will get several duplicates and only Unique ID is needed. Only issue I see is that on Rebuild DB new ID is assigned and all needs to be paired again, which should already happen anyway." eg. #14521 which DB IDs will be wrong. Using Title or any Metadata field can easily change and make thousand of tracks bad eg. Auto tag, properties change, multiple versions would trigger resync and it is harder to debug in cases of issues where if Track ID is used there is only one case where ID can be changed as you pointed in your reply. Secondly as MM5 have Auto Backup of MM5.DB it is easy for user to move DB to another PC and As user now can easily change Media ID from properties migration to other PC is easier as Track ID in both cases is same.

Ludek 2020-08-25 11:53 developer ~0059469	$Len(<Title>) -- <length> is vague in the same way as <Track ID>, both can be changed anytime -- so I don't see an advantage of using <Track ID> When MMA is syncing with new instance of MM5 than all tracks are re-paired based on metadata (Title+Artist+Album+Length). So Title+Length is more stable across varios MM instances/databases. There does not exist a global unique ID for a track.

peke 2020-08-25 20:24 developer ~0059471 Last edited: 2020-08-25 20:30	As talked offline with Ludek, Current proposal should work in MM4 as workaround but for MM5 it complicates things further, because if user changes title and resync duplicates are created, while in case of <Track ID> duplicates will not be created no matter what user do with track metadata, track ID stays same. This behavior also apply to Cloud sync like DropBox, Google Drive, ... as if you lower title to 3 chars $left(<Title>,3)<Length> to get 6-8 char filename on my library I ended with 4182 files with auto added suffix (0001) to (4182) and when using Auto-Playlist I get synced duplicates each time due the order of tracks in playlists and playlist content change on each sync. Also $len(<title>)--<length> create fairly often same result for one word songs. in all cases using <Track ID> would be different and as we have MM5.DB ID MMA2 and MM5 can easily pair db from #14521 and save much needed time in pairing even SDCard is moved to new device. Simulated tests showed 500+% increase in pairing speed and 80+% less resync cases. I do not see any reason why <Track ID> should not be added as available mask for Sync only as it do not posses any risk of fail sync for same DB even on SDcard Change and MMA reinstall, but benefits are great Especially as If we use formated Text like "MMID_??????" or "MMID??????" while having MM5.DB ID written in separate file and if using solution from #14521 Compatibility and redundancy is much higher (especially on large SDCards 200GB+ that can have 50k+ of tracks and still slow on read/write speed). Finally the most important thing is that it greatly improves compatibility with non MMA capable DAP devices that can read Playlists. Same Approach worked in iTunes for years. Even adding it as hidden feature would allow us to better target specific Device profiles and give us space to support devices that are currently mostly unusable or users are complaining about duplicates.

peke 2020-08-25 20:27 developer ~0059472	bug16901.jpg (431,582 bytes)

Ludek 2020-08-26 09:16 developer ~0059473 Last edited: 2020-08-26 09:23	Note that current MMW song ID is not unique across MM instances. We would rather need to introduce an unique GUID for each song and propagate it over the MM product databases, this is planned and I am already using something like this for cloud sync, but e.g. MMA is not prepared for it etc. Therefore I am not keen of introducing it now. Note that you tested the new "Auto-organize" auto-duplicate prevention. There seems a bug in the numbering for auto-organize, you haven't ended up with 4182 duplicates, but fair lower number. I'll fix this as 0016907 (see details in the bug) Try the mask directly with sync and you will find the numbered suffixed all right (in case of sync). Like "The4-57", "The4-57_1" Note that in case of sync MMW remembers every synced file (and its location on device) for every device profile. So pairing is fast (and based on synced location) until the device profile is deleted or a fresh MMW database is used, then the pairing needs to be based on metadata anyway as the MMW song id no longer matches anyway. Similarly everytime you change MMW sync server instance in MMA then all songs needs to be re-paired.

peke 2020-08-26 20:44 developer ~0059477	As talked offline you are right, MMA is a culprit ATM due the #14521. Also as you said we should find the way to propagate MM5.DB GUID to MMA so that MMA is capable to determine which DB should be used and to use that DB for sync. It is possible that user have multiple sync MM5s (eg. home, work, ...) and when user resync with two repairing is happening each time (like you pointed). Introducing better Bi-Directional sync (I am working on framework ATM to overcome some issues we have ATM) and your solution your will just stabilize and speed sync even further. Leave it on your plate to your discretion <Track ID> was short term quick solution before more advanced solution (as you pointed is found).

Ludek 2023-05-17 18:24 developer ~0071917	Resolving as "no change required"..