View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0011233 | MMW v4 | Synchronization | public | 2013-09-04 13:53 | 2013-09-13 02:42 |
| Reporter | Ludek | Assigned To | |||
| Priority | urgent | Severity | minor | Reproducibility | always |
| Status | closed | Resolution | fixed | ||
| Product Version | 4.1 | ||||
| Target Version | 4.1 | Fixed in Version | 4.1 | ||
| Summary | 0011233: WiFi sync: Improve performance of "Analyzing sync list" phase | ||||
| Description | Currently if user performs WiFi from MMA/MM8 and doesn't change anything (no track updated/added/removed) then one would expect that next WiFi sync should take just a second. In fact it can take minutes for large sync-lists, because MMA/MM8 downloads whole sync-list. MMA needs to read whole sync-list when - [x] Delete unsynced tracks is enabled -- so that MMA knows which files should be deleted - [x] Auto-sync to server is enabled --- so that MMA knows what to copy to PC - user deleted some tracks in MMA, such a tracks are expected to be re-downloaded on next sync (supposing they are still on sync-list) But in default config both the checkboxes are disabled and user may not delete any file in MMA. In such a case MMW could serve empty sync-list if nothing has changed from the last sync (no track updated, no track added, no track deleted in MMA). It should work as follows: On the beggining of WiFi sync MMA should tell MMW whether it needs the whole sync-list or whether the shortened sync-list (inclding only updated/added tracks) is enough. MMA can decide it based on these conditions: - no track has been deleted in MMA from the last sync - [ ] Delete unsynced tracks is disabled - [ ] Auto-sync to server is disabled or no new track needs to be uploaded Tech details: - MMW needs to remember the last served sync-list, because sync-list is dynamic (auto-playlists rules etc.) so that MMW could serve only added/updated tracks - MMA needs to remember whether a track has been deleted from the last auto-sync | ||||
| Additional Information | When we were proposing WiFi sync workflow in the past with Jiri and Marek, we also had discussed this approach: - MMA uploads its DB (mmstore.db) to MMW server - MMW processes mmstore.db - MMA downloads processed mmstore.db This approach would have some advantages: - it would be very similar workflow as in case of USB sync (same code could be used) - MMW would know MMA's content But also disadvantages: - MMA would need to upload its DB on each sync, this wouldn't be effective for large DBs with many tracks - WiFi sync would be too tied to MMA.DB structure (making harder to maintain) and making troubles when implementing sync with another general client (like MM8) - wouldn't be much usable for the future cloud sync | ||||
| Tags | No tags attached. | ||||
| Fixed in build | 1657 | ||||
|
|
Note that the workflow suggested above wouldn't be so necessary if MMA would be able to analyze sync-list faster. Currently the sync-list download itself is only 3.3% of the whole processing time, the rest is parsing and DB operations. Marek is currently studing (running profiler) what takes so long when processing sync-list. If MMA won't be able to reduce the parsing time then MMW could serve the whole sync-list in separate SQLite DB file including two tables SyncList and Playlists. This way MMA wouldn't need to parse DIDL-Lite XML, but can read the sync-list directly from DB file. But all the aspects (pros and cons) needs to be revised/evaluated. |
|
|
This is generally true, but thing I thought about is that: - Full DB is Uploaded only on Pairing or When Pairing DB is not newer than one on MMW (Eg. MMW would save last DB from sync per device) especially as MMA would GET Latest DB from MMW at the end of Sync which will than be new DB in MMA and all changes to MMA/Device would be reflected in second DB that would reflect only changes and would be much smaller even in case of large number of files. Example MMA -> MMW = DB Version is 001, MMW -> MMA = Correct Send me latest DB Changes, MMA -> MMW = Changes Sent Waiting for sync, MMW -> MMA = Execute these updates now (Tags, new Files, Delete Files, ....), MMA -> MMW = Done, Send me updated DB Changes, MMW -> MMA = Merge Changes with This DB adn Make it current. Also MMA would announce its version to MMW and MMW would check Compatibility eg. In case of incompatibility Announce user what version needs to be updated and why, which eliminate regressions and bug risks. Like now when users wants to use Newer MMA feature that is implemented in 4.1 but user have 4.0 (Playlist sync for example). - As MMW would have full MMA DB stored Locally MMA could Announce itself that is ready for sync and send Last DB HASH checks MD5/SHA1, MMW would start to prepare Sync-List, which will allow MMA to send Updates after last sync (Tags,Ratings, Play Counts, Playlists, Deletions,...), MMW would merge updates and correct sync list, Create New mmstore.db and mmstore.db.Sync that would tell MMA to fetch tracks for sync instead parsing and DB operations MMA will Execute sync based on mmstore.db.Sync -> after complete it would delete it and load new and updated mmstore.db. This would eliminate parsing and DB operations. eg. like on web when you migrate Databases MMW would get pre-prepared *.SQL call script that would be simply executed instead of parsing and complicated DB operations that would bottleneck device (that is why iTunes Initiate Factory reset not device itself) - Doesn't already MMx client announce itself to MMW so that reads correct DB structure like in USB Sync? - On MMW side we would use same structure but for each MMx Client app (MMA or MM8) we would have own importer/merger/exporter for DB. Which will allow us to for example do something that no other app is capable. eg. Move/Clone Device profiles among devices like Migrate MMA DB/files/Playlist/Sync Settings from HTC one X -> MM8 Nokia Lumia 920. My guess is that approach would considerably increase sync analyze speed. re disadvantages: - In my approach there is no need for complete upload only changed data would be sent and even 3000 tracks changes is not that big. Full syncs would be done only in cases where DB is essentially changed like on Firmware updates where physical path is changed. Even MMW take some time to Update DB to current version. - As Device plugin would handle DB format based on Client that initiated sync. Isn't it easier and faster to change how device plugin works than sync/test changes in both MMA/MM8 with MMW changes/updates. Which would also avoid issues like now where we can't release MMW until all bugs are hammered in MMA too and via verso. - From my point wouldn't cloud sync be a same thing? An stored local DB hash checked with Cloud and sync changes? |
|
|
As discussed with Marek and Jiri today, the most easy, regressionless and problably the best solution is to retain current workflow and enhance it slightly by adding following steps: - MMA sends to MMW comma separated list of tracks IDs that are actually presented on the device (previously synced). MMW will exclude these tracks from sync-list unless they have been modified in MMW (e.g. changed rating, changed auto-convert rule) - if enabled, MMA asks MMW for list of tracks that needs to be deleted from the device (are no longer presented on sync-list) All should be backward compatible, so if MMA finds that it is an older MMW build (that doesn't know the new steps/commands), then it uses the current workflow |
|
|
Implemented for MMW+MM8, to be implemented for MMA by Marek. |
|
|
Implemented for MMA |
|
|
Verified 1657 and 163 |
