View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0019087 | MMW 5 | Tagging / organizing (properties / auto-tools) | public | 2022-05-19 21:42 | 2026-09-03 17:59 |
| Reporter | rusty | Assigned To | |||
| Priority | urgent | Severity | feature | Reproducibility | always |
| Status | feedback | Resolution | reopened | ||
| Product Version | 5.0.4 | ||||
| Target Version | 2027 | Fixed in Version | 2027 | ||
| Summary | 0019087: Duplicate Manager | ||||
| Description | Placeholder--This functionality needs to be specced. Currently duplicate management is a very complex/manual process. MM should have a single tool that allows the user to: - Easily find all duplicates via a combination of existing hash (exact duplicates) and artist/title matching + fingerprinting to help automate the process for finding duplicates that are either similar sounding (different versions) or are missing track metadata. - Visually display possible duplicates in a manner that makes it obvious which items are possible duplicates (e.g. grouping them) - Facilitate selection of which items to retain and which to reject - Automatically recommend which files to retain and which to reject (e.g. via rules based on bitrate, volume leveling, etc.) - Automatically recommend which metadata to retain (e.g. most recently updated track OR merge of metadata OR selection of metadata, TBD) - Allow play history of deleted entries to be merged with the retained entries (e.g. last played should be the most recent date, played # should be the sum of all played # values). - Allow playlists to be automatically updated with the retained entries Additionally, it should be able to handle cases in which the duplicates actually exist AND/OR cases in which there are duplicate database entries (even if the tracks do not exist) Note: usecases could involve: - finding duplicates within a set of selected tracks - finding whether a set of tracks already contains duplicates elsewhere in the library | ||||
| Tags | No tags attached. | ||||
| Fixed in build | 3520 | ||||
| related to | 0019083 | feedback | rusty | MMW 5 | Media Properties: Changing Media Drive letter to existing Drive letter create duplicate |
| related to | 0016154 | new | petr | MMW 5 | Use audio fingerprints to detect duplicates |
| related to | 0005800 | new | MMW v4 | Duplicate Content Node Issues | |
| related to | 0005498 | new | MMW v4 | Duplate analysis should occur as a background process & be enabled by default | |
| related to | 0000839 | new | MMW v4 | Files to Edit: Duplicates -- add intelligence to exclude similar but different files | |
| related to | 0000231 | new | MMW v4 | Simplify Duplicate Tracks functionality | |
| related to | 0022117 | resolved | Ludek | MMW 5 | CUE files are shown under 'Duplicate content' node |
| related to | 0021601 | closed | Ludek | MMW 5 | Ability to easily copy file tags and artwork between files |
|
|
What about to add 'Remove duplicates...' as the first item to the context menu for these nodes: Files to Edit -> Duplicate Titles/Artists Files to Edit -> Duplicate Titles Files to Edit -> Duplicate Content Selecting a node (or selection of files) -> right-click -> Remove duplicates... would show dialog like this: -------------------------------- Do you wish to remove 115 duplicates from your library? (o) Delete from the database only ( ) Delete from the database and drive [x] Create database backup [x] Retain files with higher bitrate / quality [x] Join metadata (in case of conflict prefer those with newer timestamp) -------------------------------- Left assigned to me for implementation, then wording can be adjusted by Rusty. |
|
|
Mostly implemented in 3514, just 'Import missing metadata' feature is a to-do.. |
|
|
Implemented in 3515 Added 'Manage Duplicates' to both context-menu and tracklist selection context-menu for these nodes: Files to Edit -> Duplicate Titles/Artists Files to Edit -> Duplicate Titles Files to Edit -> Duplicate Content Resolved for testing. |
|
|
Tested 3520 and this is a really great start! Here's some initial feedback: 1) It's not obvious how to access the functionality a) Right-clicking on any of the duplicates node doesn't display 'Manage duplicates' unless the user has focused on the node beforehand (this is a regression that applies generally to all nodes--right-clicking on them no longer works). b) If the user visits a duplicates nodes, they have to 'Select All', and then use the context menu to access a function that they may not realize exists. It would be preferable if the toolbar displayed a 'Manage duplicates' icon (by the same logic that the playlists node contains playlist management icons) that would apply to all tracks in the view unless more than 1 track is selected. c) We may even want to consider adding this to the Tools menu (following Auto-tag). Thoughts? 2) Text tweaks: a) This action selects duplicates to delete from your library base on the following criteria: --> Choose which of the duplicates to delete based on the following criteria: b) Change the 'OK' button to 'Delete...' 3) Similarity: can you clarify how this is implemented? Is it based on audio fingerprinting for the duplicate content node and on metadata for the other two nodes? If so: - How/when are fingerprints calculated (i.e. I know that hashes can be calculated on scan, but afaik fingerprints were never calculated for my collection previously)? - How is similarity calculated based on metadata? Note: One of the goals of the feature was to eliminate the need/complexity for multiple duplicates nodes. If my understanding of the current design is correct, it continues to reinforces the need for multiple nodes, and we should discuss whether it would make sense to change this. 4) Quality & Format preference order: can you roughly describe the implementation? e.g. is 'format preference' subordinate to 'quality'? 5) Often, there are identical tracks in different locations. Currently, MM will recommend deletion of tracks from one location for some duplicates and from another location for other duplicates (making the library a mess). The spec had indicated using 'Location is in', but a better approach would be to apply the approach you used for format preferences. e.g. [ ] Location preference order _D:/Music,D:/Temp/Download,//NAS/Music,___________________________________________ [Configure...] Note: this preference should follow and be subordinate to Quality&Format preference order 6) 'Import missing metadata' is unclear since it isn't a criterion, and doesn't obviously do anything. Suggested changes: a) Change to 'Copy missing metadata from deleted tracks' b) Move it to the bottom-left (same level as OK/Cancel) c) Highlight copied metadata so that it's visible (same as how this is done in the Tag from filename dialog) 7) Do you think it would be useful to be able to use headers for sorting? e.g. I wanted to see all 'Beatles' duplicates together. The issue is the all dups in a group may not have the same value, so which track would be used for sorting purposes? |
|
|
1a, 1b, 1c, 2, 4, 5, 6a, 6b , 6c => fixed 7) => having all sort columns available for sorting is problematic, because the groups of duplicates mostly consists from groups like Track1 - Album1 - Artist1 Track1 - Album1 - Artist1 --------- Track2 - Album1 - Artist1 Track2 - Album1 - Artist1 Nevertheless I can adjust the pre-defined sort so that groups are sorted by Artist and then by Album to have all artist albums together.. 3) Re similarity: Whenever the signature hash or fingerprint is presented then it is used otherwise the similarity is calculated based on metadata => combination of [title, album, artist, length] attributes with various accurances, examples: -> above 90% all [title, album, artist, length] needs to be accurate -> below 80% the length accurance is +1s / -1s and [title, album, artist] attributes can have accents and orphans -> below 60% the length accurance is +3s / -3s and 'album' is no longer between compared attributes etc. Re: elimination of the need/complexity for multiple duplicates nodes, I guess that the multiple duplicate nodes are still useful, but I added 'Manage Duplicates' action to the tools menu so that it can be used on any tracklist or auto-playlist |
|
|
And I also added 'Quality preference order' as subordinate to 'Format preference order' |
|
|
Set locations presents a non functional dialog for me. 1 Right click on Duplicate Content node (ie. node itself is not loaded) 2 Manage duplicates 3 Set location |
|
|
Fixed in 3520 |
|
|
Tested 3520 and it's looking much better! Here are the test results/remaining issues: 1) Access to functionality a) verified b)i) The 'Manage duplicates' icon currently looks like a 'Copy' icon. To differentiate, it should have a gear overlaid on the bottom right of the Copy icon. b)ii) If the user has a single track selected, the function isn't applied to all tracks in the view. This is minor, but could be confusing if the user enters the view, plays a track, and then 'Manage duplicates' and no dups are found. On the other hand, other functions don't work like this, so perhaps we should leave as is. Please triage. c) Navigating to a node (e.g. Music > Albums) and then clicking Tools > Manage Duplicates --> Nothing happens! 2) Tweaks a/b) Verified c) I would suggest another tweak to indicate the number of duplicates to be deleted (currently there's no indication of this). Suggested change: Selected duplicates to be deleted (x/y): where x indicates the number to be deleted, and y is the total number of dups. This is also useful since changing 'Similarity' will give users an indication of the effect on the number of tracks to be deleted. 3) Similarity We can explain how this works in the docs, but in regards to 'Duplicate content', there's no way to know whether hashes/fingerprints have been calculated for the tracks since: a) MM prompts to calculate hashes for tracks that are missing hashes, but doesn't do this for fingerprints (my suspicion is that they've never been calculated and aren't being used for the similarity calculation). b) MM doesn't display hash/fingerprint data anywhere in the UI (e.g. in Properties / Details) 4) Quality/Format preferences: a) There's an issue with the current design, that I'm not sure how to best resolve. The current Format preference order followed by the Quality preference order will result in e.g. MP3 of 128kbps being preferred over an OGG of 256kbps. One suboptimal solution (but the best I can think of) is to change the order (and make Quality above Format). It's suboptimal because it would result in wav being preferred above flac, regardless of the user's codec preference. A better solution is for MMW to internally store some sort of quality curve for each format so that it can determine what combination of Codec/Bitrate is superior, but I'm not sure if this is feasible. Do you have a better idea? b) Format preference order, Quality preference order, and Location preference order should all be optional (they should have checkboxes) 5) Location preference order: a) The first time it's run, it's pre-populated by some incorrect/strange paths: C:\\\NAS2\,\Users\russe\Music\Downloads\Nirvana\Nevermind\03 Nirvana - Come As You Are.mp3.mp3,C:\,D:\ . i.e. C:\\\NAS2\ doesn't make sense, nor does the Nirvana path. I would have expected MM to prepopulate this list the first time by examining the duplicates and listing the 'lowest common denominator paths'. e.g. if the duplicates in the list are: D:\Temp\test... D:\Temp\dl... D:\Temp... D:\Media... \\NAS2\ D:\Media etc... Then all 5 of the paths above should be included in the list of suggested path orders b) When clicking the 'Set Locations...' button which should presumably give the user full control over editing/re-ordering locations: i) there's no means of adding new locations! MediaMonkey just shows a list that can be re-ordered, but there's no means of adding new locations. There should be an 'Add Location' button. ii) It should be possible to delete a location using the <Delete> key. iii) there's no means of editing individual pre-existing paths c) We may want to add a button to 'Suggest paths' to add paths to the list based on the logic in a) 6) b) '[ ] Copy missing metadata from deleted tracks' should ideally be lower (vertically aligned with Delete/Cancel). c) Highlight copied metadata so that it's visible (same as how this is done in the Tag from filename dialog) 7) Sorting: still open. Maybe the best approach would be to sort by the first instance of any group of dups, whether the first instance is based on the user's preferences of which track to keep? |