Can you Ignore duplicate inodes? #35

Activity

1. KSXGitHub commented on Jun 15, 2021

   

   KSXGitHub

   on Jun 15, 2021

   Last edited by KSXGitHub

   Owner

   More actions

   Firstly, thank you for creating dust, for despite its limitations, its way of representing data inspired me to create this tool.

   I am well aware of pdu's limitations.

   There are 2 main reasons why I decided not caring about hardlinks:

   • There is no distinguishing between a hardlink and a main inode. I could make some arbitrary rules (such as the first inode is the main inode) but the parallel nature of pdu makes testing impossible.
   • I haven't found a way to accurately demonstrate the existence of hardlinks other than assume 1 to be real and others to be fake (which is not completely true). Therefore, I decided that it is better to treat all hardlinks as equal.

   I'm curious to see how you would solve the above problem

   I'm in the middle of a dust re-write to see if I can make it faster based on some of your ideas. :-).

   Not caring about hardlinks was a design choice, not a trade-off choice between speed and utility. But if, hypothetically, I want to detect hardlinks, I wouldn't aggregate all inodes into one HashSet because it would require Mutex lock, which would render parallelization utterly pointless. Instead, what I would turn DataTree::data into a Vec of inodes/sizes pair (by using rayon's into_par_iter().map().collect()), then aggregate and filter them later during visualization (which is single-threaded). Of course, this advice would only work with the data model that I chose (DataTree is a hierarchical tree, not a pseudo-iterator).

   When you finish your rewrite, can you notify me? I want to update the benchmarks.
2. KSXGitHub

   added

   questionFurther information is requestedFurther information is requested

   on Jun 15, 2021
3. KSXGitHub

   closed this as completedon Jun 15, 2021
4. Repository owner locked and limited conversation to collaborators on Jun 15, 2021