Synchronize directory tree with deduplication?
I have 2 Linux web servers that have a huge quantity of information (1TB npls) that require to be integrated over a slow-moving link (100 KB/s).
A great deal of the information overlaps, yet remain in various areas.
I would certainly such as some type of rsync / unison device where I can mirror the web servers.
It would certainly require to be extra smart and also recognize if the documents exists at the location (perhaps in an additional area with the very same checksum). If it does, after that it relocates the documents in your area on the location web server as opposed to replicating the documents from square one from the resource web server.
May not be the solution you are seeking, yet the most effective I can locate from memory.
-y, --fuzzy find similar file for basis if no dest file
(sorry, should have stated, that is an rsync command/flag)
(in addition, I have NO IDEA just how, especially, it functions)