filesystem for numerous tiny files

Which Linux filesystem would certainly you pick for ideal rate in the adhering to circumstance:

  • a hundred million files
  • ~ 2k documents dimension in ordinary
  • 95% read accessibility

  • rather arbitrary accessibility
  • high concurrency (> 100 procedures)

Note: The files are saved in a deep ordered tree to stay clear of huge directory sites. Each fallen leave directory site has concerning one thousand files.

Just how would certainly you benchmark it?

2019-05-18 22:47:12
Source Share
Answers: 3

Here is some results comparing all major linux FSes with bonnie+npls that you can make use of as a beginning factor.

In regards to arbitrary looks for Reiser success, adhered to by EXT4, adhered to by JFS. I'm not exactly sure if this will certainly associate specifically to directory site lookups, yet it feels like it would certainly be an indication. You'll need to do your very own examinations for that especially. EXT2 defeats the trousers off every little thing for documents production times, likely as a result of its absence of a journal, still EXT4 defeats every little thing other than Reiser which you might not intend to make use of as a result of hans reiser is existing standing.

You could intend to check into drives that sustain NCQ, and also see to it your install is arrangement to utilize it. Under hefty seeking it needs to give a rate increase.

Last but not least, see to it your equipment has a lots of ram. Given that the documents aren't usually upgraded, linux will certainly end up caching a lot of them to ram if it is broken out room. If your use patterns are right, this will certainly offer you a substantial rate increase.

2019-05-21 08:57:46

I concur with a lot of what Andrew claimed, other than that I would certainly advise Reiser4 or the older (yet far better sustained) ReiserFS. As those examinations (and also the documents for ReiserFS) show, it is made for precicesly the scenario you are inquiring about (lots of tiny documents or directory sites). I have actually made use of ReiserFS in the past with Gentoo and also Ubuntu with no troubles.

Regarding the standing of Hans Reiser, I do not see it as being a trouble with the code or security of the File System itself. Reiser4 is also funded by both DARPA and also Linspire so while I concur that the more growth of the Reiser File System is obscure, I do not point that needs to be a determining variable regarding whether any person need to utilize it or otherwise.

2019-05-21 08:49:33

I recognize this is not a straight response to your inquiry, yet in these instances I assume a data source could be preferable to host this. Tiny documents can be saved in binary layout in a data source table and also fetched at wil. The software program that is making use of these documents need to have the ability to sustain this though ...

2019-05-21 08:24:32