As a (future) Object Storage provider it’s very important for me to be sure that clients don’t hit “unusual limits”. One of the previous OpenIO limitations was the number of files that can be uploaded into a single “bucket”. This was fixed by implementing a sharding feature.
Unfortunately this feature is non-documented and I personally couldn’t find much on this subject (disclosure: I received the necessary information directly from Guillaume).
I think that most users will hit the 1+ million files mark at some point, on which case they will already be in trouble.
So here are some questions that might interest prospect OpenIO users:
- is the sharding feature ready for production?
- can the sharding feature be activated on a production/running cluster, or it needs to be setup at start?
- in theory what is the recommended number of items that can be uploaded into a bucket, for non-sharded environments?