You have file objects in separate forms of storage (internal SSD, SFTP, S3, etc), but want to have a centralized directory to access all those files. You want to be able to shift these files across these storage solutions automatically.
This script creates a folder with a huge list of soft links to every object across your storage mediums. You can also define "lifetimes" for each object in each form of storage such that it is slowly "downgraded" in storage priority as it becomes less and less accessed. This allows you to move those files transparently from faster storage to cheaper archive storage.
The entire system relies on the infamous `atime` parameter for POSIX-compliant filesystems which records the last time a file has been recorded. By calculating `atime` values with `find`, we can quickly filter files to be moved between paths.
1.`atime` is disabled on most Linux systems because of it's performance impact to disk performance. The filesystem attempts a write every time it wants to record a new `atime` value, leading to 1000s of writes and reads that cripple the disk. Want to check if your filesystem supports this? Run `$ mount | grep your_mount_path`. If `noatime` is there, you can't use file-o-bot.
1. This requires a POSIX-compliant storage path for each of the storage mediums, which means you have to configure the mounting separately and maintain them accordingly. There is no fail-safe here.
## Installation
1. Clone this repo and change to the directory.
1. Run the install script. `$ bash ./install.sh`
### Docker Support
Storage aliases are still in progress, but that should enable file-o-bot to accurately link the files.
Soft links work over network file shares like NFS, Samba, etc but hard links do not. Do note that soft links also have their program-breaking limitations on certain filesystems.
