I have recently encountered a S3 storage simulator. It can turn any old home computer into an AWS S3 storage. The software is called “MinIO” and can be found here: https://min.io/
It is available for Linux, Windows, MacOS and Docker. There is also the source code if anyone wants to compile it from scratch.
The installation is simple: all that is needed is to download the executable “minio” , copy it to /usr/local/bin and make executable:
chmod +x /usr/local/bin/minio
Now, we can start minio by issuing the following command:
minio server /my/dir
MinIO will start a server, which doesn’t go into the background utilizing /my/dir as the S3 storage. MinIO should not be run as “root”, creating a dedicated user is recommended. However, There are still few things left to do:
MinIO needs some additional configuration to support https. In particular, it needs certificates. When minio is first started, it creates the directory .minio in the $HOME directory of the user who started it. I used very creatively selected username “minio”:
[mgogala@medo ~]$ grep minio /etc/passwd
minio:x:1001:1001::/home/minio:/bin/bash
In order to make MinIO support https, we need to create the certificate and the key in $HOME/.minio/certs directory. I did that by using the following commands:
cd $HOME/.minio/certs
openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out public.crt -keyout private.key
MinIO server needs to be restarted after that. Now, we can attempt login. The first thing to do is to try with your favorite browser. I mostly use Chrome:
MinIO runs on the port 9000. The security risk that Chrome warned me about is caused by self-signed certificate generated above, by using “openssl” command. After “accepting the risk” and proceeding to the site, the following screen is shown:
The login key requires two pieces of information: access key and secret key. They can be found in the directory which MinIO presents as S3 storage. We started our MinIO by issuing the following command:
minio server /my/dir
MinIO will automatically create sub-directory /my/dir/.minio.sys. The content of that directory looks like this:
[mgogala@medo s3store]$ cd .minio.sys/
[mgogala@medo .minio.sys]$ ls
config format.json multipart tmp
The sub-directory “config” contains the file named “config.json”. The content of the config.json file looks like this:
{
"version": "33",
"credential": {
"accessKey": "ZW7R5TIAZUKSWLYQ3GEF",
"secretKey": "Hm4AEETYL+wfEDXjhz814t9MZFpV3I4UfTIFd1Om",
"expiration": "1970-01-01T00:00:00Z",
"status": "enabled"
},
"region": "",
"worm": "off",
"storageclass": {
"standard": "",
"rrs": ""
},
The keys needed for login are at the very beginning of the file. Now, we can login and configure AWS CLI. However, with the configuration so far, we need to start MinIO manually every time the system is rebooted. The “minio.service” file, needed to start MinIO as an OS service, using systemd can be found here:
https://github.com/minio/minio-service
The site also contains the instructions how to customize the service. Now, we have S3 storage in a box and can play with AWS CLI:
mgogala@medo config]$ systemctl status minio
● minio.service - MinIO
Loaded: loaded (/etc/systemd/system/minio.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2019-04-29 06:33:11 EDT; 39min ago
Docs: https://docs.min.io
Process: 6058 ExecStartPre=/bin/bash -c if [ -z "${MINIO_VOLUMES}" ]; then echo "Variable MINIO_VOLUMES not set in /etc/default/minio"; exit 1; fi (code=exited, status=0/SUCCESS)
Main PID: 6064 (minio)
Tasks: 19
CGroup: /system.slice/minio.service
└─6064 /usr/local/bin/minio server /data1/s3store
Apr 29 06:43:45 medo.home.com minio[6064]: 2019/04/29 06:43:45 http: TLS han...e
Apr 29 06:43:45 medo.home.com minio[6064]: 2019/04/29 06:43:45 http: TLS han...e
Apr 29 06:44:18 medo.home.com minio[6064]: 2019/04/29 06:44:18 http: TLS han...e
Apr 29 06:44:18 medo.home.com minio[6064]: 2019/04/29 06:44:18 http: TLS han...e
Apr 29 06:45:18 medo.home.com minio[6064]: 2019/04/29 06:45:18 http: TLS han...e
Apr 29 06:45:18 medo.home.com minio[6064]: 2019/04/29 06:45:18 http: TLS han...e
Apr 29 06:50:18 medo.home.com minio[6064]: 2019/04/29 06:50:18 http: TLS han...e
Apr 29 06:50:18 medo.home.com minio[6064]: 2019/04/29 06:50:18 http: TLS han...e
Apr 29 06:53:36 medo.home.com minio[6064]: 2019/04/29 06:53:36 http: TLS han...e
Apr 29 06:53:36 medo.home.com minio[6064]: 2019/04/29 06:53:36 http: TLS han...e
Hint: Some lines were ellipsized, use -l to show in full.
[mgogala@medo config]$
There is also something called s3fs-fuse, which is a user mode file system module that presents S3 bucket as a network disk. Since my whole purpose was to play with AWS CLI, I didn’t install it.
For practicing AWS CLI, one should also download the MinIO client called “mc”. The original AWS CLI cannot be pointed to the local host, the URL “https://s3.amazonaws.com” is hard-coded. The “mc” client uses a very decent subset of the AWS CLI:
NAME:
mc - MinIO Client for cloud storage and filesystems.
USAGE:
mc [FLAGS] COMMAND [COMMAND FLAGS | -h] [ARGUMENTS...]
COMMANDS:
ls list buckets and objects
mb make a bucket
rb remove a bucket
cat display object contents
head display first 'n' lines of an object
pipe stream STDIN to an object
share generate URL for temporary access to an object
cp copy objects
mirror synchronize object(s) to a remote site
find search for objects
sql run sql queries on objects
stat show object metadata
diff list differences in object name, size, and date between two buckets
rm remove objects
event configure object notifications
watch listen for object notification events
policy manage anonymous access to buckets and objects
admin manage MinIO servers
session resume interrupted operations
config configure MinIO client
update update mc to latest release
version show version info
GLOBAL FLAGS:
--config-dir value, -C value path to configuration folder (default: "/home/mgogala/.mc")
--quiet, -q disable progress bar display
--no-color disable color theme
--json enable JSON formatted output
--debug enable debug output
--insecure disable SSL certificate verification
--help, -h show help
--version, -v print the version
VERSION:
RELEASE.2019-04-24T00-09-41Z
mgogala@umajor:~$
However, if you want https to work, you will have to copy the certificate generated for the MinIO server to $HOME/.mc/certs/CAs:mgogala@umajor:~$
ls ~/.mc/certs/CAs/
public.crt
mgogala@umajor:~$
Now, you can add the host running MinIO like this:
mc config host add minio https://medo:9000 'ZW7R5TIAZUKSWLYQ3GEF' 'Hm4AEETYL+wfEDXjhz814t9MZFpV3I4UfTIFd1OmT
The syntax is self evident. The string “minio” is the name for your server, the host names, port and access keys are added. Your AWS CLI is ready, you can plow away. Instead of using “aws s3 ls” you will be using “mc minio ls” and the same for the other commands that are implemented in mc. The “mc” command can also access the real S3 storage. Enjoy.
