Elixir vs Object storage in AWS S3 style

We had couple of problems with file storage for our Phoenix app. So I decided to use AWS S3. But!!! We have couple of issues with AWS S3 legal stuff so we can not use 3rd party service like this, it’s our fault. So I opened magic page, I’m pretty sure you know that page but I give you clues: first letter is G and the last one is E and a lot of people on the planet use it as home page.

I found Minio, some information from their site:

Minio is a high performance distributed object storage server, designed for
large-scale private cloud infrastructure. Minio is widely deployed across the
world with over 107.5M+ docker pulls.

How you can see on their homepage they give you also installation instructions for a lot of platforms (Docker, Kubernetes,…). We use Kubernetes/Tectonic so I checked their installations script. It’s very good, you can choose using simple Deployment (Standalone) or StatefulSet(Distributed). Let’s start.

Minio

When you develop something you need to run it on your localhost, no doubt about it. When you open Minio’s homepage you can see Docker installation. Problem is when you run simple docker run with their code it generates new credentials on every run. And you have to change your local Phoenix config again, again, again. It’s easy to fix it, you have to use their ENVs. See my example:

Let’s say something about it:

• -p 9000:9000 This will open port 9000 from container to your computer. You can access it with browser http://127.0.0.1:9000 to see UI or use it with CLI or some client
• MINIO_ACCESS_KEY and MINIO_SECRET_KEYThese environments are for setting your credentials and it will use these keys every time when you start the container, even fresh pulled or on other machine
• /Users/medium_user/Projects/working_dir/project/data:/data Long line but good line. This create persistent storage, so you will not lost all uploaded data after you kill container
• minio/minio is standard Docker image name without tag (default is latest)
• server /data Docker execute command server with param /data, first param set which folder contain data inside container.

So done. Check your browser http://127.0.0.1:9000 and you will see working Minio.

Don’t forget to create some bucket via UI with plus button (right bottom)!!!

Elixir Configuration

Minio doesn’t have own client for Elixir but their API is same like AWS S3 so it’s possible to use clients for AWS.

As you can see there is ex_aws and ex_aws_s3. There are main libs, they use xml so that is reason why sweet_xml. Also I had to show you two another libs. xml_builder is there because we use it in our code and they use it (some how) but they use old version, so I had to override: true to use our new version which I need. storage is nice lib for uploading files from forms. I will show you some code later in this article.

Let see some code. There is static config for Phoenix app:

The most important part of this settings is region. It has to be local so it use your URL (host)instead of AWS.

There is static config for development dev.exs :

• host is IP address of your Minio server (localhost)
• bucket is the name of bucket which we created in UI. It creates error if this bucket doesn’t exist in your Minio server. Maybe there is a way but I don’t know now about it.

Important part is don’t forget add your credentials in dev.secret.exs

We use system ENV for config in production.

Elixir Integration

Integration is very easy after hard setup. I will show you easy integration. I think there is also another way like upload from memory not from disk, etc. but it’s for another article.

Upload

• bucket() is function which return name of bucket from config
• path_to_file has to be string with path to file which we want to upload
• path_in_bucket is also string with path to file in bucket

I use same paths for both. First path (which is path to file which I want to upload) use relative address. For example data/accounts/1/logo.jpg. So it’s folder inside app root directory and when you use this path like path in bucket it will create these folders also in bucket without ani problems. So you don’t have to care about two strategies for uploading files/folders. Is it possible to upload multiple file with Stream with good performance but I think you can read&copy&paste it from their docs.

Download

It’s same like above and both path can be same. Nothing special here.

It’s very important all folders and subfolder of your local path have to exist when you download some file. So there is simple script to create it:

That’s it! You can use it in your code and it has to work (because it works for me ;-) ).

Storage Adapter for Minio

We use also this nice lib Storage for file management with our Ecto and so on. There is one default adapter for local storage. So I create this small and very easy adapter which store your file on your local server but also upload it to Minio. It’s first version and there is a lot of space to improve it but for now it’s enough.

You have to set default adapter in config file:

But it didn’t help me and I had to set also adapter for Storage Object implementation:

Now it works and also it uploads/downloads all files without any problem.

Just don’t forget create bucket also on your production Minio server!! ( I did)

I use Minio just a few days and it looks very well. I see some space to improve but I think it can be huge tool right now. My code example is very easy and I created it for couple of hours right after I found Minio. So there is more space to improve but it’s very good start also for other people. That is reason for this article.

This is my first article so I hope it helped you. If you found some bug here or you have some improvements please write a comment and I will add it to the article with big thank to you.