Using rexray/efs plugin outside of EC2
“What’s EFS?” “It’s a limitless hard drive in the sky”
This is my first blog post in a while so it may be a little rusty.
“ShipItCon has been and gone, now it’s time for DockerCon.”
Docker Plugins 101
Docker plugins are incredibly useful. Easy to install, configure, update and remove. Plugins run in containers and only access the parts of the filesystem/permissions required to work. Check them out here : https://store.docker.com/
2 particular favorites are local-persist and rexray/efs. This post is about an issue with rexray/efs when run outside of an EC2 instance.
Persistent storage using rexray/efs plugin for Docker
When running Docker in EC2 instances the rexray/efs plugin works like a charm. It does what it says on the tin. Persistent storage is made available to your containers.
For more info on EFS check out : https://aws.amazon.com/efs/
When running a Docker host outside of EC2 things aren’t quite so smooth. According to a github issue back in Febuary 2017 the plugin only works from an EC2 instance.
“So a dumb question, but are you running this on an Amazon EC2 instance? Because the EFS and EBS drivers are only supported on EC2 instances. Likewise, the VirtualBox driver is only supported on a VBox VM.”
This may have changed since but installing the plugin on Docker 17.06.1 on an instance outside AWS failed with this error :
docker plugin install rexray/efs EFS_ACCESSKEY=AKA EFS_SECRETKEY=bK EFS_SECURITYGROUPS="sg-0 sg-6" EFS_REGION="us-west-2" REXRAY_LOGLEVEL=debug Plugin "rexray/efs" is requesting the following privileges: - network: [host] - mount: [/dev] - allow-all-devices: [true] - capabilities: [CAP_SYS_ADMIN] Do you grant the above permissions? [y/N] y latest: Pulling from rexray/efs 5fa4398d663c: Download complete Digest: sha256:9d507174acf741b233c827d54778d1e8e4ce40c386edda085448a593cec58dcb Status: Downloaded newer image for rexray/efs:latest Error response from daemon: dial unix /run/docker/plugins/0a476ca462f421ff00dd09b21a081fbccc5ecdbe5916a573c419200d037a29a2/rexray.sock: connect: no such file or directory
This looked like it might be an auth issue. It wasn’t. After poking around with the plugin I tried installing rexray on the host and creating a volume using the same EFS credentials.
curl -sSL https://dl.bintray.com/emccode/rexray/install | sh -s -- stable
Storage config file :
libstorage: service: efs server: services: efs: driver: efs efs: accessKey: AK secretKey: bK securityGroups: - sg-0 - sg-6 - sg-5 region: us-west-2 tag: test disableSessionCache: false statusMaxAttempts: 6 statusInitialDelay: 1s statusTimeout: 2m
Things are looking good now. Lot’s of output and then this error :
args=[efs supported] cmd=/var/lib/libstorage/lsx-linux host=unix:///var/run/libstorage/274597856.sock server=stream-stallion-ye service=efs stderr=error: error getting supported: Head http://169.254.169.254/latest/meta-data/: dial tcp 169.254.169.254:80: getsockopt: connection refused storageDriver=libstorage time=1504524907263 txCR=1504524907 txID=0e2cb37c-af34-459c-4998-5b811457f93a
Rexray is trying to query the metadata service which is provided by EC2. I’m not in EC2 … brick wall.
Fake it til you break it
Why does Rexray need to query the metadata service? Is something being returned that is required to make rexray/efs work? Doesn’t look like it. Using a fake metadata service, running in a container we can get past this. The following was performed on CentOS.
Step 1 : Clone the fake ec2 metadata repo
git clone https://github.com/bpholt/fake-ec2-metadata-service
Step 2 : Setup a loopback device and bind to 169.254.169.254 :
ifconfig lo:0 169.254.169.254 netmask 255.255.255.255 up
Step 3 : Update metadata endpoints
We need to edit ec2-metadata-service.rb and mock out a few endpoints to keep the rexray/efs plugin happy.
The edited file can be found here
The spec file also needs to be replaced. The spec file can be found here and should be dropped into the fake-ec2-metadata-service/spec directory.
Step 4 : Build the fake-ec2-metadata-service image
docker build -t fake-ec2-metadata-service .
Step 5 : Update docker-compose.yml
Edit the docker-compose.yml and replace the “image” name to “fake-ec2-metadata-service:latest” Change port 8169 to 80.
The edited file can be found here
Step 6 : Start the service
docker-compose up -d
Test it works
At this point you have a fake metadata service running.
Step 1 : Install plugin
docker plugin install rexray/efs EFS_ACCESSKEY=AKA EFS_SECRETKEY=bK EFS_SECURITYGROUPS="sg-0 sg-6" EFS_REGION="us-west-2" REXRAY_LOGLEVEL=debug
The plugin will query the fake metadata service and check that the instanceID exists. I just used a pre-existing t2.micro instanceID to get around this.
The plugin was installed successfully and you should be able to create a volume. Let’s try it out.
Step 2 : Create a volume
docker volume create --driver rexray/efs --name test-vol-1
It may take a few seconds for the volume to be created in EFS. Recommend logging into the AWS Console and checking the status.
Step 3 : Start a container and mount the volume
docker run -d --name test --mount type=volume,volume-driver=rexray/efs,src=test-vol-1,target=/app nginx:latest
Plugins are awesome, give them a try. I really like EFS but I like optionality. Having the option of running containers on hosts outside of EC2 is important. It’s worth noting this is just a dirty great workaround and will likely be out of date by the time this is commited.