amazon s3 - Docker Volumes in AWS - Write to EBS, EFS, S3 -


how can write docker containers in swarm write data file system mount such ebs, efs, or read data s3?

got efs, ebs, s3 plugins aws policy working

full answer details @ https://github.com/codedellemc/rexray/issues/820#issuecomment-323624765.

  • i used rexray docker volume plugin ebs, efs, , s3 support
    • the volume allows automatically load 1 of file-system mount inside container without using aws api. is, transparently container!!!

here example

docker service create --replicas 10      --name echo-to-efs      --mount type=volume,source=echo_data,target=/test,volume-driver=rexray/efs      alpine /bin/sh -c 'echo "writing $hostname" >> \                  /test/echo.log ; cat /test/echo.log ; sleep 30' iy4rzd4imq776abjlqdqf8l10

efs mounts enter image description here

ebs mounts enter image description here

install volume plugin

repeat each of them, using docker command-line arguments...

[root@ip-10-zz-yy-xx ec2-user]# sudo docker plugin install --grant-all-permissions rexray/s3fs latest: pulling rexray/s3fs 5a6437c91cc6: download complete digest: sha256:6ba105974d5298ff49def3c83e5547d41e573e0a3cb8604f1dcb1a0f90ee070b status: downloaded newer image rexray/s3fs:latest installed plugin rexray/s3fs

docker plugin ls

  • after setting userdata install 3 plugins , works properly!

[ec2-user@ip-10-82-48-59 ~]$ docker plugin ls id name description enabled 0eb15d4a9594 rexray/s3fs:latest rex-ray fuse driver amazon simple stor... true 6bae636cb377 rexray/ebs:latest rex-ray amazon ebs true 9e231d9177c2 rexray/efs:latest rex-ray amazon efs true

docker containers automagically creating/mounting volumes

as rule of thumb, learned following:

  • efs = concurrent writes (x replicas scenarios)
  • ebs = single writes (1 replicate scenarios)

so, example below shows following:

  • create 10 docker containers concurrently writing same file

last login: mon aug 21 00:34:46 2017 10.82.49.230 [ec2-user@ip-10-zz-yy-xx ~]$ docker service create --replicas 10 --name echo-to-efs
--mount type=volume,source=echo_data,target=/test,volume-driver=rexray/efs alpine /bin/sh -c 'echo "writing $hostname" >> /test/echo.log ; cat /test/echo.log ; sleep 30' iy4rzd4imq776abjlqdqf8l10

  • the docker ps shows of them writing , failures
    • if can explain why failures in case please let me know...
    • i have rexray_preempt=true so, not sure if that's not related

[ec2-user@ip-10-82-48-59 ~]$ docker service ps echo-to-efs id name image node desired state current state error ports 8c8lxqcoc3n5 echo-to-efs.1 alpine:latest ip-10-82-48-59.vpc.internal running running 8 seconds ago 23meciw5cvad echo-to-efs.2 alpine:latest ip-10-82-49-52.vpc.internal running running 2 seconds ago si7lg30kb13b \_ echo-to-efs.2 alpine:latest ip-10-82-48-93.vpc.internal shutdown failed minute ago "starting container failed: er…" rmj163xu97ai echo-to-efs.3 alpine:latest ip-10-82-48-96.vpc.internal running running 7 seconds ago 5jtao2yu5udt echo-to-efs.4 alpine:latest ip-10-82-48-95.vpc.internal running running 8 seconds ago wot9hd5nbjsz echo-to-efs.5 alpine:latest ip-10-82-49-71.vpc.internal running running 7 seconds ago hcbbgfiuga9c echo-to-efs.6 alpine:latest ip-10-82-48-140.vpc.internal running starting 49 seconds ago wrnnrsm16usr \_ echo-to-efs.6 alpine:latest ip-10-82-48-25.vpc.internal shutdown rejected 58 seconds ago "create echo_data: volumedrive…" w3v4x0pf9uri \_ echo-to-efs.6 alpine:latest ip-10-82-48-37.vpc.internal shutdown rejected minute ago "create echo_data: volumedrive…" okgda40hb4hy \_ echo-to-efs.6 alpine:latest ip-10-82-49-107.vpc.internal shutdown rejected minute ago "create echo_data: volumedrive…" bzoafno0gb4d echo-to-efs.7 alpine:latest ip-10-82-48-74.vpc.internal running running 7 seconds ago rk7uebz4cdql echo-to-efs.8 alpine:latest ip-10-82-49-111.vpc.internal running running 6 seconds ago hy1rfid4zitl echo-to-efs.9 alpine:latest ip-10-82-48-162.vpc.internal running running 7 seconds ago k5lmmmu44yh3 echo-to-efs.10 alpine:latest ip-10-82-48-187.vpc.internal running running 8 seconds ago

  • now can verify if file being written...

[ec2-user@ip-10-82-48-59 ~]$ docker service logs echo-to-efs echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal | writing 3da21bc313f7 echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal | writing 3da21bc313f7 echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal | writing 334b26ce3747 echo-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal | writing 3da21bc313f7 echo-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal | writing 334b26ce3747 echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal | writing 334b26ce3747 echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal | writing fcefd6eb5035 echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal | writing 84ffd56fdcfa echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal | writing b7ecbd01debe echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal | writing e5ef2a2821a2 echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal | writing 5179f81f04f9 echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal | writing 5e859dce0c2f echo-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal | writing fcefd6eb5035 echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal | writing fcefd6eb5035 echo-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal | writing 84ffd56fdcfa echo-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal | writing b7ecbd01debe echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal | writing 84ffd56fdcfa echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal | writing b7ecbd01debe echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal | writing e5ef2a2821a2 echo-to-efs.4.5jtao2yu5udt@ip-10-82-48-95.vpc.internal | writing 3da21bc313f7 echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal | writing 5179f81f04f9 echo-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal | writing e5ef2a2821a2 echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal | writing 5e859dce0c2f ... ...

  • now see current value in file.
    • the command here mount volume , @ same , read file efs mount.
    • it take long while until container prepares mount... 2min... not sure if that's expected.

[ec2-user@ip-10-82-48-59 ~]$ docker run -ti --volume-driver=rexray/efs -v echo_data:/test alpine cat /test/echo.log unable find image 'alpine:latest' locally latest: pulling library/alpine digest: sha256:1072e499f3f655a032e88542330cf75b02e7bdf673278f701d7ba61629ee3ebe status: downloaded newer image alpine:latest writing 3da21bc313f7 writing 334b26ce3747 writing fcefd6eb5035 writing 84ffd56fdcfa writing b7ecbd01debe writing e5ef2a2821a2 writing 5179f81f04f9 writing 5e859dce0c2f writing af94a5a29b4a writing a3f860caaf03 writing b7b3d30d2f8e writing cf65a52760c0 writing a551b529e599 writing 0533784ce9e5 writing d78f781baa4d writing 06b84229ec32 writing 4e3974cfb8a9 writing 007cd66062e7 writing 665f76589ae2 writing 68b04985559a writing 9a2e19e110fb writing 1ed7903abf86 writing 763c7eea52fb writing 529c7fff9c71 writing 7172d5fb56d9 writing c33294ce8d9d writing b95a5b5f1bc1 writing ff269d8a78ba writing b1299af12f30 writing 4f970a6e234d writing a4a1f331b3c9 writing bb3915d9d242 writing 0fbafba12a0c writing 017f70a769da writing 8d657b23571f writing 570e2fa76e16 writing 45a92f3d19b4 writing 3b02cb3675e4 writing e2d49358df6f

docker volumes status

  • docker volumes maps mounts plugins
    • this major win me!

[ec2-user@ip-10-82-48-59 ~]$ docker volume ls driver volume name rexray/s3fs:latest 7820-3357-5600-us-west-2 rexray/s3fs:latest cf-templates-8hmsxuf2muaa-us-west-2 rexray/efs:latest echo_data rexray/s3fs:latest marcello-12345 rexray/s3fs:latest marcello-testing-bucket-1 rexray/s3fs:latest marcello-testing-bucket-2 rexray/s3fs:latest new-config-publisher rexray/s3fs:latest publisher.intuit-caas rexray/s3fs:latest publisher.intuit-caas-new-update rexray/ebs:latest quay-registry-configs rexray/ebs:latest quay-registry-images-datastore rexray/ebs:latest quay-registry-mysql-datastore rexray/ebs:latest registry_configs rexray/ebs:latest registry_mysql-data rexray/s3fs:latest swarmbucket-vpc-155e5072


Comments

Post a Comment

Popular posts from this blog

ubuntu - PHP script to find files of certain extensions in a directory, returns populated array when run in browser, but empty array when run from terminal -

i running os ububtu 16.04. ultimate goal able run php script cron job every minute. here php script located @ /var/www/html/tests/test/index.php : <?php $directorypath = $_server['document_root']."/tests/test/data/"; echo "directorypath: $directorypath<br><br>";//check $imagefilepathsarray = glob($directorypath . "*.{png,gif}", glob_brace); echo "<br><br>imagefilepathsarray: "; print_r($imagefilepathsarray);//check ?> when open php script in browser visiting url my-ip-address/tests/test/index.php , get: directorypath: /var/www/html/tests/test/data/ imagefilepathsarray:array( [0] => /var/www/html/tests/test/data/pic.png [1] => /var/www/html/tests/test/data/pic.gif ) before writing cron job in crontab file, need check if can run anywhere using terminal . opened ubuntu terminal @ /home/ location, , executed in terminal: sudo php /var/www/html/tests/test/index.php i got following out...
Read more

php - How can i create a user dashboard -

today have questions you, want create user dashboard site, include (obviously), users data, example -email -username -and others stuff how can information, , print them in html page? if can, how can these information via php sessions? shuld use session? example, if user login, , start new session ( session_start() ), how can data print in dashboard? should use php "project"? in advice, have day , thank time!
Read more

javascript - How to detect toggling of the fullscreen-toolbar in jQuery Mobile? -

i have jquery mobile page fixed full-screen toolbar, data-tap-toggle enabled. straight under toolbar have positioned banner should slide upwards when toolbar hiding , slide downward when toolbar showing. jquery mobile toggles toolbar applying , removing ui-fixed-hidden class - sadly, can't find in documentation of toolbar widget toggle , hide or show event that. how can detect when toolbar toggled, reposition banner? .banner { position: fixed; background-color: darkseagreen; top: 46px; min-height: 48px; width: 100%; text-align: center; line-height: 48px; } <!doctype html> <html> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no"> <link rel="stylesheet" href="https://code.jquery.com/mobile/1.4.5/jquery.mobile-1.4.5.css"> <script src="https://code.jquery.com/jquery-1....
Read more