淡淡回忆 发表于 2015-4-17 10:22:08

docker不稳定 short running containers with

  正常运行以下命令
  sudo docker run --rm busybox echo helloworld

/var/log/upstart/docker.log 日志如下:



2014/08/07 00:12:02 POST /v1.13/containers/create
+job create()
-job create() = OK (0)
2014/08/07 00:12:02 POST /v1.13/containers/5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0/attach?stderr=1&stdout=1&stream=1
+job container_inspect(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
-job container_inspect(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
+job attach(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
2014/08/07 00:12:02 POST /v1.13/containers/5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0/start
+job start(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
+job allocate_interface(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
-job allocate_interface(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
-job start(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
+job release_interface(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
-job release_interface(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
-job attach(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
2014/08/07 00:12:03 POST /v1.13/containers/5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0/wait
+job wait(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
-job wait(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
2014/08/07 00:12:03 GET /v1.13/containers/5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0/json
+job container_inspect(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
-job container_inspect(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
2014/08/07 00:12:03 DELETE /v1.13/containers/5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0?v=1
+job container_delete(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
-job container_delete(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
  
  我把这个写入crontab中,每分钟执行一次,有几率出现如下错误(如果更加频繁执行docker run那出现错误的几率更大,目前一分钟一次的出错概率大概是2天一个error)



Error response from daemon: Cannot destroy container b12b979ec303276ad8c7fedcdf687c7c3760c822297fb3b7f39f52954b90057e: Driver aufs failed to remove root filesystem b12b979ec303276ad8c7fedcdf687c7c3760c822297fb3b7f39f52954b90057e: rename /var/lib/docker/aufs/mnt/b12b979ec303276ad8c7fedcdf687c7c3760c822297fb3b7f39f52954b90057e /var/lib/docker/aufs/mnt/b12b979ec303276ad8c7fedcdf687c7c3760c822297fb3b7f39f52954b90057e-removing: device or resource busy


Error response from daemon: Cannot destroy container b103443f2f2d74862e4aa37132043b9ca4aa1ade7bbac37a514c051f859d0473: Unable to remove filesystem for b103443f2f2d74862e4aa37132043b9ca4aa1ade7bbac37a514c051f859d0473: remove /var/lib/docker/containers/b103443f2f2d74862e4aa37132043b9ca4aa1ade7bbac37a514c051f859d0473: directory not empty
  
  原因:基本还是docker不稳定造成的,并行快速的运行container的时候,有几率出现如如上错误
  与之有关的issue如下:
  https://github.com/docker/docker/issues/4095
  https://github.com/docker/docker/issues/3968
  https://github.com/docker/docker/issues/3823
  
  我机器的信息如下:



root@localhost:~# docker info
Containers: 4
Images: 12
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Dirs: 20
Execution Driver: native-0.2
Kernel Version: 3.8.0-44-generic
WARNING: No swap limit support
root@localhost:~# docker version
Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): d84a070
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): d84a070
root@hadoop-yarn-three:~# uname -a
Linux localhost 3.8.0-44-generic #66~precise1-Ubuntu SMP Tue Jul 15 04:01:04 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

  
  如果哪位发现docker解决了这个issue了,请帖出docker version和机器信息,多谢~
  @Tom哥,说 rhel6.5 内核版本2.6.32,docker version 1.1.2, 没有这个问题
  @IT人, 说ubuntu 14.04centos7.07.1,没有这个问题
  我还没有验证过。
  
页: [1]
查看完整版本: docker不稳定 short running containers with