docker深入2-容器删除操作失败
1、报错Error response from daemon: driver "overlay" failed to remove root filesystem for xxx: remove /var/lib/docker/overlay/xxx/merged: device or resource busy
原因：被其他容器占用了资源。
分析方法：查找 /proc/*/mountinfo 来确认是哪个 pid 在 mount 资源。
解决办法：重启监控类型的 service
2、版本信息
# docker info
Server Version: 17.06.0-ce
Storage Driver: overlay
Backing Filesystem: extfs
Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
Kernel Version: 3.10.0-514.21.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.797GiB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
3、分析过程
1）尝试删除一个处于 dead 状态的容器：
# docker rm -f $(docker ps -a --filter status=dead -q |head -n 1)
Error response from daemon: driver "overlay" failed to remove root filesystem for 808acab2716420275cdb135ab964071cfc33406a34481354127635d3a282fa31: remove /var/lib/docker/overlay/88440438ea95b47e7459049fd765b51282afee4ad974107b0bf96d08d9c7763e/merged: device or resource busy
2）查找 /proc/*/mountinfo 来确认是哪个 pid 在 mount 资源：
# grep -l --color `docker ps -a --filter status=dead -q |head -n 1` /proc/*/mountinfo
3）查找 pid 确认进程的用途
# ps -f 7360
UID PIDPPIDC STIME TTY STAT TIME CMD
root 736073441 Aug16 ? Ssl 73:57 /usr/bin/cadvisor -logtostderr
4）还可以验证，使用的不是同一个 mount namespace
# ls -l /proc/$(cat /var/run/docker.pid)/ns/mnt /proc/7360/ns/mnt
lrwxrwxrwx 1 root root 0 Aug 21 15:55 /proc/11460/ns/mnt -> mnt:
lrwxrwxrwx 1 root root 0 Aug 21 15:55 /proc/7360/ns/mnt -> mnt:
#
5）尝试重启 cadvisor 服务：
# docker service ls |grep cadvisor
5f001c9293cf cadvisor global 3/3 google/cadvisor:latest
# docker service update --force cadvisor
#
6）再次删除：
# docker rm -f $(docker ps -a --filter status=dead -q |head -n 1)
808acab27164
#
符合预期。
4、结论
有其他的容器或服务，挂载的卷包含了 '/var/lib/docker' or '/' 后，将导致资源占用，从而引发异常。
如何绕过：找到这样的容器或者服务，重启，然后再重试删除操作
如何解决：未知，待跟进。
ZYXW、参考
1、github
https://github.com/moby/moby/issues/22260
https://github.com/moby/moby/issues/22260#issuecomment-323645024
2、docker: devicemapper fix for “device or resource busy” (EBUSY)
http://blog.hashbangbash.com/2014/11/docker-devicemapper-fix-for-device-or-resource-busy-ebusy/
页:
[1]