设为首页 收藏本站

运维网

 找回密码
 立即注册
查看: 1491|回复: 0

[经验分享] ORACLE Bug 4431215 引发的血案—原因分析篇

[复制链接]
12343052

尚未签到
发表于 2018-9-26 12:11:59 | 显示全部楼层 |阅读模式
　　抱歉上文编辑不下，才续写此文
可为什么该目录会丢失呢，原因是因为ORACLE的BUG，且看故障元凶：Bug 4431215的定位过程，看如下三个日志，节点1在归档到8211发生错误，节点2在归档到6852发生错误，而1点钟有个RMAN定时JOB发起2级备份，备份后会删除归档日志，从RMAN日志可以看出，最大的备份号分别为8210和6851，这和节点ALERT日志是完全一致的，可见在RMAN备份后归档便发生了错误，而RMAN备份后的瞬间，归档目录是空的，这难免让人想到ORACLE在ASM环境下的一个BUG：“在ASM上建立的目录名称和SID前缀相同，则这个目录下的所有目录一旦为空，就会被ASM实例自动删除，不管这个目录是ASM自动建立，还是用户手工建立的。”。登陆ORACLE官方站点确认有该BUG：“Bug 4431215 - "backup archivelog all delete input" from RMAN removes ASM directory [ID 4431215.8]”，至此故障调查完成。

　　


  • 节点1日志：
  • Wed Jul 13 01:22:55 2011
  • Thread 1 advanced to log sequence 8210 (LGWR switch)
  •   Current log# 2 seq# 8210 mem# 0: +DATA/port/onlinelog/group_2.264.697238179
  •   Current log# 2 seq# 8210 mem# 1: +DATA/port/onlinelog/group_2.265.697238179
  • Wed Jul 13 01:23:00 2011
  • ALTER SYSTEM ARCHIVE LOG
  • Wed Jul 13 01:23:01 2011
  • Thread 1 cannot allocate new log, sequence 8211
  • Checkpoint not complete
  •   Current log# 2 seq# 8210 mem# 0: +DATA/port/onlinelog/group_2.264.697238179
  •   Current log# 2 seq# 8210 mem# 1: +DATA/port/onlinelog/group_2.265.697238179

  • 节点2日志：
  • Wed Jul 13 01:20:27 2011
  • Thread 2 advanced to log sequence 6851 (LGWR switch)
  •   Current log# 3 seq# 6851 mem# 0: +DATA/port/onlinelog/group_3.268.697238217
  •   Current log# 3 seq# 6851 mem# 1: +DATA/port/onlinelog/group_3.269.697238219
  • Thread 2 cannot allocate new log, sequence 6852
  • Checkpoint not complete
  •   Current log# 3 seq# 6851 mem# 0: +DATA/port/onlinelog/group_3.268.697238217
  •   Current log# 3 seq# 6851 mem# 1: +DATA/port/onlinelog/group_3.269.697238219
  • Wed Jul 13 01:20:36 2011
  • Thread 2 advanced to log sequence 6852 (LGWR switch)
  •   Current log# 4 seq# 6852 mem# 0: +DATA/port/onlinelog/group_4.270.697238219
  •   Current log# 4 seq# 6852 mem# 1: +DATA/port/onlinelog/group_4.271.697238221

  • 故障发生前的RMAN备份日志，备份完成后删除备份的归档日志：
  • channel d1: specifying archive log(s) in backup set
  • input archive log thread=1 sequence=8180 recid=22515 stamp=756327382
  • input archive log thread=1 sequence=8181 recid=22516 stamp=756327390
  • input archive log thread=1 sequence=8182 recid=22517 stamp=756327399
  • input archive log thread=1 sequence=8183 recid=22519 stamp=756327406
  • input archive log thread=1 sequence=8184 recid=22520 stamp=756327418
  • input archive log thread=1 sequence=8185 recid=22521 stamp=756327426
  • input archive log thread=1 sequence=8186 recid=22522 stamp=756327434
  • input archive log thread=1 sequence=8187 recid=22524 stamp=756327443
  • input archive log thread=1 sequence=8188 recid=22525 stamp=756327453
  • input archive log thread=1 sequence=8189 recid=22527 stamp=756327462
  • input archive log thread=1 sequence=8190 recid=22528 stamp=756327469
  • input archive log thread=1 sequence=8191 recid=22529 stamp=756327477
  • input archive log thread=1 sequence=8192 recid=22531 stamp=756327488
  • input archive log thread=1 sequence=8193 recid=22532 stamp=756327496
  • input archive log thread=1 sequence=8194 recid=22533 stamp=756327505
  • input archive log thread=1 sequence=8195 recid=22535 stamp=756327514
  • input archive log thread=1 sequence=8196 recid=22536 stamp=756327523
  • input archive log thread=1 sequence=8197 recid=22537 stamp=756327538
  • input archive log thread=1 sequence=8198 recid=22538 stamp=756327653
  • input archive log thread=1 sequence=8199 recid=22540 stamp=756327664
  • input archive log thread=1 sequence=8200 recid=22541 stamp=756327673
  • input archive log thread=1 sequence=8201 recid=22543 stamp=756327686
  • input archive log thread=1 sequence=8202 recid=22544 stamp=756327695
  • input archive log thread=1 sequence=8203 recid=22545 stamp=756327704
  • input archive log thread=1 sequence=8204 recid=22547 stamp=756327717
  • input archive log thread=1 sequence=8205 recid=22548 stamp=756329007
  • input archive log thread=1 sequence=8206 recid=22549 stamp=756338512
  • input archive log thread=1 sequence=8207 recid=22551 stamp=756343058
  • input archive log thread=1 sequence=8208 recid=22552 stamp=756345806
  • input archive log thread=1 sequence=8209 recid=22553 stamp=756350578
  • input archive log thread=1 sequence=8210 recid=22555 stamp=756350584
  • input archive log thread=2 sequence=6800 recid=22340 stamp=756264103
  • input archive log thread=2 sequence=6801 recid=22345 stamp=756265938
  • input archive log thread=2 sequence=6802 recid=22356 stamp=756266232
  • input archive log thread=2 sequence=6803 recid=22357 stamp=756266255
  • input archive log thread=2 sequence=6804 recid=22359 stamp=756266260
  • input archive log thread=2 sequence=6805 recid=22362 stamp=756266269
  • input archive log thread=2 sequence=6806 recid=22363 stamp=756266272
  • input archive log thread=2 sequence=6807 recid=22365 stamp=756266287
  • input archive log thread=2 sequence=6808 recid=22369 stamp=756267485
  • input archive log thread=2 sequence=6809 recid=22376 stamp=756291174
  • input archive log thread=2 sequence=6810 recid=22380 stamp=756293726
  • input archive log thread=2 sequence=6811 recid=22386 stamp=756299397
  • input archive log thread=2 sequence=6812 recid=22390 stamp=756311219
  • input archive log thread=2 sequence=6813 recid=22396 stamp=756313749
  • input archive log thread=2 sequence=6814 recid=22402 stamp=756316744
  • input archive log thread=2 sequence=6815 recid=22406 stamp=756319318
  • input archive log thread=2 sequence=6816 recid=22412 stamp=756325509
  • input archive log thread=2 sequence=6817 recid=22415 stamp=756326651
  • input archive log thread=2 sequence=6818 recid=22420 stamp=756326683
  • input archive log thread=2 sequence=6819 recid=22425 stamp=756326710
  • input archive log thread=2 sequence=6820 recid=22428 stamp=756326732
  • input archive log thread=2 sequence=6821 recid=22433 stamp=756326756
  • input archive log thread=2 sequence=6822 recid=22438 stamp=756326780
  • input archive log thread=2 sequence=6823 recid=22441 stamp=756326801
  • input archive log thread=2 sequence=6824 recid=22445 stamp=756326823
  • input archive log thread=2 sequence=6825 recid=22450 stamp=756326850
  • input archive log thread=2 sequence=6826 recid=22455 stamp=756326878
  • input archive log thread=2 sequence=6827 recid=22460 stamp=756326905
  • input archive log thread=2 sequence=6828 recid=22464 stamp=756326930
  • input archive log thread=2 sequence=6829 recid=22468 stamp=756326951
  • input archive log thread=2 sequence=6830 recid=22472 stamp=756326969
  • input archive log thread=2 sequence=6831 recid=22477 stamp=756326991
  • input archive log thread=2 sequence=6832 recid=22481 stamp=756327012
  • channel d1: starting piece 1 at 13-JUL-11
  • channel d1: finished piece 1 at 13-JUL-11
  • piece handle=/u01/rmanbak/arch_ZJPORT_1hmha00r_1_1 tag=BKARCH comment=NONE
  • channel d1: backup set complete, elapsed time: 00:03:17
  • channel d1: deleting archive log(s)
  • archive log filename=+DATA/zjport/archivelog/1_8180_697238176.dbf recid=22515 stamp=756327382
  • archive log filename=+DATA/zjport/archivelog/1_8181_697238176.dbf recid=22516 stamp=756327390
  • archive log filename=+DATA/zjport/archivelog/1_8182_697238176.dbf recid=22517 stamp=756327399
  • archive log filename=+DATA/zjport/archivelog/1_8183_697238176.dbf recid=22519 stamp=756327406
  • archive log filename=+DATA/zjport/archivelog/1_8184_697238176.dbf recid=22520 stamp=756327418
  • archive log filename=+DATA/zjport/archivelog/1_8185_697238176.dbf recid=22521 stamp=756327426
  • archive log filename=+DATA/zjport/archivelog/1_8186_697238176.dbf recid=22522 stamp=756327434
  • archive log filename=+DATA/zjport/archivelog/1_8187_697238176.dbf recid=22524 stamp=756327443
  • archive log filename=+DATA/zjport/archivelog/1_8188_697238176.dbf recid=22525 stamp=756327453
  • archive log filename=+DATA/zjport/archivelog/1_8189_697238176.dbf recid=22527 stamp=756327462
  • archive log filename=+DATA/zjport/archivelog/1_8190_697238176.dbf recid=22528 stamp=756327469
  • archive log filename=+DATA/zjport/archivelog/1_8191_697238176.dbf recid=22529 stamp=756327477
  • archive log filename=+DATA/zjport/archivelog/1_8192_697238176.dbf recid=22531 stamp=756327488
  • archive log filename=+DATA/zjport/archivelog/1_8193_697238176.dbf recid=22532 stamp=756327496
  • archive log filename=+DATA/zjport/archivelog/1_8194_697238176.dbf recid=22533 stamp=756327505
  • archive log filename=+DATA/zjport/archivelog/1_8195_697238176.dbf recid=22535 stamp=756327514
  • archive log filename=+DATA/zjport/archivelog/1_8196_697238176.dbf recid=22536 stamp=756327523
  • archive log filename=+DATA/zjport/archivelog/1_8197_697238176.dbf recid=22537 stamp=756327538
  • archive log filename=+DATA/zjport/archivelog/1_8198_697238176.dbf recid=22538 stamp=756327653
  • archive log filename=+DATA/zjport/archivelog/1_8199_697238176.dbf recid=22540 stamp=756327664
  • archive log filename=+DATA/zjport/archivelog/1_8200_697238176.dbf recid=22541 stamp=756327673
  • archive log filename=+DATA/zjport/archivelog/1_8201_697238176.dbf recid=22543 stamp=756327686
  • archive log filename=+DATA/zjport/archivelog/1_8202_697238176.dbf recid=22544 stamp=756327695
  • archive log filename=+DATA/zjport/archivelog/1_8203_697238176.dbf recid=22545 stamp=756327704
  • archive log filename=+DATA/zjport/archivelog/1_8204_697238176.dbf recid=22547 stamp=756327717
  • archive log filename=+DATA/zjport/archivelog/1_8205_697238176.dbf recid=22548 stamp=756329007
  • archive log filename=+DATA/zjport/archivelog/1_8206_697238176.dbf recid=22549 stamp=756338512
  • archive log filename=+DATA/zjport/archivelog/1_8207_697238176.dbf recid=22551 stamp=756343058
  • archive log filename=+DATA/zjport/archivelog/1_8208_697238176.dbf recid=22552 stamp=756345806
  • archive log filename=+DATA/zjport/archivelog/1_8209_697238176.dbf recid=22553 stamp=756350578
  • archive log filename=+DATA/zjport/archivelog/1_8210_697238176.dbf recid=22555 stamp=756350584=======节点1的归档备份到8210
  • archive log filename=+DATA/zjport/archivelog/2_6800_697238176.dbf recid=22340 stamp=756264103
  • archive log filename=+DATA/zjport/archivelog/2_6801_697238176.dbf recid=22345 stamp=756265938
  • archive log filename=+DATA/zjport/archivelog/2_6802_697238176.dbf recid=22356 stamp=756266232
  • archive log filename=+DATA/zjport/archivelog/2_6803_697238176.dbf recid=22357 stamp=756266255
  • archive log filename=+DATA/zjport/archivelog/2_6804_697238176.dbf recid=22359 stamp=756266260
  • archive log filename=+DATA/zjport/archivelog/2_6805_697238176.dbf recid=22362 stamp=756266269
  • archive log filename=+DATA/zjport/archivelog/2_6806_697238176.dbf recid=22363 stamp=756266272
  • archive log filename=+DATA/zjport/archivelog/2_6807_697238176.dbf recid=22365 stamp=756266287
  • archive log filename=+DATA/zjport/archivelog/2_6808_697238176.dbf recid=22369 stamp=756267485
  • archive log filename=+DATA/zjport/archivelog/2_6809_697238176.dbf recid=22376 stamp=756291174
  • archive log filename=+DATA/zjport/archivelog/2_6810_697238176.dbf recid=22380 stamp=756293726
  • archive log filename=+DATA/zjport/archivelog/2_6811_697238176.dbf recid=22386 stamp=756299397
  • archive log filename=+DATA/zjport/archivelog/2_6812_697238176.dbf recid=22390 stamp=756311219
  • archive log filename=+DATA/zjport/archivelog/2_6813_697238176.dbf recid=22396 stamp=756313749
  • archive log filename=+DATA/zjport/archivelog/2_6814_697238176.dbf recid=22402 stamp=756316744
  • archive log filename=+DATA/zjport/archivelog/2_6815_697238176.dbf recid=22406 stamp=756319318
  • archive log filename=+DATA/zjport/archivelog/2_6816_697238176.dbf recid=22412 stamp=756325509
  • archive log filename=+DATA/zjport/archivelog/2_6817_697238176.dbf recid=22415 stamp=756326651
  • archive log filename=+DATA/zjport/archivelog/2_6818_697238176.dbf recid=22420 stamp=756326683
  • archive log filename=+DATA/zjport/archivelog/2_6819_697238176.dbf recid=22425 stamp=756326710
  • archive log filename=+DATA/zjport/archivelog/2_6820_697238176.dbf recid=22428 stamp=756326732
  • archive log filename=+DATA/zjport/archivelog/2_6821_697238176.dbf recid=22433 stamp=756326756
  • archive log filename=+DATA/zjport/archivelog/2_6822_697238176.dbf recid=22438 stamp=756326780
  • archive log filename=+DATA/zjport/archivelog/2_6823_697238176.dbf recid=22441 stamp=756326801
  • archive log filename=+DATA/zjport/archivelog/2_6824_697238176.dbf recid=22445 stamp=756326823
  • archive log filename=+DATA/zjport/archivelog/2_6825_697238176.dbf recid=22450 stamp=756326850
  • archive log filename=+DATA/zjport/archivelog/2_6826_697238176.dbf recid=22455 stamp=756326878
  • archive log filename=+DATA/zjport/archivelog/2_6827_697238176.dbf recid=22460 stamp=756326905
  • archive log filename=+DATA/zjport/archivelog/2_6828_697238176.dbf recid=22464 stamp=756326930
  • archive log filename=+DATA/zjport/archivelog/2_6829_697238176.dbf recid=22468 stamp=756326951
  • archive log filename=+DATA/zjport/archivelog/2_6830_697238176.dbf recid=22472 stamp=756326969
  • archive log filename=+DATA/zjport/archivelog/2_6831_697238176.dbf recid=22477 stamp=756326991
  • archive log filename=+DATA/zjport/archivelog/2_6832_697238176.dbf recid=22481 stamp=756327012
  • channel d1: starting compressed archive log backupset
  • channel d1: specifying archive log(s) in backup set
  • input archive log thread=2 sequence=6833 recid=22484 stamp=756327033
  • input archive log thread=2 sequence=6834 recid=22488 stamp=756327058
  • input archive log thread=2 sequence=6835 recid=22492 stamp=756327082
  • input archive log thread=2 sequence=6836 recid=22497 stamp=756327112
  • input archive log thread=2 sequence=6837 recid=22501 stamp=756327145
  • input archive log thread=2 sequence=6838 recid=22505 stamp=756327170
  • input archive log thread=2 sequence=6839 recid=22510 stamp=756327198
  • input archive log thread=2 sequence=6840 recid=22513 stamp=756327222
  • input archive log thread=2 sequence=6841 recid=22518 stamp=756327253
  • input archive log thread=2 sequence=6842 recid=22523 stamp=756327287
  • input archive log thread=2 sequence=6843 recid=22526 stamp=756327311
  • input archive log thread=2 sequence=6844 recid=22530 stamp=756327336
  • input archive log thread=2 sequence=6845 recid=22534 stamp=756327363
  • input archive log thread=2 sequence=6846 recid=22539 stamp=756327505
  • input archive log thread=2 sequence=6847 recid=22542 stamp=756327535
  • input archive log thread=2 sequence=6848 recid=22546 stamp=756327566
  • input archive log thread=2 sequence=6849 recid=22550 stamp=756341841
  • input archive log thread=2 sequence=6850 recid=22554 stamp=756350429
  • input archive log thread=2 sequence=6851 recid=22556 stamp=756350436
  • channel d1: starting piece 1 at 13-JUL-11
  • channel d1: finished piece 1 at 13-JUL-11
  • piece handle=/u01/rmanbak/arch_ZJPORT_1imha073_1_1 tag=BKARCH comment=NONE
  • channel d1: backup set complete, elapsed time: 00:00:16
  • channel d1: deleting archive log(s)
  • archive log filename=+DATA/zjport/archivelog/2_6833_697238176.dbf recid=22484 stamp=756327033
  • archive log filename=+DATA/zjport/archivelog/2_6834_697238176.dbf recid=22488 stamp=756327058
  • archive log filename=+DATA/zjport/archivelog/2_6835_697238176.dbf recid=22492 stamp=756327082
  • archive log filename=+DATA/zjport/archivelog/2_6836_697238176.dbf recid=22497 stamp=756327112
  • archive log filename=+DATA/zjport/archivelog/2_6837_697238176.dbf recid=22501 stamp=756327145
  • archive log filename=+DATA/zjport/archivelog/2_6838_697238176.dbf recid=22505 stamp=756327170
  • archive log filename=+DATA/zjport/archivelog/2_6839_697238176.dbf recid=22510 stamp=756327198
  • archive log filename=+DATA/zjport/archivelog/2_6840_697238176.dbf recid=22513 stamp=756327222
  • archive log filename=+DATA/zjport/archivelog/2_6841_697238176.dbf recid=22518 stamp=756327253
  • archive log filename=+DATA/zjport/archivelog/2_6842_697238176.dbf recid=22523 stamp=756327287
  • archive log filename=+DATA/zjport/archivelog/2_6843_697238176.dbf recid=22526 stamp=756327311
  • archive log filename=+DATA/zjport/archivelog/2_6844_697238176.dbf recid=22530 stamp=756327336
  • archive log filename=+DATA/port/archivelog/2_6845_697238176.dbf recid=22534 stamp=756327363
  • archive log filename=+DATA/port/archivelog/2_6846_697238176.dbf recid=22539 stamp=756327505
  • archive log filename=+DATA/port/archivelog/2_6847_697238176.dbf recid=22542 stamp=756327535
  • archive log filename=+DATA/port/archivelog/2_6848_697238176.dbf recid=22546 stamp=756327566
  • archive log filename=+DATA/port/archivelog/2_6849_697238176.dbf recid=22550 stamp=756341841
  • archive log filename=+DATA/port/archivelog/2_6850_697238176.dbf recid=22554 stamp=756350429
  • archive log filename=+DATA/port/archivelog/2_6851_697238176.dbf recid=22556 stamp=756350436==========节点2的归档备份到8210
  • Finished backup at 13-JUL-11

  • released channel: d1

  • executing global script: global_bkctl

  • allocated channel: d1
  • channel d1: sid=906 instance=port1 devtype=DISK

  • Starting backup at 13-JUL-11
  • channel d1: starting compressed full datafile backupset
  • channel d1: specifying datafile(s) in backupset
  • including current control file in backupset
  • channel d1: starting piece 1 at 13-JUL-11
  • channel d1: finished piece 1 at 13-JUL-11
  • piece handle=/u01/rmanbak/backupctl.ctl tag=BKCTL comment=NONE
  • channel d1: backup set complete, elapsed time: 00:00:02
  • Finished backup at 13-JUL-11

  • released channel: d1

  • Recovery Manager complete.
　　

事实上只要处理时间稍微充裕那么一点，我们不难发现更简便的方法，通过ASMCMD直接创建丢失的archivelog目录就可以了。

我们最后的处理：

通过ASMCDM在+DATA下新建了archivelog目录，然后在此目录下新建子目录，为什么不建立文件（因为笔者没有找到10G ASM下建文件的命令，如果有人知道，烦请指导，谢谢）





运维网声明 1、欢迎大家加入本站运维交流群:群②:261659950 群⑤:202807635 群⑦870801961 群⑧679858003
2、本站所有主题由该帖子作者发表，该帖子作者与运维网享有帖子相关版权
3、所有作品的著作权均归原作者享有，请您和我们一样尊重他人的著作权等合法权益。如果您对作品感到满意，请购买正版
4、禁止制作、复制、发布和传播具有反动、淫秽、色情、暴力、凶杀等内容的信息，一经发现立即删除。若您因此触犯法律，一切后果自负，我们对此不承担任何责任
5、所有资源均系网友上传或者通过网络收集，我们仅提供一个展示、介绍、观摩学习的平台，我们不对其内容的准确性、可靠性、正当性、安全性、合法性等负责，亦不承担任何法律责任
6、所有作品仅供您个人学习、研究或欣赏，不得用于商业或者其他用途，否则，一切后果均由您自己承担，我们对此不承担任何法律责任
7、如涉及侵犯版权等问题，请您及时通知我们，我们将立即采取措施予以解决
8、联系人Email：admin@iyunv.com 网址：www.yunweiku.com

所有资源均系网友上传或者通过网络收集,我们仅提供一个展示、介绍、观摩学习的平台,我们不对其承担任何法律责任,如涉及侵犯版权等问题,请您及时通知我们,我们将立即处理,联系人Email:kefu@iyunv.com,QQ:1061981298 本贴地址：https://www.yunweiku.com/thread-602341-1-1.html 上篇帖子： oracle占用系统资源很高 sqlplus 无响应 下篇帖子： WindowsXP下ORACLE 10g安装与操作图解
回复

使用道具 举报

返回列表
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

扫码加入运维网微信交流群X

扫码加入运维网微信交流群

扫描二维码加入运维网微信交流群，最新一手资源尽在官方微信交流群！快快加入我们吧...

扫描微信二维码查看详情

客服 E-mail:iyunvcom@gmail.com

本站由青云提供云计算服务

运维网--中国最专业的运维工程师交流社区

豫ICP备20007574号-1 Copyright © 2012-2025

关于我们

发展历程

联系我们

关注我们

广告服务

广告案例

隐私条款

免责声明

用户登录

用户注册

版主守则

申请版主

手机版

金币获取

帮助中心

每日签到

Good Good

Study

Day Day

UP


客服E-mail:kefu@iyunv.com 客服QQ：1061981298


QQ群⑦：运维网交流群⑦ QQ群⑧：运维网交流群⑧ k8s群：运维网kubernetes交流群


提醒：禁止发布任何违反国家法律、法规的言论与图片等内容；本站内容均来自个人观点与网络等信息，非本站认同之观点.


本站大部分资源是网友从网上搜集分享而来，其版权均归原作者及其网站所有，我们尊重他人的合法权益，如有内容侵犯您的合法权益，请及时与我们联系进行核实删除！



合作伙伴: 青云cloud

快速回复 返回顶部 返回列表