FAL[client, USER]: Error 16191 connecting to PRODNY for fetching gap sequence Error 1017 received logging on to the standby

SQL> select * from v$archive_gap;

   THREAD# LOW_SEQUENCE# HIGH_SEQUENCE#
---------- ------------- --------------
         1          2192           2212

SQL>

Stand by Alert log error:-

FAL[client, USER]: Error 16191 connecting to PRODNY for fetching gap sequence
Sat Dec 29 10:12:21 2012
Fetching gap sequence in thread 1, gap sequence 2192-2212
Error 1017 received logging on to the standby
------------------------------------------------------------
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
      returning error ORA-16191
------------------------------------------------------------
FAL[client, USER]: Error 16191 connecting to PRODNY for fetching gap sequence
Sat Dec 29 10:12:31 2012
Fetching gap sequence in thread 1, gap sequence 2192-2212
Error 1017 received logging on to the standby

Solution :-

We have to recover Archive log from catalog database.

[orprod@muthu]$ rman target / catalog rman/muth123@RMAN

Recovery Manager: Release 11.2.0.2.0 - Production on Sat Dec 29 09:44:53 2012

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: PROD (DBID=2808283451)
connected to recovery catalog database

RMAN> run {
set archivelog destination to '/oracle/PROD/db/apps_st/archive';
restore archivelog from logseq=2192  until logseq=2212  thread=1;
 }2> 3> 4>

executing command: SET ARCHIVELOG DESTINATION

Starting restore at 29-DEC-12
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=512 device type=DISK

archived log for thread 1 with sequence 2211 is already on disk as file /oracle/PROD/db/apps_st/archive/1_2211_785817078.dbf
archived log for thread 1 with sequence 2212 is already on disk as file /oracle/PROD/db/apps_st/archive/1_2212_785817078.dbf
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2192
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31041
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31041 tag=PROD ARCLOG BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2193
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31042
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31042 tag=PROD ARCLOG BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2194
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31043
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31043 tag=PROD ARCLOG BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2195
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31044
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31044 tag=PROD ARCLOG BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2196
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31045
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31045 tag=PROD ARCLOG BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2197
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31046
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31046 tag=PROD ARCLOG BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2199
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31048
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31048 tag=PROD ARCLOG BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2198
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31047
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31047 tag=PROD ARCLOG BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2201
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31050
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31050 tag=PROD ARCLOG BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2200
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31049
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31049 tag=PROD ARCLOG BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2203
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31052
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31052 tag=PROD ARCLOG BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2202
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31051
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121228_31051 tag=PROD ARCLOG BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2206
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121229_31069
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121229_31069 tag=TAG20121229T000928
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2205
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121229_31068
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121229_31068 tag=TAG20121229T000928
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2204
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121229_31067
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121229_31067 tag=TAG20121229T000928
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2208
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121229_31074
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121229_31074 tag=PROD ARCLOG BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2207
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121229_31073
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121229_31073 tag=PROD ARCLOG BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2210
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121229_31076
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121229_31076 tag=PROD ARCLOG BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archived log restore to user-specified destination
archived log destination=/oracle/PROD/db/apps_st/archive
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2209
channel ORA_DISK_1: reading from backup piece /backup_pip/Prod_EBS/rman/PROD_ARCH_20121229_31075
channel ORA_DISK_1: piece handle=/backup_pip/Prod_EBS/rman/PROD_ARCH_20121229_31075 tag=PROD ARCLOG BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
Finished restore at 29-DEC-12

RMAN> exit

Login to stand by server stop and start the mrp services:-

SQL> alter database recover managed standby database cancel;

Database altered.

SQL> alter database recover managed standby database disconnect from session;

Database altered.

SQL>

Stand by alert log detail:-

Archived Log entry 350 added for thread 1 sequence 2211 rlc 785817078 ID 0xbc84a3b dest 2:
RFS[3]: Opened log for thread 1 sequence 2212 dbid -1486683845 branch 785817078
Sat Dec 29 10:23:01 2012
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2202_785817078.dbf
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2203_785817078.dbf
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2204_785817078.dbf
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2205_785817078.dbf
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2206_785817078.dbf
Media Recovery Waiting for thread 1 sequence 2207 (in transit)
Archived Log entry 351 added for thread 1 sequence 2212 rlc 785817078 ID 0xbc84a3b dest 2:
Sat Dec 29 10:23:10 2012
Archived Log entry 352 added for thread 1 sequence 2207 rlc 785817078 ID 0x0 dest 2:
Sat Dec 29 10:23:11 2012
RFS[2]: Opened log for thread 1 sequence 2215 dbid -1486683845 branch 785817078
RFS[1]: Opened log for thread 1 sequence 2216 dbid -1486683845 branch 785817078
Sat Dec 29 10:23:11 2012
RFS[3]: Opened log for thread 1 sequence 2217 dbid -1486683845 branch 785817078
Archived Log entry 353 added for thread 1 sequence 2216 rlc 785817078 ID 0xbc84a3b dest 2:
Archived Log entry 354 added for thread 1 sequence 2217 rlc 785817078 ID 0xbc84a3b dest 2:
Sat Dec 29 10:23:12 2012
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2207_785817078.dbf
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2208_785817078.dbf
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2209_785817078.dbf
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2210_785817078.dbf
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2211_785817078.dbf
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2212_785817078.dbf
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2213_785817078.dbf
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2214_785817078.dbf
Media Recovery Waiting for thread 1 sequence 2215 (in transit)
RFS[1]: Opened log for thread 1 sequence 2218 dbid -1486683845 branch 785817078
Sat Dec 29 10:23:17 2012
Primary database is in MAXIMUM PERFORMANCE mode
RFS[4]: Assigned to RFS process 24407
RFS[4]: No standby redo logfiles created for thread 1
RFS[4]: Opened log for thread 1 sequence 2219 dbid -1486683845 branch 785817078
Archived Log entry 355 added for thread 1 sequence 2218 rlc 785817078 ID 0xbc84a3b dest 2:
Sat Dec 29 10:23:25 2012
Archived Log entry 356 added for thread 1 sequence 2215 rlc 785817078 ID 0xbc84a3b dest 2:
Sat Dec 29 10:23:30 2012
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2215_785817078.dbf
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2216_785817078.dbf
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2217_785817078.dbf
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2218_785817078.dbf
Media Recovery Waiting for thread 1 sequence 2219 (in transit)
Sat Dec 29 10:26:04 2012
Archived Log entry 357 added for thread 1 sequence 2219 rlc 785817078 ID 0xbc84a3b dest 2:
RFS[4]: No standby redo logfiles created for thread 1
RFS[4]: Opened log for thread 1 sequence 2220 dbid -1486683845 branch 785817078
Sat Dec 29 10:26:06 2012
Media Recovery Log /oracle/PROD/db/apps_st/archive/1_2219_785817078.dbf
Media Recovery Waiting for thread 1 sequence 2220 (in transit)


verify the archive log gap:-

SQL> select * from v$archive_gap;

no rows selected

SQL>