Dear all,
Duplication Status 150 and 191 (Could not get raw segment - no such object. Refer to the replication logs on the source storage server for more information. V-454-105)
We are experiencing duplication status 150 for duplications to from disk to disk and 191 from disk to tape. We believe that the issue occurres due to big workload at the media server at 02:00 am (both backups and duplications jobs are trying to access the disk pool and read/write data). We have procceed at the following tune up's but the issue still occures:
- Maximum I/O streams on the Disk Pool = 60
- Changes to the LIFECYCLE_PARAMETERS file
- Changes the Media server connect timeouts to 60 seconds from 30 which were the default value.
Old LIFECYCLE_PARAMETERS
DUPLICATION_SESSION_INTERVAL_MINUTES = 5
MAX_MINUTES_TIL_FORCE_SMALL_DUPLICATION_JOB = 30
MAX_GB_SIZE_PER_DUPLICATION_JOB = 200
MIN_GB_SIZE_PER_DUPLICATION_JOB = 16
New LIFECYCLE_PARAMETERS
DUPLICATION_SESSION_INTERVAL_MINUTES = 30
MAX_MINUTES_TIL_FORCE_SMALL_DUPLICATION_JOB = 60
MAX_GB_SIZE_PER_DUPLICATION_JOB = 400
MIN_GB_SIZE_PER_DUPLICATION_JOB = 80
The logs when the issue occurs:
Detailed Status from failed duplication:
9 Μαϊ 2015 2:15:35 πμ - Info bpdm (pid=2545) Creating OPT_DUP_END marker
9 Μαϊ 2015 2:23:35 πμ - Critical bpdm (pid=2545) Storage Server Error: (Storage server: PureDisk:bs-kifissos) async_get_job_status: Replication started but failed to complete successfully: __forward_SO: Could not get raw segment a7e4f1f4ff09cf55be3caee38aad60a0, len 16777216: no such object. Refer to the replication logs on the source storage server for more information. V-454-105
9 Μαϊ 2015 2:23:35 πμ - Critical bpdm (pid=2545) sts_copy_extent failed: error 2060014 operation aborted
9 Μαϊ 2015 2:23:42 πμ - Error bpduplicate (pid=19933) host bs-pireos backup id r6-osn6-kif_1431126003 optimized duplication failed, termination requested by administrator (150).
9 Μαϊ 2015 2:23:42 πμ - Info bs-pireos (pid=2545) StorageServer=PureDisk:bs-kifissos; Report=PDDO Stats for (bs-kifissos): scanned: 2 KB, CR sent: 0 KB, CR sent over FC: 0 KB, dedup: 100.0%
9 Μαϊ 2015 2:23:42 πμ - Error bpduplicate (pid=19933) db_IMAGE() failed: Images are in process. Please, run the nbstlutil command to get a list of those images if you need to cancel or expire them. (1519)
9 Μαϊ 2015 2:23:42 πμ - Error bpduplicate (pid=19933) Duplicate of backupid r6-osn6-kif_1431126003 failed, termination requested by administrator (150).
Replication logs on the source storage server (bs-kifissos):
May 09 02:16:31 INFO [140159293536000]: sessionStartAgent: Server is Version 7.0006.0013.112, Protocol Version 6.6.1
May 09 02:16:52 INFO [140159294064384]: sessionStartAgent: Server is Version 7.0006.0013.112, Protocol Version 6.6.1
May 09 02:17:54 ERR [140159294064384]: 2: __forward_SO: couldn't get raw segment ssize a7e4f1f4ff09cf55be3caee38aad60a0 len 16777216
May 09 02:17:54 ERR [140159294064384]: -1: __forward_SO: Could not get raw segment a7e4f1f4ff09cf55be3caee38aad60a0, len 16777216: no such object
May 09 02:17:54 ERR [140159294064384]: 2: __replicate_SOs_of_DO:replication failed so fingerprint a7e4f1f4ff09cf55be3caee38aad60a0 (16777216 bytes)
May 09 02:17:54 ERR [140159294064384]: 2: __process_batch: replication failed batch send failed at DO 5ec160006a8a7722bd5068dc4da99166
May 09 02:17:54 ERR [140159294064384]: 2: CRReplicate: replication failed batch send failed at DO 5ec160006a8a7722bd5068dc4da99166
May 09 02:17:54 ERR [140159294064384]: 2: CRReplicate of batch failed: no such object
May 09 02:17:54 ERR [140159294064384]: 2: forward fingerprints failed: no such object (2)
May 09 02:17:59 INFO [140159294064384]: WSRequestExt: submitting
Thank you in regards.