We are seeing delays in VMware backup jobs finishing after upgrading to Netbackup 7.7.1. We were on version 7.6.0.2 prior to the upgrade. Our master server is Solaris 10. We have 3 MSDP servers running Linux 6.7. We have two VMware backup host servers. One is running Windows Server 2012 and the other is running Linux 6.7. The VMware backups write to the MSDP servers. Any one of the three MSDP server can write to each others disk. The delay are causing other backups to fail, since they aren't able to run during their back windows. Below is the jobs details from one of the slow jobs.
12/03/2015 08:39:37 - Info nbjm (pid=12665) starting backup job (jobid=4065919) for client sql53092vm, policy Omaha_VMware_SQL, schedule Incremental
12/03/2015 08:39:38 - estimated 38973558 kbytes needed
12/03/2015 08:39:38 - Info nbjm (pid=12665) started backup (backupid=sql53092vm_1449153577) job for client sql53092vm, policy Omaha_VMware_SQL, schedule Incremental on storage unit XP53TAPE008 using backup host 53tape001.botw.ad.bankofthewest.com
12/03/2015 08:39:38 - started process bpbrm (pid=40696)
12/03/2015 08:39:40 - Info bpbrm (pid=40696) sql53092vm is the host to backup data from
12/03/2015 08:39:40 - Info bpbrm (pid=40696) reading file list for client
12/03/2015 08:39:40 - Info bpbrm (pid=40696) accelerator enabled
12/03/2015 08:39:49 - connecting
12/03/2015 08:39:49 - connected; connect time: 0:00:00
12/03/2015 08:39:49 - Info bpbrm (pid=40696) starting bpbkar on client
12/03/2015 08:39:50 - Info bptm (pid=40986) start
12/03/2015 08:39:50 - Info bpbkar (pid=2844) Backup started
12/03/2015 08:39:50 - Info bpbrm (pid=40696) bptm pid: 40986
12/03/2015 08:39:50 - Info bpbkar (pid=2844) accelerator enabled backup, archive bit processing:<disabled>
12/03/2015 08:39:50 - Info bpbkar (pid=2844) INF - Backing up vCenter server 53apps178vm, ESX host v53host030.bankofthewest.com, BIOS UUID 42045880-2f95-1fee-fa56-4f3c65d4164a, Instance UUID 50049e93-61b0-0ab2-3f5b-b141cb28f6a1, Display Name sql53092vm, Hostname SQL53092VM.botw.ad.bankofthewest.com
12/03/2015 08:39:51 - Info bptm (pid=40986) using 262144 data buffer size
12/03/2015 08:39:51 - Info bptm (pid=40986) using 30 data buffers
12/03/2015 08:39:55 - Info bptm (pid=40986) start backup
12/03/2015 08:40:25 - Info bptm (pid=40986) backup child process is pid 41677
12/03/2015 08:40:25 - begin writing
12/03/2015 08:40:34 - Info bpbkar (pid=2844) INF - Transport Type = san
12/03/2015 08:47:19 - Info bptm (pid=40986) waited for full buffer 2 times, delayed 148 times
12/03/2015 08:47:21 - Info bpbkar (pid=2844) accelerator sent 717710848 bytes out of 33181077504 bytes to server, optimization 97.8%
12/03/2015 10:10:23 - Info bptm (pid=40986) EXITING with status 0 <----------
12/03/2015 10:10:23 - Info xp53tape009 (pid=40986) StorageServer=PureDisk:xp53tape008; Report=PDDO Stats for (xp53tape008): scanned: 32406122 KB, CR sent: 142286 KB, CR sent over FC: 0 KB, dedup: 99.6%, cache disabled
12/03/2015 10:10:29 - Info bpbrm (pid=40696) validating image for client sql53092vm
12/03/2015 10:10:30 - Info bpbkar (pid=2844) done. status: 0: the requested operation was successfully completed
12/03/2015 10:10:30 - end writing; write time: 1:30:05
the requested operation was successfully completed (0)
We are seeing the below error in the bptm logs.
09:09:55.303 [26144] <16> 4065881:bptm:26144:xp53tape010: [ERROR] PDSTS: fbu_align_extents: (6130) stream_offset(48750986240) != extent.stream_offset=(48750957568)
09:09:56.701 [26144] <16> 4065881:bptm:26144:xp53tape010: [ERROR] PDSTS: fbu_align_extents: (6134) stream_offset(48755704832) != extent.stream_offset=(48755033088)
09:09:56.701 [26144] <16> 4065881:bptm:26144:xp53tape010: [ERROR] PDSTS: fbu_align_extents: (6134) stream_offset(48755712512) != extent.stream_offset=(48755033088)
09:09:56.717 [26144] <16> 4065881:bptm:26144:xp53tape010: [ERROR] PDSTS: fbu_align_extents: (6134) stream_offset(48756687872) != extent.stream_offset=(48755033088)
Has anyone experience the above issue?