How to verify connectivity from Netbackup media server to a HPE StoreOnce using Catalyst

Obtain a list of defined storage server

# nbdevquery -listdp -stype hp-StoreOnceCatalyst

Last column in the output is the defined Storage Server. Then run this command from the Netbackup media server :

bpstsinfo -serverinfo -storage_server {storage server name}   -stype hp-StoreOnceCatalyst

Output from command should look like this, any other message indicate connectivity issue:

Server Info:
Server Name: hp-StoreOnceCatalyst: {storage server name}
Supported Stream Formats:
Maximum Connections: 0
Current Connections: 1
Supported Interfaces:
Supported Credentials:

If diskpool is in up state, and the command above works, but backup fails with status 2074, check the OST credentials configured with tpconfig. The Cayalyst debug log in /usr/openv/hpe/ost/logs will contain lines with “OSCLT_ERR_PERMISSION_DENIED”.

Netbackup TIR data


Netbackup initial store TIR (True Image Restore) data in the catalog, but not longer than “Keep true Image restoration (TIR) information” setting in the master server properties. After that period of time TIR data is pruned from the catalog.

If Netbackup need older TIR data than the prune setting, e.g. for a bare metal restore (BMR), TIR data is restored from media. This can be seen as an additional restore job during BMR.

NBU_TIR_example 2

Once thing to note is that both client and master server is the same host.


Netbackup Bare Metal Restore with Ovirt Tools.

Netbackup Bare Metal Restore has a issue with KVM Ovirt tools version (latest version as time for writing).

During backup, the control job will fail with status code 1:

Error bpbrm (pid=3319) BMRERR: Received BMR error: Failed to import Config file. (27)
Error bpbrm (pid=3319) BMRERR: Received BMR error: Insert for BMR configuration information failed. (13)
Error bpbrm (pid=3319) BMRERR: Received BMR error: Failed sending the discovery. (21)
Error bpbrm (pid=3319) BMRERR: Received BMR error: BMR information discovery failed. (35)

On systems where Ovirt Tools has been upgraded, a BMR restore will result in the message : “The restore has stopped due to the following error: Unable to make Registry modification required to boot the system”.

The message appear after all data disk has been restored, just before the system needs rebooting.

NBU Bare metal restore error message

To fix the issue, revert to Ovirt tools – this version is as time of writing the latest stable version.

Netbackup Java GUI certificate

From Netbackup 7.7.x certificates are issues to all connecting Java admin consoles. That is not a bad thing as this tighten up security, however if the systems domain name changes internally e.g. from srv to the stored certificate doesn’t match and you will get a logon error.

To clear these stored certificates simply delete directory


Creating Symantec Recovery disk image (srt) fails using Netbackup Bare Metal Restore option.

Problem: During creation of Netbackup Bare Metal Restore Shared Resource Tree, the step creating the “Creating Symantec Recovery disk image” fails

BMR - creating symantec recovery point disk image

C:\Windows\Logs\DISM\dism.log contains the following lines:

2015-06-25 10:13:11, Error                 DISM   DISM Package Manager: PID=18712 TID=6388 Failed while processing command add-package. – CPackageManagerCLIHandler::ExecuteCmdLine(hr:0x80070005)

2015-06-25 10:13:11, Info                  CBS    Failed to unload offline registry: {bf1a281b-ad7b-4476-ac95-f47682990ce7}C:/Program Files/Veritas/NetBackup/BareMetal/server/data/baseSrd/EN-x86/mount/Users/default/ntuser.dat, the client may still need it open. [HRESULT = 0x80070005 – E_ACCESSDENIED]

Background for the issues can be found McAfee KB76867

Cause: McAfee is interfering with the dism commad which Netbackup uses.

Resoloution: Temporary disable the followint services

  • McAfee Framework Services
  • McAfee MCSheild
  • MacAfee Task manager

Or Install VirusScan Enterprise 8.8 Patch 5.

MSDP convert in 7.6.1

The convert process in Netbackup 7.6.1 is chatty and log thease messages in /var/log/messages:

spad[14271]: ERR [140433449379616]: -1: /usr/openv/pdde/pdcr/bin/stconv –fixformatfile
convert[15198]: ERR: -1: Failed to load storage format version from /msdp/data/.format
convert[28069]: ERR: -1: Failed to load storage format version from /msdp/data/.format
convert[30023]: ERR: -1: Failed to load storage format version from /msdp/data/.format
convert[30933]: ERR: -1: Failed to load storage format version from /msdp/data/.format
convert[30980]: ERR: -1: Failed to load storage format version from /msdp/data/.format
convert[30989]: ERR: -1: Failed to load storage format version from /msdp/data/.format
convert[31002]: ERR: -1: Failed to load storage format version from /msdp/data/.format
spad[16322]: ERR [139805757482784]: -1: /usr/openv/pdde/pdcr/bin/stconv –fixformatfile
spoold[16418]: ERR [139758022735648]: -1: /usr/openv/pdde/pdcr/bin/stconv –fixformatfile

The messages are information only and are not errors even throut they look serious. The log for the conversion process can be found in /{MSDP_PATH}/log/convert

If the conversion is successful, you will find the following messages in the log file found in /{MSDP_PATH}/log/convert/

[INF] *** Congratulations! The conversion has finished successfully!

bplist – a handy tool for the Netbackup admin

The bplist is a handy tool for the Netbackup admin to list files and folders , however it’s pretty picky with it’s arguments. The cause is bplist assumptions.

  • If no path is specified current path is presumed
  • if no start time is specified , bplist searches 12 hours back.
  • if no type (-t) is specified standard is assumed (type 0).

Always  remember to :

Specify a path, / is a good start

Specify a start time unless you know the backup is younger than 12 hour.

Always specify the -R option.

Specify a type (-t) unless you are listing UNIX type backup.

For Windows hosts.

List files from a Windows client use:

# bplist -B -C  -l -R -t 13 -s 06/05/2010 /

YES, the / as path actual work. You can substitute the / with a drive letter, but you must use drive letter in upper case.

# bplist -B -C  -t 13 -l -R  -s 06/07/2010 /C/

# bplist -B -C  -t 13 -l -R  -s 06/07/2010 /c/ WRONG !!

For Unix host

Since UNIX system uses type 0 (Standard) you can omit the -t option.

# bplist -B -C  -l -R  -s 06/05/2010 /

Listing a Oracle backup, Oracle is type 4

# bplist -B -C -R -l -t 4 -s 01/01/2015 /

For NDMP host

bplist will also work with NDMP backup’s

# bplist -B -C -l -R -t 19 /

 Netbackup types:

As a service, a list of the Netbackup types (Source: Symantec Tech note TECH27299). Use the Netbackup types jointly with bplist -t option.

0	Standard
1	Proxy	NetBackup internal setting
2	Non-Standard	NetBackup internal setting
3	Apollo-wbak	NetBackup DataCenter only
4	Oracle
5	Any policy type
6	Informix-On-BAR
7	Sybase
8	MS-Sharepoint	NetBackup Server /Enterprise Server only
9	MS-Windows	Not active, use MS-Windows-NT (13) for Windows 2000/NT/XP/2003 clients.
10	NetWare
11	DataTools-SQL-BackTrack
12	Auspex-FastBackup
13	MS-Windows-NT
14	OS/2
15	MS-SQL-Server
16	MS-Exchange-Server
17	SAP	NetBackup Server /Enterprise Server only
18	DB2	NetBackup Server /Enterprise Server only
20	FlashBackup	NetBackup Server /Enterprise Server only
21	Split-Mirror	NetBackup Server /Enterprise Server only
22	AFS	NetBackup Server /Enterprise Server only
23	DFS	Not an active policy type.
24	DataStore
25	Lotus-Notes
26	NCR-Teradata	No longer supported, contact NCR to see if they provide an extension.
27	OpenVMS
28	MPE/iX
29	FlashBackup-Windows	NetBackup Server /Enterprise Server only
30	Vault
31	BE-MS-SQL-Server
32	BE-MS-Exchange-Server
33	Macintosh	Not active, use Standard (0) for Macintosh OSX clients.
34	Disk Staging	NetBackup Server /Enterprise Server only
35	NBU-Catalog	NetBackup 6.0 only

After BMR restore login display “Desktop refer to a location that is unavaiable”


After Netbackup BMR restore has been completed and firt login is attempted the follwing message is begin displayed :BMR_destop_error

Restore using the classic “system overwrite” method does not show the error message.

Exclude list contained the follwing files:


Remove the files on the exclude list. A new backup is required to have a “working” BMR backup.

Netbackup Admins test tools

As a Netbackup admin, solving other problems may not be a stranger to you. Most seen is very bad backup performance. This article describe the test tool i use in my day to day work.


Netbackup is able to generate huge amount of random data generated in memory to performance test underling hardware via a file directive. It’s a quite a hidden gem.  The GEN_DATA file directive work on UNIX and Linux- Sorry no Windows.

See tech note:
Documentation: How to use the GEN_DATA file list directives with NetBackup for UNIX/Linux Clients for Performance Tuning


A utility created by Symantec (previous Veritas). It’s splendid tool for finding bad disk performance, read or write. Works best on VXFS file system (obvious). Vxbench has diffrent workload built in (seq read/write – random read/write) and you can specify block size as well. I always check new disk storage unit with vxbench before putting them i production. Vxbench is available for Solaris, AI, HP-UX and Linux – The package is called VRTSspt and can be downloaded from

Symantec site.


vxbench -w write  -i iosize=128,iocount=262144 /diskstu4/dsu/testfile1


total:  111.531 sec  300852.32 KB/s  cpu: 48.65 sys 0.04 user


A freeware java based network performance utility. Can be run on any OS with a Java VM. Can be obtained from Netcordia.  In order for reliable figures the amount of data send/received must be tweaked.

On the reciver side: java ttcp -r -l 65536 -n 16384

And on the reciver side : java ttcp -l 65536 -n 16384

Output look like this:

Transmit: buflen= 65536  nbuf= 16384 port= 5001
Transmit connection:  Socket[addr=lena/,port=5001,localport=59154].
Transmit: 1073741824 bytes in 10913 milli-seconds = 98391.08 KB/sec (787128.6 Kbps).


Whenever a firewall closes inn on you, tcpdump is you’re find. You don’t need to understand all the stuff, it’s reasonable easy to see connections in and out.

Here is a list of my most often used tcpdump commands. I always use the following arguments

-i  To specify what interface to listen to eg. eth6

-f Causes tcpdump to print internet addresses in numerical notation

-n Prevent service port to get translated into names (prints 13720 insted of bpcd).

Listen for traffic for a entire network

# tcpdump -n -f -i eth6  net

Listen for traffic for just one host

# tcpdump -n -f -i eth6 host

Or just one service  port.

# tcpdump -n -f -i eth2 port ssh

# You can also trace traffic for two host on a IP only layer.

tcpdump -n -f -i eth1 ip host or

Listen for traffic but don’t clutter the picture with your’e own SSH traffic

# tcpdump -n -f -i eth5 ip and not port 22

Using Netbackup bpbkar as test tool

You can run bpbkar (the process responsible for reading from disk) by hand to see how performance is when network/tape drive layer is cut off. When issuing bpbkar by hand data is read from disk and thrown in the bit bucket. This will enable the admin to find out weather the problem is on the client side or server side.

# Windows

d:\VERITAS\NetBackup\bin\bpbkar32.exe -nocont  D:\  1> nul 2> nul

# Unix

/usr/openv/netbackup/bin/bpbkar  -nocont -nofileinfo -nokeepalives /var  > /dev/null 2> /tmp/file.out

Make sure you have created the bpbkar debug directory in  C:\Program Files\VERITAS\NetBackup\logs before starting. The command is return immediately, but the process will be visible in task manager, and the debug log will grow in size as well.

if bpbkar run by hand takes the same amount of time as a real backup, you know the problem is on the client and know where to chase the next bottleneck.