Creating Symantec Recovery disk image (srt) fails using Netbackup Bare Metal Restore option.

Problem: During creation of Netbackup Bare Metal Restore Shared Resource Tree, the step creating the “Creating Symantec Recovery disk image” fails

BMR - creating symantec recovery point disk image

C:\Windows\Logs\DISM\dism.log contains the following lines:

2015-06-25 10:13:11, Error                 DISM   DISM Package Manager: PID=18712 TID=6388 Failed while processing command add-package. – CPackageManagerCLIHandler::ExecuteCmdLine(hr:0x80070005)

2015-06-25 10:13:11, Info                  CBS    Failed to unload offline registry: {bf1a281b-ad7b-4476-ac95-f47682990ce7}C:/Program Files/Veritas/NetBackup/BareMetal/server/data/baseSrd/EN-x86/mount/Users/default/ntuser.dat, the client may still need it open. [HRESULT = 0x80070005 – E_ACCESSDENIED]

Background for the issues can be found McAfee KB76867

Cause: McAfee is interfering with the dism commad which Netbackup uses.

Resoloution: Temporary disable the followint services

  • McAfee Framework Services
  • McAfee MCSheild
  • MacAfee Task manager

Or Install VirusScan Enterprise 8.8 Patch 5.

MSDP convert in 7.6.1

The convert process in Netbackup 7.6.1 is chatty and log thease messages in /var/log/messages:

spad[14271]: ERR [140433449379616]: -1: /usr/openv/pdde/pdcr/bin/stconv –fixformatfile
convert[15198]: ERR: -1: Failed to load storage format version from /msdp/data/.format
convert[28069]: ERR: -1: Failed to load storage format version from /msdp/data/.format
convert[30023]: ERR: -1: Failed to load storage format version from /msdp/data/.format
convert[30933]: ERR: -1: Failed to load storage format version from /msdp/data/.format
convert[30980]: ERR: -1: Failed to load storage format version from /msdp/data/.format
convert[30989]: ERR: -1: Failed to load storage format version from /msdp/data/.format
convert[31002]: ERR: -1: Failed to load storage format version from /msdp/data/.format
spad[16322]: ERR [139805757482784]: -1: /usr/openv/pdde/pdcr/bin/stconv –fixformatfile
spoold[16418]: ERR [139758022735648]: -1: /usr/openv/pdde/pdcr/bin/stconv –fixformatfile

The messages are information only and are not errors even throut they look serious. The log for the conversion process can be found in /{MSDP_PATH}/log/convert

If the conversion is successful, you will find the following messages in the log file found in /{MSDP_PATH}/log/convert/

[INF] *** Congratulations! The conversion has finished successfully!

bplist – a handy tool for the Netbackup admin

The bplist is a handy tool for the Netbackup admin to list files and folders , however it’s pretty picky with it’s arguments. The cause is bplist assumptions.

  • If no path is specified current path is presumed
  • if no start time is specified , bplist searches 12 hours back.
  • if no type (-t) is specified standard is assumed (type 0).

Always  remember to :

Specify a path, / is a good start

Specify a start time unless you know the backup is younger than 12 hour.

Always specify the -R option.

Specify a type (-t) unless you are listing UNIX type backup.

For Windows hosts.

List files from a Windows client use:

# bplist -B -C  host.acme.com  -l -R -t 13 -s 06/05/2010 /

YES, the / as path actual work. You can substitute the / with a drive letter, but you must use drive letter in upper case.

# bplist -B -C  host.acme.com  -t 13 -l -R  -s 06/07/2010 /C/

# bplist -B -C  host.acme.com  -t 13 -l -R  -s 06/07/2010 /c/ WRONG !!

For Unix host

Since UNIX system uses type 0 (Standard) you can omit the -t option.

# bplist -B -C  host.acme.com  -l -R  -s 06/05/2010 /

Listing a Oracle backup, Oracle is type 4

# bplist -B -C ora.acme.com -R -l -t 4 -s 01/01/2015 /

For NDMP host

bplist will also work with NDMP backup’s

# bplist -B -C nasdkba001.acme.com -l -R -t 19 /

 Netbackup types:

As a service, a list of the Netbackup types (Source: Symantec Tech note TECH27299). Use the Netbackup types jointly with bplist -t option.

0	Standard
1	Proxy	NetBackup internal setting
2	Non-Standard	NetBackup internal setting
3	Apollo-wbak	NetBackup DataCenter only
4	Oracle
5	Any policy type
6	Informix-On-BAR
7	Sybase
8	MS-Sharepoint	NetBackup Server /Enterprise Server only
9	MS-Windows	Not active, use MS-Windows-NT (13) for Windows 2000/NT/XP/2003 clients.
10	NetWare
11	DataTools-SQL-BackTrack
12	Auspex-FastBackup
13	MS-Windows-NT
14	OS/2
15	MS-SQL-Server
16	MS-Exchange-Server
17	SAP	NetBackup Server /Enterprise Server only
18	DB2	NetBackup Server /Enterprise Server only
19	NDMP
20	FlashBackup	NetBackup Server /Enterprise Server only
21	Split-Mirror	NetBackup Server /Enterprise Server only
22	AFS	NetBackup Server /Enterprise Server only
23	DFS	Not an active policy type.
24	DataStore
25	Lotus-Notes
26	NCR-Teradata	No longer supported, contact NCR to see if they provide an extension.
27	OpenVMS
28	MPE/iX
29	FlashBackup-Windows	NetBackup Server /Enterprise Server only
30	Vault
31	BE-MS-SQL-Server
32	BE-MS-Exchange-Server
33	Macintosh	Not active, use Standard (0) for Macintosh OSX clients.
34	Disk Staging	NetBackup Server /Enterprise Server only
35	NBU-Catalog	NetBackup 6.0 only

After BMR restore login display “Desktop refer to a location that is unavaiable”

Description:

After Netbackup BMR restore has been completed and firt login is attempted the follwing message is begin displayed :BMR_destop_error

Restore using the classic “system overwrite” method does not show the error message.

Cause:
Exclude list contained the follwing files:

NTUSER.DAT*
UsrClass.dat*

Remove the files on the exclude list. A new backup is required to have a “working” BMR backup.

Netbackup Admins test tools

As a Netbackup admin, solving other problems may not be a stranger to you. Most seen is very bad backup performance. This article describe the test tool i use in my day to day work.

GEN_DATA

Netbackup is able to generate huge amount of random data generated in memory to performance test underling hardware via a file directive. It’s a quite a hidden gem.  The GEN_DATA file directive work on UNIX and Linux- Sorry no Windows.

See tech note:
Documentation: How to use the GEN_DATA file list directives with NetBackup for UNIX/Linux Clients for Performance Tuning

vxbench

A utility created by Symantec (previous Veritas). It’s splendid tool for finding bad disk performance, read or write. Works best on VXFS file system (obvious). Vxbench has diffrent workload built in (seq read/write – random read/write) and you can specify block size as well. I always check new disk storage unit with vxbench before putting them i production. Vxbench is available for Solaris, AI, HP-UX and Linux – The package is called VRTSspt and can be downloaded from

Symantec site.

Examples:

vxbench -w write  -i iosize=128,iocount=262144 /diskstu4/dsu/testfile1

output:

total:  111.531 sec  300852.32 KB/s  cpu: 48.65 sys 0.04 user

ttcp

A freeware java based network performance utility. Can be run on any OS with a Java VM. Can be obtained from Netcordia.  In order for reliable figures the amount of data send/received must be tweaked.

On the reciver side: java ttcp -r -l 65536 -n 16384

And on the reciver side : java ttcp -l 65536 -n 16384

Output look like this:

Transmit: buflen= 65536  nbuf= 16384 port= 5001
Transmit connection:  Socket[addr=lena/10.1.22.134,port=5001,localport=59154].
Transmit: 1073741824 bytes in 10913 milli-seconds = 98391.08 KB/sec (787128.6 Kbps).

tcpdump

Whenever a firewall closes inn on you, tcpdump is you’re find. You don’t need to understand all the stuff, it’s reasonable easy to see connections in and out.

Here is a list of my most often used tcpdump commands. I always use the following arguments

-i  To specify what interface to listen to eg. eth6

-f Causes tcpdump to print internet addresses in numerical notation

-n Prevent service port to get translated into names (prints 13720 insted of bpcd).

Listen for traffic for a entire network

# tcpdump -n -f -i eth6  net  10.10.10.0/24

Listen for traffic for just one host

# tcpdump -n -f -i eth6 host 10.1.1.1

Or just one service  port.

# tcpdump -n -f -i eth2 port ssh

# You can also trace traffic for two host on a IP only layer.

tcpdump -n -f -i eth1 ip host 10.224.13.1 or 10.224.13.2

Listen for traffic but don’t clutter the picture with your’e own SSH traffic

# tcpdump -n -f -i eth5 ip and not port 22

Using Netbackup bpbkar as test tool

You can run bpbkar (the process responsible for reading from disk) by hand to see how performance is when network/tape drive layer is cut off. When issuing bpbkar by hand data is read from disk and thrown in the bit bucket. This will enable the admin to find out weather the problem is on the client side or server side.

# Windows

d:\VERITAS\NetBackup\bin\bpbkar32.exe -nocont  D:\  1> nul 2> nul

# Unix

/usr/openv/netbackup/bin/bpbkar  -nocont -nofileinfo -nokeepalives /var  > /dev/null 2> /tmp/file.out

Make sure you have created the bpbkar debug directory in  C:\Program Files\VERITAS\NetBackup\logs before starting. The command is return immediately, but the process will be visible in task manager, and the debug log will grow in size as well.

if bpbkar run by hand takes the same amount of time as a real backup, you know the problem is on the client and know where to chase the next bottleneck.

Deleting an Application Rollback shadow image

The normal vssadmin command can’t delete shadow copy’s of type “ApplicationRollback”. To delete the shadow image get the Volume Shadow Copy Service SDK from Microsoft using Windows 2003. Windows 2008 and newer has the command built-in .

The vshadow command can do very powerful stuff – indeed a very interesting command.

N:\>vssadmin list shadows
vssadmin 1.1 – Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Contents of shadow copy set ID: {047a3cb0-04fe-4298-bfe9-0124ec79410b}
Contained 1 shadow copies at creation time: 8/27/2008 2:51:14 PM
Shadow Copy ID: {0d456a73-e8f4-4695-b0dd-59e55c190753}
Original Volume: (D:)\\?\Volume{8a9334e5-c416-11dc-95ba-806e6f6e6963}\
Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1
Originating Machine: appdkba795.acme.com
Service Machine: appdkba795.acme.com
Provider: ‘Microsoft Software Shadow Copy provider 1.0’
Type: ApplicationRollback
Attributes: Persistent, No auto release, Differential

Run vshadow command and delete all snaps – text below for reference.

D:\vss_tools>vshadow -da

VSHADOW.EXE 2.2 – Volume Shadow Copy sample client
Copyright (C) 2005 Microsoft Corporation. All rights reserved.

(Option: Delete all shadow copies)
This will delete all shadow copies in the system. Are you sure? [Y/N] y

– Setting the VSS context to: 0xffffffff
– Deleting shadow copy {0d456a73-e8f4-4695-b0dd-59e55c190753} on \\?\Volume{8a9334e5-c416-11dc-95ba-806e6f6e6963}\ from provider {b5946137-7b9f-4925-af80-51abd6
0b20d5} [0x00020009]…

Alternate file location using install_client_files

I bet we all have tried this, you need to install a Netbackup client, just to discover that there is no place in /tmp. What now ??. Well, there is hope. With a little script hack it possible to let the “install_client_files” script install files in another directory than /tmp.
  • Identify client you want to push
  • Go to /usr/openv/netbackup/client/{HW}/{OS}/ for a SUSE2.6 that would be /usr/openv/netbackup/client/Linux/Linux/SuSE2.6
  • Edit file install_client
  • Find variable DEST_DIR=/tmp/bp.${pid} on line 891 (as of 6.5.4), and change the destination folder e.g. DEST_DIR=/var/bp.${pid}
  • Save file.
  • Push agent
  • Revert change back

How to use a LUN larger than 2TB with Veritas Volume Manager (VXVM)

If you want to use LUN’s larger than 2TB with VXVM on a LINUX host you need to use parted instead of fdiske to create partitions. fdisk is limited to 2TB partitions. The CDS feature is a no-go as well

We assume disk is sdu. Before VxVM can use the disk, it must have a GPT label

# parted /dev/sdu mklabel gpt (the change is instant !!!!)

The default format for vxdisksetup is CDS format, we can’t use so we revert to the old format:

# /etc/vx/bin/vxdisksetup -i sdu format=simple

From here it’s business as usual 😀

Some links to Symantec support pages (will open a new browser window):

Initializing a LUN that is greater than 2TB creates usable space equal to, or less than 2TB with Volume Manager 5.0 on Red Hat Enterprise Linux 4 Update 3 or higher.

Unable to process. Duplication session in progress. return value = [134]

From time to time the the nbstlutil command may return the message:

Unable to process. Duplication session in progress. return value = [134]

When activating or inactivating Storage Lifecycle Policies (slp). Since the nbstlutil command always return status code 0 this is quite a problem if the command is part of a script. However specifying the -wait argument causes nbstlutil command to retry the command until duplication session is no longer in progress.

Please note the -wait argument is picky about where it’s specified.

This command will retry/wait

# nbstlutil -wait inactive -lifecycle {slp_name}

But this doesn’t

# nbstlutil inactive -lifecycle {slp_name} -wait

Are your’e PCIe card running at full speed ?

Are youre PCIe card running at full speed?

Its pretty simple to check if you are running Linux. Just at a command prompt type

# lspci -vv

The command will return a list of PCI-Express devices along with the supported and actual speed/width.

Any difference in the supported and actual speed (or width) should trigger consideration whether or not to reconfigure the HBA layout.

This HBA run isn’t properly configured, the current PCIe slot has a width of 4 but the card actual REQUIRE a x8 slot :

4c:00.0 Ethernet controller: NetXen 10G Ethernet PCI Express (rev 25)
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 0
Link: Latency L0s <64ns, L1 <1us
Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x4

Sample of a HBA running as intended :

4f:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express
Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s, Port 0
Link: Latency L0s <4us, L1 unlimited
Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch+
Link: Speed 2.5Gb/s, Width x4