Cluster Mode mhost troubleshooting

If you need any help, feel free to ask.

1. go to the systemshell
set diag
systemshell -node cl1-01

2. unmount mroot
cd /etc
./netapp_mroot_unmount
logout

3. run cluster show a couple of times and see that health is false
cluster show

4. run cluster ring show to see that M-host is offline
cluster ring show
Node UnitName Epoch DB Epoch DB Trnxs Master Online
——— ——– ——– ——– ——– ——— ———
cl1-01 mgmt 6 6 699 cl1-01 master
cl1-01 vldb 7 7 84 cl1-01 master
cl1-01 vifmgr 9 9 20 cl1-01 master
cl1-01 bcomd 7 7 22 cl1-01 master
cl1-02 mgmt 0 6 692 – offline
cl1-02 vldb 7 7 84 cl1-01 secondary
cl1-02 vifmgr 9 9 20 cl1-01 secondary
cl1-02 bcomd 7 7 22 cl1-01 secondary

5. try to create a volume and see that the status of the aggregate
cannot be determined if you pick the aggregate from the broken M-host.

6. now vldb will also be offline.

5. remount mroot by starting mgwd from the systemshell
set diag
systemshell -node cl1-01
/sbin/mgwd -z &

7. when you run cluster ring show it should show vldb offline
cl1::*> cluster ring show
Node UnitName Epoch DB Epoch DB Trnxs Master Online
——— ——– ——– ——– ——– ——— ———
cl1-01 mgmt 6 6 738 cl1-01 master
cl1-01 vldb 7 7 87 cl1-01 master
cl1-01 vifmgr 9 9 24 cl1-01 master
cl1-01 bcomd 7 7 22 cl1-01 master
cl1-02 mgmt 6 6 738 cl1-01 secondary
cl1-02 vldb 0 7 84 – offline
cl1-02 vifmgr 0 9 20 – offline
cl1-02 bcomd 7 7 22 cl1-01 secondary

Watch vifmgr has gone bad as well.

8. start vldb by running spmctl -s -h vldb
or run /sbin/vldb
in this case, do the same for vifmgr.

Please leave your comments, that would be helpful.

cluster netapp convert snapmirror to snapvault

Steps:

• Break the data protection mirror relationship by using the snapmirror break command.
• The relationship is broken and the disaster protection volume becomes a read-write volume.
• Delete the existing data protection mirror relationship, if one exists, by using the snapmirror delete command.
• Remove the relationship information from the source SVM by using the snapmirror release command.
• (This also deletes the Data ONTAP created Snapshot copies from the source volume.)
• Create a SnapVault relationship between the primary volume and the read-write volume by using the snapmirror create command with the -type XDP parameter.
• Convert the destination volume from a read-write volume to a SnapVault volume and establish the SnapVault relationship by using the snapmirror resync command.
• (Warning: all data newer than the snapmirror.xxxxxx snapshot copy will be lost and also: the snapvault destination should
• not be the source of another snapvault relationship)

Please leave your comments.. that would be helpful.

systemshell Commands for NetApp Cluster mode

ONLY USE IF YOU KNOW WHAT YOU ARE DOING

spmctl, rdb_dump, kenv

cl1-01% spmctlList of managed processes. (Error=0)
Exec=/usr/sbin/ucoreman –log;Handle=069264cf-377e-4f06-84c2-05fc01b4ee5f;Pid=906;State=Running
Exec=/usr/bin/raid_lm;Handle=0a3fbbf9-da62-4104-8e57-846977ad6a3c;Pid=1922;State=Running
Exec=/sbin/schmd;Handle=1120522e-9e6f-4b75-b67d-e7a8684ec164;Pid=1658;State=Running
Exec=/sbin/vifmgr -n;Handle=17bde8b6-7fbb-4157-9e70-5365ac1e6e58;Pid=1797;State=Running
Exec=/usr/sbin/mhostexecd -D;Handle=2d16144e-d733-49bb-a94b-cb40de81f205;Pid=1844;State=Running
Exec=/sbin/nchmd;Handle=3909a2b9-37c7-4009-bf7b-914892d4af8d;Pid=1662;State=Running
Exec=/usr/sbin/time_state -l;Handle=51329b28-5a89-4e7a-8786-ae17ea355089;Pid=902;State=Running
Exec=/sbin/notifyd -n;Handle=56548532-c334-4633-8cd8-77ef97682d3d;Pid=829;State=Running
Exec=/sbin/bcomd;Handle=7feb285e-aba3-44fb-8f3b-daadb7e702b9;Pid=1801;State=Running
Exec=/usr/sbin/httpdmgr;Handle=86fadb47-8fcd-47c4-bf16-98dc7d10e416;Pid=1621;State=Running
Exec=/sbin/coresegd -m;Handle=99e1bb86-fce4-413e-b66b-21348215d253;Pid=1683;State=Running
Exec=/sbin/shmd;Handle=a534ba91-c0d1-44a6-b061-e280ba8b62e5;Pid=1654;State=Running
Exec=/sbin/mntsvc -n;Handle=a71c0340-e8e1-4bab-a7f6-25ae97d4da5b;Pid=1816;State=Running
Exec=/sbin/ndmpd;Handle=c0c9ac97-8eb1-4d8b-a1c9-15546c4b464d;Pid=1645;State=Running
Exec=/sbin/cmd;Handle=c2b7cc49-d650-4cfe-b167-95c946ca0abb;Pid=1803;State=Running
Exec=/sbin/mgwd -z;Handle=c6b932ea-acc2-46f4-a8e7-83774fb03d99;Pid=939;State=Running
Exec=/usr/sbin/named -f -c /tmp/named.conf -S 1024;Handle=ccb0609e-d26e-40a9-9e40-15569206ae3a;Pid=1789;State=Running
Exec=/sbin/vldb -n;Handle=cd4f8a3c-126a-4895-bfb9-5b67fdbea320;Pid=1799;State=Running
Exec=/sbin/mdnsd -z;Handle=cd9fc4fa-4f45-4d09-9f02-03a193229618;Pid=1672;State=Running
Exec=/usr/bin/sktlogd -m;Handle=df663bec-40ed-4278-9d54-220a52d905b9;Pid=1813;State=Running
Exec=/sbin/secd;Handle=fe60b0f2-1c9b-4e32-bfae-640fc9c36310;Pid=1778;State=Running

cl1-01% rdb_dump
Local time Wed Dec 12 15:58:47 2012
RDB Unit “Management” (id 1) on host “cl1-01″ (site 1000)
At Wed Dec 12 15:58:47 2012.
App Version: <1,1>, RDB Version: <2,0>, DBSet Version: <2,220>

Online Status:
Local 1000 is Master (epoch: 2, master: 1000)
1. id 1000, state: online *** Master (local)

RDB Unit “VifMgr” (id 2) on host “cl1-01″ (site 1000)
At Wed Dec 12 15:58:47 2012.
App Version: <1,1>, RDB Version: <2,0>, DBSet Version: <2,20>

Online Status:
Local 1000 is Master (epoch: 2, master: 1000)
1. id 1000, state: online *** Master (local)

RDB Unit “VLDB” (id 0) on host “cl1-01″ (site 1000)
At Wed Dec 12 15:58:47 2012.
App Version: <1,1>, RDB Version: <2,0>, DBSet Version: <2,69>

Online Status:
Local 1000 is Master (epoch: 2, master: 1000)
1. id 1000, state: online *** Master (local)

RDB Unit “Bcom” (id 5) on host “cl1-01″ (site 1000)
At Wed Dec 12 15:58:47 2012.
App Version: <1,1>, RDB Version: <2,0>, DBSet Version: <2,13>

Online Status:
Local 1000 is Master (epoch: 2, master: 1000)
1. id 1000, state: online *** Master (local)

————-

cl1-01% kenvAUTOBOOT=”true”
BIOS_VERSION=”245″
BOARDNAME=”VMWARE”
BOOTED_FROM=”PRIMARY”
BOOT_FILE=”x86_64/freebsd/image1/kernel”
LINES=”24″
LOADER_VERSION=”1.0″
MOBO_REV=”ZZ”
MOBO_SERIAL_NUM=”999999″
NETAPP_BACKUP_KERNEL_URL=”x86_64/freebsd/image2/kernel”
NETAPP_PRIMARY_KERNEL_URL=”x86_64/freebsd/image1/kernel”
SYS_MODEL=”SIMBOX”
SYS_REV=”ZZ”
SYS_SERIAL_NUM=”4061490-31-8″
acpi_load=”YES”
bootarg.bsdportname=”e0c”
bootarg.dblade.wafl_use_delete_log=”false”
bootarg.init.boot_clustered=”true”
bootarg.init.cfdevice=”/dev/ad0s2″
bootarg.init.clearvarfsnvram=”false”
bootarg.init.dhcp.disable=”true”
bootarg.mgwd.autoconf.disable=”true”
bootarg.mgwd.scsi_blade_uuid=”8231ca39-4371-11e2-b1a0-31d956941ea5″
bootarg.new_varfs=”false”
bootarg.nvram.sysid=”4061490318″
bootarg.setup.auto.internal=”true”
bootarg.sim=”true”
bootarg.sim.vardev=”/dev/ad1s1″
bootarg.sim.vdev=”/dev/ad3″
bootarg.sim.vdevinit=”false”
bootarg.srm.disk.san_reservations=”true”
bootarg.srm.nvram.setup_sim=”true”
bootarg.srm.nvram.vnvram=”true”
bootarg.vm=”true”
bootarg.vm.data_diskmodel=”vha”
bootarg.vm.no_poweroff_on_halt=”true”
bootarg.vm.rapidsavecore=”false”
bootarg.vm.run_vmtools=”true”
bootarg.vm.sim=”true”
bootarg.vm.sim.vdev=”/dev/ad3″
bootarg.vm.sim.vdevinit=”false”
bootarg.vm.sys_diskmodel=”standard”
bootarg.vm.vardev=”/dev/ad1s1″
bootarg.vm.varfs=”false”
bootarg.vnvram.size=”32″
bootfile=”kernel”
comconsole_speed=”9600″
console=”vidconsole”
currdev=”disk1s2″
fc-ports-valid=”true”
fmmbx-lkg-0=”ED993E2811E2439134129CAD123456780000008C000000004154454E20205050312D44564D3030305A462D423032352D353838313030353200000000000000004154454E20205050312D44564D3030305A462D423032352D35383831313035320000000000000000″
fmmbx-lkg-0b=”ED993E2811E2439134129CAD123456780000008C000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000″
interpret=”OK”
kernel=”kernel”
kernel_options=”"
kernelname=”x86_64/freebsd/image1/kernel”
last-OS-booted-raid-ver=”11″
last-OS-booted-ver=”8.1.1X34″
last-OS-booted-wafl-ver=”22331″
loader_conf_files=”/boot/device.hints /boot/loader.conf /boot/loader.conf.local”
mac_ifoff=”NO”
module_path=”/boot/modules”
ntap.init.cfdevice=”/dev/ad0s2″
nvram_discard=”false”
nvram_emulation=”true”
prompt=”VLOADER>”
setvsimenv=”true”
smbios.bios.reldate=”07/02/2012″
smbios.bios.vendor=”Phoenix Technologies LTD”
smbios.bios.version=”6.00″
smbios.chassis.maker=”No Enclosure”
smbios.chassis.serial=”None”
smbios.chassis.tag=”No Asset Tag”
smbios.chassis.version=”N/A”
smbios.planar.maker=”Intel Corporation”
smbios.planar.product=”440BX Desktop Reference Platform”
smbios.planar.serial=”None”
smbios.planar.version=”None”
smbios.socket.enabled=”2″
smbios.socket.populated=”2″
smbios.system.maker=”VMware, Inc.”
smbios.system.product=”VMware Virtual Platform”
smbios.system.serial=”VMware-56 4d fb 8e 66 a8 07 a9-62 a9 a6 2e 7f 51 41 45″
smbios.system.uuid=”564dfb8e-66a8-07a9-62a9-a62e7f514145″
smbios.system.version=”None”
sysvar.system_id=”0″
wafl-disable-mbuf-backed-buffers?=”TRUE”
hint.vga.0.at=”isa”
hint.sc.0.at=”isa”
hint.sc.0.flags=”0×100″
hint.atkbdc.0.at=”isa”
hint.atkbdc.0.port=”0×060″
hint.atkbd.0.at=”atkbdc”
hint.atkbd.0.irq=”1″
hint.uart.0.at=”isa”
hint.uart.0.port=”0x3F8″
hint.uart.0.flags=”0×40″
hint.uart.0.irq=”4″
hint.uart.1.at=”isa”
hint.uart.1.port=”0x2F8″
hint.uart.1.flags=”0×20″
hint.uart.1.irq=”3″
bootarg.init.booterr=”0″
bootarg.init.cf_mounted=”true”
bootarg.init.rootimage=”/cfcard/x86_64/freebsd/image1/rootfs.img”
bootarg.init.cfimagebase=”/cfcard/x86_64/freebsd”
ntap.init.kernelname=”/cfcard/x86_64/freebsd/image1/kernel”
bootarg.init.defaultimage=”image1″
bootarg.init.kldload_mem_size=”200413184″
bootarg.init.last_boot=”true”
bootarg.notifyd.optout=”false”
sysvar.ngsh_remote_ip=”"
bootarg.mgwd.nblade_uuid=”a13ce2d2-4371-11e2-8bf5-f33e01db9e8d”
bootarg.mgwd.spinvfs_uuid=”a13d1922-4371-11e2-9461-6d02405690a9″
bootarg.bootmenu.root_dsid=”1″
bootarg.bootmenu.root_msid=”2147483649″
bootarg.bootmenu.root_uuid=”ee6ee1f3-4391-11e2-ad9c-123478563412″
bootarg.bootmenu.node_uuid=”a13d3f5f-4371-11e2-8c70-fdb3cc539f73″
bootarg.prevent_sendhome=”false”
bootarg.init.boot_mode=”normal”
bootarg.dblade.root_volume.local_dsid=”0x723a34a9″
bootarg.dblade.root_volume.local_uuid=”ee6ee1f3-4391-11e2-ad9c-123478563412″
bootarg.dblade.root_volume.local_name=”vol0″
bootarg.dblade.root_volume.local_aggr_uuid=”ec807eda-4391-11e2-ad9c-123478563412″
bootarg.dblade.root_volume.local_event=”NORMAL”
bootarg.dblade.root_volume.local_size=”808_MB”
bootarg.dblade.root_volume.local_space=”691_MB”
REBOOT_REASON=”REBOOT_UNKNOWN”
bootarg.mgwd.mroot_found=”true”
sysvar.init.populate_mroot_late_done=”true”
bootarg.mgwd.cluster_uuid=”2770419f-438c-11e2-ad9d-123478563412″
bootarg.mgwd.cluster_name=”cl1″
bootarg.mgwd.booted=”true”
bootarg.mgwd.movestatus.sampling_int=”20″
bootarg.mgwd.movestatus.cleanup_int=”288″
bootarg.mgwd.movestatus.cleanup_now=”0″
cl1-01%