Adding new LUN from VNX on Linux - part2

Ok, so you have established communication with storage system and server.
After your storage admin create and assign LUN to your server you can precede with adding that LUN in your server.
Procedure is same as in Adding new LUN from VNX on Linux - part1 with small differences.

After rescanning of FC and scsi bus your LUNZ should be automatically replaced with LUN. LUN will have same scsi ID as LUNZ. This is best case scenario.

In case that you do not see LUN you have to remove LUNZ from system and that do rescan of FC and scsi bus. Now that your system see new scsi block devices you can add new LUN to PowerPath.

#powermt config


This command checks for available EMC LUN and add those to PowerPath configuration list.

Now issue this command

#powermt display dev=all

and you will see new LUN in your PowerPath configuration.

And that is it!

rsyslog remote log CentOS

In case that you want to collect system logs from remote server to your local server this is the procedure.
I used CentOS 6.3 with rsyslog service running for log creation. Rsyslog version is rsyslog-5.8.10-7.

Idea is to sent logs from remote server to one central log server. On this central server you can install some application that is specialised for log processing(we use LogAnalyzer for this) or just to have all remote server logs in one place.

On remote server side vi have to change file /etc/rsyslog.conf. At the end of file you have explanation for forwarding rules. Communication between remote and central server is going trough TCP port 514 or UDP port 514. It is up to you to decide which one to use. In case that you use TCP 514 you should add to /etc/rsyslog.conf file this line:

*.*   @@192.168.1.2:514 

This means
*.* -send all logs. If you want for example to send only secure log then this would be secure.*
@ @- use TCP
192.168.1.2- IP address of cental log server
514-use port 514

In case that you use UDP 514 then you should add this line:

*.*  @192.168.1.2:514

 Only difference is in monkey sign. For UDP you use one monkey. For TCP you should use two monkey signs.

Restart rsyslog service.

/etc/init.d/rsyslog restart


On central log server in file /etc/sysconfig/rsyslog you have options for rsyslog service. Default options are:

SYSLOGD_OPTIONS="-c 5 "

To enable rsyslog service to receive logs from remote servers your settings should be like this:

SYSLOGD_OPTIONS="-c 2 -r"

In file /etc/rsyslog.conf uncomment these setting depending if you want to use UDP or TCP for communication.

UDP:
#$ModLoad imudp
#$UDPServerRun 514

TCP:
#$ModLoad imtcp
#$InputTCPServerRun 514

Now add settings for remote host
:FROMHOST-IP, isequal, "192.168.1.3" /var/log/messages-remote


where

192.168.1.3- IP address of remote server from which we receive logs
/var/log/messages-remote - file in which logs from remote server will be stored

Restart rsyslog service.

Check if anything is listening on port 514.

# netstat -na |grep 514

udp        0      0 0.0.0.0:514                 0.0.0.0:*
udp        0      0 :::514                      :::*

So rsyslog service is listening on udp 514.

Check if you have anything in /var/log/messages-remote.

#tail -f  /var/log/messages-remote

On remote server do something that will trigger logs to generate (wrong password, rsyslog restart,etc.). If everything is OK you should have new entry in /var/log/messages-remote.

In case that you don't have anything in messages-remote, check your iptables, check if you are communicating through right protocol, did you restart rsyslog service,etc.

Remove LUN from Linux and PowerPath

If you want to remove LUN from Linux this is the procedure. Just to mention I am running SLES 11.1 x64 and for communication with storage I use PowerPath.
First list your powerpath devices.

#powermt display dev=all

This will list all powerpath devices. For one device it will look like this:

Pseudo name=emcpowern
CLARiiON ID=xxxxxxxxxxxx [XY]
Logical device ID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx [XY]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0;
Owner: default=SP A, current=SP A       Array failover mode: 4
==============================================
--------------- Host ---------------   - Stor -   -- I/O Path --  -- Stats ---
###  HW Path               I/O Paths    Interf.   Mode    State   Q-IOs Errors
==============================================
   0 lpfc                     sdba      SP A0     active  alive       0      0
   0 lpfc                     sdbb      SP B0     active  alive       0      0
   1 lpfc                     sdbc      SP A2     active  alive       0      0
   1 lpfc                     sdbd      SP B2     active  alive       0      0

From here you have all information you need for save removal LUN called XY from system.
First you have to remove LUN from powerpath.

#powermt remove hba=all dev=emcpowern

This will remove LUN XY from powerpath. In /var/log/messages you will have this

EMCPP: emcpAudit: Info: cmd=powermt: remove class=all dev=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx (user ID real=0 effective=0)

So we have removed LUN from powerpath.
If you want your system to be "clean" from unused scsci block devices then you should remove scsi block devices that were associated with removed LUN.

#cd /sys/block/sdba/device

if you list content of this folder you will see delete file.

#echo 1 > delete


will delete sdba scsi block device from system.

To release emcpowern block device do this

# powermt release

If you don't release emcpower (in are case emcpowern) block device, next time when you add new LUN it wont't use letter r because system see that emcpowerr device as used. It will have letter o.
You should repeat this step for rest of scsi block devices associating with removed LUN.

And that's it! :)

Adding new LUN from VNX on Linux - part1

Hi!
This is procedure for adding new LUN to Linux OS.
OS is SLES 11.1 x64, kernel is 2.6.32.54-0.3-default, new storage system is EMC VNX 7500, communication is with PowerPath version 5.5 (EMCpower.LINUX-5.5.0.00.00-275).

System already have attached LUN's from EMC Clarrion.

First thing to do is to display and save you powerpath settings.

#powermt display dev=all >state_1.txt

List your scsi devices from proc

 # cat /proc/scsi/scsi

Depending of how many paths you have to LUN, you will have that many scsi devices listed in this file. For example, this is for one LUN and this LUN have 4 paths.

Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: DGC      Model: RAID 5           Rev: 0430
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi0 Channel: 00 Id: 01 Lun: 00
  Vendor: DGC      Model: RAID 5           Rev: 0430
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi1 Channel: 00 Id: 00 Lun: 00
  Vendor: DGC      Model: RAID 5           Rev: 0430
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi1 Channel: 00 Id: 01 Lun: 00
  Vendor: DGC      Model: RAID 5           Rev: 0430
  Type:   Direct-Access                    ANSI  SCSI revision: 04

This is due to hardware architecture of your SAN network and how many FC cards do you have. In are case, we have 2 FC card Emulex A8002A. That is host attribute scsi0 for first FC and scsi1 for second FC. ID attribute is for storage FC cards. So ID 00 is for first storage FC card ID 01 is for second FC card. So you have 4 paths to single LUN. This is very good idea because you have double redundancy, redundancy of FC cards on server and redundancy od FC cards on storage system.
First you have to establish communication with your EMC VNX. This is called zoning. This means that you have to connect storage with server thru storage switch. Network admin do this. After this is you may need to restart your unihost agent. Do this in case that your storage admin tell you that he have warnings even dough he have communication with server.

#/etc/init.d/hostagent restart



This will not do any harm to your system and LUN's already attached.

After communication is established, your system will automatically recognise new storage system. Monitor your /var/log/messages for any changes. In your log you should see this.

kernel: scsi 1:0:2:0: Direct-Access     DGC      LUNZ             0532 PQ: 0 ANSI: 4
kernel: scsi 1:0:3:0: Direct-Access     DGC      LUNZ             0532 PQ: 0 ANSI: 4
kernel:  scsi 0:0:2:0: Direct-Access    DGC      LUNZ             0532 PQ: 0 ANSI: 4
kernel: scsi 0:0:3:0: Direct-Access     DGC      LUNZ             0532 PQ: 0 ANSI: 4

And in your /proc/scsi/scsi you should have new entry
Host: scsi1 Channel: 00 Id: 02 Lun: 00
  Vendor: DGC      Model: LUNZ             Rev: 0532
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi1 Channel: 00 Id: 03 Lun: 00
  Vendor: DGC      Model: LUNZ             Rev: 0532
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi0 Channel: 00 Id: 02 Lun: 00
  Vendor: DGC      Model: LUNZ             Rev: 0532
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi0 Channel: 00 Id: 03 Lun: 00
  Vendor: DGC      Model: LUNZ             Rev: 0532
  Type:   Direct-Access                    ANSI  SCSI revision: 04

LUNZ are not real LUN. they are just indication they you have communication with storage system. They are show only in first communication established with storage system and server.
If system don't automatically establish communication with storage you have to next procedure.

#echo 1 > /sys/class/fc_host/hostX/issue_lip 

where X is number of host adapter. This will restart FC interface buy doing link down and link up. Because of redundancy there will be no problems on accessing LUN that are connected through this FC card. When you are doing this step it s not a bad idea to wait a little bit before continue on another command. You can skip this step if your network admin has done link down and link up on FC link after done zoning. In messages you will see something like this

kernel: lpfc 0000:05:00.0: 0:1305 Link Down Event x4 received Data: x4 x20 x110 x0 x0
kernel: lpfc 0000:05:00.0: 0:1303 Link Up Event x5 received Data: x5 x0 x10 x0 x0 x0 0

When you finish with this on both FC cards it is time to rescan for new scsi devices.
Procedure is this:

#echo "- - -" >/sys/class/scsi_host/hostX/scan

where X is number of host adapter. You need to rescan only(but you can scan all of them) host adapter that are used for LUN. You can see what host adapters are used from several places:
-from /proc/scsi/scsi from info about already connected LUN
 from Id: 00 we know that we need to rescan host adapter 0
 from Id: 01 we know that we need to rescan host adapter 1
-from  /sys/class/scsi_disk/ where you can see attached scsi disk with their label

server:/sys/class/scsi_disk # ls
0:0:0:0
0:0:1:0
1:0:0:0
1:0:1:0

First number is representing number of host adapter (host0 and host1).
When you start rescanning for new scsi devices in your log should appear something like this
(for host1)
kernel: scsi 1:0:2:0: Direct-Access     DGC      LUNZ             0532 PQ: 0 ANSI: 4
kernel: scsi 1:0:3:0: Direct-Access     DGC      LUNZ             0532 PQ: 0 ANSI: 4

(for host0)
kernel: scsi 0:0:2:0: Direct-Access     DGC      LUNZ             0532 PQ: 0 ANSI: 4
kernel: scsi 0:0:3:0: Direct-Access     DGC      LUNZ             0532 PQ: 0 ANSI: 4

Now list /proc/scsi/scsi and new scsi devices(LUNZ) are there.
When you see all these LUNZ devices, you have connection with your storage system!!!

Huawei U8650 rooting and downgrade from Andorid 2.3.6 to 2.3.3

       I bought this phone almost 2 years ago through my telecom carrier. This was my first Android phone. I was so amazed how fast it is, how you could sync your mails accounts, how you have big screen to surf on web, in one word I was thinking that this is best phone in this price range.

But over time it became slower and slower due to new applications with higher memory usage and my subjective feeling that is more and more slow than before. Thing that especially annoyed me was that you can not uninstall application that you don't use such as FB, twitter,Documents to Go,etc.

Friend of mine rooted his U8650 and tell me that is totally different phone now, super fast, no hick ups.

So after a lot of thinking (what if something goes wrong,what if this, what if that,...) I finally decided that I should do rooting.

Friend recommended me web forum from where he rooted his phone.

So I started to do steps they recommended. Steps are:

01. Dial: *#*#2846579#*#*
02. Go to : projectmenu > background settings > log settings > log switch > select Log on
03. Go to Settings > Applications > Development > select USB Debugging
04. Restart phone
05. Download SuperOneClick v2.3.1
06. Connect phone tu computer
07. DO NOT START USB STORAGE
08. Install drivers for phone on computer. Here are link to drivers for WIN7 . WIN Xp drivers did not work for me
09. Start SuperOneClick and press ROOT button
10. Phone is rooted
11. Disable logging from step 2.

So I started to do these steps.
But when I start step 9 error occurred.

I repeated process few times but it always stops here. I first thought that this is memory card issue so I change it but same happened again.
My problem was that my phone was Android 2.3.6(customise for my telecom carrier) and SuperOneClick required Android 2.3.3. So if fisrt step if I wanted to root phone is to downgrade 2.3.6 to 2.3.3.
To downgrade Android is same as upgrade procedure. Recommendation is to downgrade it to official ROM and that is 2.3.3. Procedure for this is this:

01. Backup your contact, sms and generally things that is important to you
02. Unzip downloaded ROM
03. Copy folder dload to your sdcard. Inside of this folder is file update.app. Path to this file MUST be /dload/update.app
04. Shutdown your phone
05. Press VOLUME+ , VOLUME - and POWER button together and hold it until installation start. It takes few seconds to start
06. Leave phone until it ends installation
07. Phone will reboot
08. You have U8650 with Android 2.3.3

For rooting, now repeat procedure with SuperOneClick.

Phone is rooted. And downgraded.

When I started to use phone now I count believe that this is same phone! I is super fast!  First there was no commercial application like FB, Tweeter and stuff like that. Because is Huawei, it still has Hi market (something like google play for China) and Huawei Music. Because this was phone was from telecom operator now I didn't have APN automatically set up. You can call your operator to send you these settings or you can insert them yourself. And one more important thing, there is no that slider that holds wi-fi,blue tooth, gprs, sync. And this is quite annoying because you can not fast turn on-off wifi, gprs, etc.

First impression on this clean 2.3.3 on Huawei U8650 is very, very good!!!

Things that annoyed me before downgrade and root is:

- when I am writing something ( sms ,notes, web, basically anything) I had delay off 2-3 seconds before letters start to show. Now there is no delay. 

- I coun't  use Viber because when I start it phone gets suuuupppeeeerrrr slllloooooowwwww. So slow that you could not use VIBER. Now Viber runs like Ferrari.

- Same thing with Skype

I compared with my colleagues my U8650 and theirs U8650 with telecom carrier 2.3.6. It is much, much, faster!

Before this I was considering of changing my phone but now ..... I stay with Huawei U8650!

HP Data Protector Express Single Server 4.00-sp1 restore issue

We were using HP Data Protector Express Single Server 4.00-sp1 for quite some time. More that 4 year and everything was working perfectly! Backup, restore, backup, restore, perfectly! DPX was running on Windows2003 Server and have HP StorageWorks Ultrium 960 tape device. We have to give away that specific tape device. In my earlier post, I wrote about issue with tape devices and how I handle that.

Reason why we need HP tape drive and not ape device from some other vendor like IBM(and we do have IBM System Storage Ultrium LTO3 tape drive) is because DPX Single Server only support HP tape drive devices.

Here is compatibility matrix for DPX 4.00-sp1.

http://h18000.www1.hp.com/products/storage/software/datapexp/pdf/DPXMatrix_0310.pdf

So we were stuck with HP tape drives if we wanted to continue using DPX Single Server or we could buy licence for DPX but this was not an option.

Ok, so I attached resurrected HP StorageWorks Ultrium 960 tape drive to server and start restoring data from LTO3 tapes. I had to restore 6 tapes. I managed to restored data from first two tapes. When I start to restore data from third tape and then problems start to show.
Errors were

Error 1040:No devices specified or all devices are now offline
Error 4408: Object end not found 

It is very important to say that this error appears after 18 hours of DPX working on tape and that is REALLY frustrating!!! If you want to cancel process you have to reboot server because job that you want to cancel is in waiting for termination state and that last for hours!

 

First I suspected on tape devices but I tested them using HP Library and Tape Tools(LTT) and all test (drive test and tape test) passed. I then suspected on SCSI cable so I replace him but still same issue appeared. After all this excluded only thing that is left is Data Protector.  I was to afraid to do anything with DPX because of licence issues and what will happened if I upgrade it to newer version.

So I downloaded DPX 5.00 version just to try it. I installed it on RH EL 6.3. HP tape drives were recognised  instantly. I imported tape to DPX 5.0. This process lasted for 23 hours. When I start to restore data to file system errors start to appeared! But this time it was different.

Error 3005: Invalid stream

Error 3010: Incompatible format

I found on some forum that if have error 3010 using tape device it could only be that when you were doing backup option native data stream format was checked. This option is checked by default in DPX 4.00-sp1. This means that data can be only restored to same file system and as I try to restore data on EXT3 instead on NTFS.

I install Win2003, install DPX 5.00 and now I am restoring data with no problem.

I still don't know what caused this problem but I suspect on DPX 4.00 database.

I hope that this can help someone and lots of time!

HP Data StorageWorks Ultrium 960 tape drive power issue

Few months ago we have to give away are HP Data StorageWorks Ultrium 960 tape drive. That device worked perfectly. And now it was gone. For good. And we need another device but it has to be HP tape device and we needed fast.
Instead of that perfect device we receive same (HP Ultrium 960) device but not so perfect. I will call this device D1. Device was showing that it has power/fan issue. Back LED was steady amber colour. I opened device thinking that if is a fan issue it can be easy replaced with different fan. When I attached fan wire, to battery fan was working perfectly. So it was power issue. As I have no spare power supply, this was dead end fow device D1. :(
After a while we realize that we two more HP  data drive (Data StorageWorks Ultrium 960). One was on server that don't have need for tape device any more. We will call this device D2. Second tape drive was on server that periodicly need tape drive. We will call this device D3.

I detached D2 device from server OS(RHEL 5.5 x64) using

echo 0 > /sys/class/scsi_tape/device/delete 

and than physicaly detach device. At that time back LED was green and front LED that is on power button was green to. Device was attached to this server more that 5 year with no power off.
 When I attached that perfectly D2 device to first server I was showing errors! Power supply/fan error! Again! On totally different tape device that 20 minutes ago was working perfectly. My first tought was that it got something to do with voltage because those devices were on different location and probably different voltage level burn something in power supply or fan. So I opened D2 device and attach fun to battery, it was working perfectly. So power supply issue again. Dead end again. :(

Same thing happened with device D3. :(
So I have 3 tape device that was working for quite some time(5 years and more) and now all of them have same issue-bad power supply.
I even try to switch power supply from one device to another hoping that somehow it will start to work. But no luck.
After few days I was thinking why I would not attach PC power supply to tape drive? It has same jacks,same 12V voltage. I found old PC that was lying around in a server room. I connected power to PC and connect it to dismantled tape drive and turn PC on. After a few seconds tape drive was in READY state, back LED for power/fan issue was blinking amber. I attached tape drive to server and after rescaning scsi_host I could see tape drive! Finally succsess!!! I found two more power supplies from PC and now ALL tape device are in READY state and working! All device are fully operational even if they are have yellow blinking LED from power/fan issue and have external power supply! :)
Working of devices was tested using tar for backup and restore and using HP Data Protector for restoring data from LTO3 tapes! :)

I hope this will help someone and save some critical time!