HDD is read-only & server cant boot, what to do on IPMI console?

[ICPH]

Hello,

can You please give me an advice, it appears my partition / is whole read only suddenly, and cant remount into writable sate:

[root@serv ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
1866322480 554196528 1215792948 32% /
/dev/sda1 101086 38675 57192 41% /boot
tmpfs 12312444 0 12312444 0% /dev/shm

[root@serv ~]# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

mount: warning: /etc/mtab is not writable (e.g. read-only filesystem).
It's possible that information reported by mount(8) is not
up to date. For actual information about system mount points
check the /proc/mounts file.
 
[root@serv ~]# fdisk -l
 
Disk /dev/sda: 2000.3 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
Disk /dev/sda doesn't contain a valid partition table

[root@serv ~]# fdisk /dev/sda
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 243201.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
 

[root@serv ~]# blkid
/dev/mapper/VolGroup00-LogVol01: TYPE="swap"
/dev/mapper/VolGroup00-LogVol00: UUID="9a2e6ab0-00b5-46c0-bbc6-e0bcc41f1315" TYPE="ext3"
/dev/sda1: LABEL="/boot" UUID="722940e7-59fd-417f-8b31-fbf574090120" TYPE="ext3" SEC_TYPE="ext2"
/dev/VolGroup00/LogVol00: UUID="9a2e6ab0-00b5-46c0-bbc6-e0bcc41f1315" TYPE="ext3"
/dev/VolGroup00/LogVol01: TYPE="swap"

smartctl dont appears to output any errors..

How can i try to fix this please?

 

[Francisco]

You need to run a FSCK to get it back online.

You should be able to just reboot and it'll FSCK itself, likely ask you for your root password as well.

Check /var/log/messages and see if it says anything about IO errors (not ext3/4 errors). It's possible the node is having a RAID failure.

Francisco

 

[ICPH]

Thank You very much,

this is last 50 lines of /var/log/messages: http://pastebin.com/LDQ5WG0r

you say i should reboot, You think fstab is ok and will boot?

 

[Francisco]

Should be fine.

Francisco

 

[ICPH]

Hi, after power-on server i see via IPMI:

 

Reboot and select proper boot device or insert boot media in selected boot device

No DHCP or proxyDHCP offers were received

Media test failure, check cable

Exitting intel boot agent

please kindly which commands i need to do via Dell IPMI console?

 

[MartinD]

Thread title updated as per OP's request.

 

[MartinD]

Do you only have 1 drive in the machine?

 

[ICPH]

Do you only have 1 drive in the machine?

Yes, only one, thank you..

 

[MartinD]

You're kinda screwed then.

You'll need to mount some kind of rescue ISO via IPMI to get access to the drive and see what's going on. If it's a drive based, I'd check the MBR. Wouldn't hurt to have the DC check the drive cable, too.

 

[Francisco]

The fact you can still see the drive should give you hope.

It's possible you somehow lost your partition table, which isn't good, but if that's the worst of the damage you can actually recover that fairly easy.

If you're still trying to recover it, just drop me a message on IRC later tonight and I'll lend you a hand.

Francisco

 

[ICPH]

MartinD, thank You alot for your valuable info.

Francisco, thank You for willing to help me out. I sent You a PM, i hope you will receive it.

Thank you for your words

 

[Francisco]

I'm confused, wasn't this node using LVM prior?

I'm only seeing a single partition making up all of /dev/sda*

I could run fsck on it but didn't want to do that until I got your feedback.

Francisco

 

[ICPH]

Thank You,

there has been an try to scan HDD and restore partitions, so this might be an unwanted result i think.. if you think fsck can help to restore booting or help recover data from /home, please do it

Thank you very much,

V.

 

[Francisco]

I'm sorry to tell you, but you're pretty much boned on this one.

FSCK ran and did what it could but it didn't even attach anything to lost+found.

The drive itself is in good condition, SMART reports no issues and the smart tests that have been run are also clean.

You'll need to recover from backups if you have any

I'm sorry I wasn't of better help.

Francisco

 

[ICPH]

Thank You very much for being so kind to take a look at it. Thank you for help.

V.