|
Discussion Forums
|
Thread: Debian Lenny and XFS causes kernel panic: Suggested Workaround
|
|
Posts:
8
Registered:
12/13/08
|
|
|
|
Re: Debian Lenny and XFS causes kernel panic: Suggested Workaround
Posted:
Mar 21, 2009 4:31 AM PDT
in response to: AndrewC@AWS
|
|
|
Oh, I didn't mean to emphasize workaround at all. I think I pasted the thread title into the editor which became bold automatically and I didn't notice it. Sorry.
I am setting up a private beta system using the new module (at least temporarily) on the weekend. I'm going to let you know how it went!
|
|
Posts:
428
Registered:
2/5/08
|
|
|
|
Re: Debian Lenny and XFS causes kernel panic: Suggested Workaround
Posted:
Mar 21, 2009 6:04 AM PDT
in response to: Norbert Mocsnik
|
|
|
I've been bitten by pasting of formatted text here as well. I just wanted to make sure we weren't missing something fundamental.
Also, you wrote that you've still seen hangs with the updated module. Were the symptoms the same (CPU soft lockup)?
Has anybody else seen a lockup with the new module?
Andrew
|
|
Posts:
8
Registered:
12/13/08
|
|
|
|
Re: Debian Lenny and XFS causes kernel panic: Suggested Workaround
Posted:
Mar 21, 2009 3:02 PM PDT
in response to: AndrewC@AWS
|
|
|
I have successfully installed MySQL to an EBS device on Debian Lenny using the xfs tools provided by Fio in this thread. No hangs whatsoever. Also created a snapshot from the EBS volume, then created a new EBS volume from the snapshot, attached and mounted the new volume and it worked perfectly. I have increased the volume size from 10 to 30 GB using xfs_growfs from the new package and that also worked fine.
One thing though, I noticed that xfs_freeze which is mentioned in the following tutorial:
http://developer.amazonwebservices.com/connect/entry!default.jspa?categoryID=112&externalID=1663
is missing from the package. I went ahead and did an "aptitude install xfsprogs" and used xfs_freeze from that and it did the job. It would be nice to have it included in the new package.
|
|
Posts:
9
Registered:
3/9/09
|
|
|
|
Re: Debian Lenny and XFS causes kernel panic: Suggested Workaround
Posted:
Mar 23, 2009 8:34 PM PDT
in response to: Lenny
|
|
|
Quick FYI: the two times that my Hardy instance locked up with a MySQL DB on an XFS EBS volume, the instance had been running for a few weeks with no problems at all.
|
|
Posts:
1,134
Registered:
7/7/07
|
|
|
|
Re: Debian Lenny and XFS causes kernel panic: Suggested Workaround
Posted:
Mar 23, 2009 9:47 PM PDT
in response to: critke1
|
|
|
critke1:
Based on my understanding of the XFS+2.6.21 kernel incompatibility, I don't think it happens on Ubuntu 8.04 Hardy, so what you're experiencing is probably something else.
You might consider starting a new thread with as many details as you can about the issue so you can receive ideas from the community. Mention "Ubuntu" in the subject and I'll see it.
|
|
Posts:
104
Registered:
5/14/08
|
|
|
|
Re: Debian Lenny and XFS causes kernel panic: Suggested Workaround
Posted:
Mar 25, 2009 11:34 AM PDT
in response to: AndrewC@AWS
|
|
|
Andrew's statement is correct. All I did was to rebuild the xfs kernel module against the full FC8 source tree built with the same parameters as the underlying AKI.
My theory is that the problems you are experiencing are unrelated.
One thing you could do is run the AMI with this aki:
IMAGE aki-46e7002f aki-linux.2.6.21.7-2.fc8xen-xfs/vmlinuz.manifest.xml amazon available public i386 kernel
This would allow to rule out with 100% certainty any possible xfs module / kernel incompatibilities, as this AKI is the rebuilt kernel and the xfs module comes from the same set of source code.
Best Regards,
-- Fio
|
|
Posts:
9
Registered:
3/18/09
|
|
|
|
Re: Debian Lenny and XFS causes kernel panic: Suggested Workaround
Posted:
Mar 25, 2009 3:04 PM PDT
in response to: Norbert Mocsnik
|
|
|
Norbert:
There is a question in my mind as to whether we need the updated xfsprogs at all. I used Fio's kernel module combined with xfsprogs from the Lenny distribution, and it worked fine for me. I suspect all we actually need is the kernel module.
Fio:
Based on your understanding of this bug, why do we need your updated xfsprogs? In your post earlier in this thread where you announced the release of the updated kernel module, you also released updated xfsprogs, but made it sound like we only needed some of them. I'm also curious to learn why the xfsprogs were recompiled, and why we should use the updated mkfs.xfs and xfs_repair. Should we also update the other components of xfsprogs, or just these two, and if so, why only these two?
Thanks,
Matt
|
|
Posts:
8
Registered:
12/13/08
|
|
|
|
Re: Debian Lenny and XFS causes kernel panic: Suggested Workaround
Posted:
Mar 25, 2009 3:49 PM PDT
in response to: Matt Olson
|
|
|
Matt: That may be true, I'm going to give it a try soon.
|
|
Posts:
104
Registered:
5/14/08
|
|
|
|
Re: Debian Lenny and XFS causes kernel panic: Suggested Workaround
Posted:
Mar 30, 2009 2:17 PM PDT
in response to: Matt Olson
|
|
|
Hello, apologies for my late replies, and also apologies to all for not having answered previous questions as to why I also provided a recompiled xfsprogs.
When I did my initial investigation, I rebuilt everything (kernel, xfs module and xfsprogs) using the FC8 defaults.
After some quick testing (also I should publicly say thanks to some early adopters to did a private set of tests before making the general announcement), I managed to quickly rule out the kernel as a source of the problem.
My belief at this point (also based on customer's experiences) is that the only culprit was a badly built xfs.ko kernel module, but because at the time of the bug fix release I could not be 100% sure, I suggested people to also use what was at the time the latest stable version of xfsprogs.
The only reason I still provide the xfsprogs is that they are compiled with some kernel headers, so technically they're still dependent on kernel build parameters. I would assume that whatever structures and constants xfsprogs use from kernel headers should be independent from build parameters -- otherwise it'd be rather difficult to upgrade xfs to a newer version (or even use the same version but built with different parameters) and still be able to read/write the old filesystems.
Because I cannot possibly test all combinations of AKIs, ARIs and AMIs, I cannot say with 100% certainty "go ahead and use whichever version of xfsprogs you want", which is why I still provide the custom built version of xfsprogs, just in case ;-)
And yes, these considerations would apply to all utilities contained in xfsprogs.
I hope this clarifies the issue.
Best Regards, and again apologies for the late reply.
-- Fio
|
|
Posts:
1
Registered:
4/3/09
|
|
|
|
Re: Debian Lenny and XFS causes kernel panic: Suggested Workaround
Posted:
Apr 3, 2009 2:39 AM PDT
in response to: fio@AWS
|
|
|
hi fio, can your kernel image above be migrated to the EU region?
|
|
Posts:
104
Registered:
5/14/08
|
|
|
|
Re: Debian Lenny and XFS causes kernel panic: Suggested Workaround
Posted:
Apr 3, 2009 11:16 PM PDT
in response to: George Agnelli
|
|
|
Greetings,
I've done a very cursory test and found that the US xfs.ko kernel module seems to work with the Lenny OS AMI I've tried (see below for details).
Please note, I've done a very cursory test and I've not subjected it to any kind of extensive stress tests. You are more than welcome to try the steps I describe below, with the understanding that we have not subjected the code to the kind of tests we have subjected the AMIs mentioned in earlier in the thread.
For EU, the download / install process is sligthly different, you only need to download the tarball which contains the xfs.ko kernel module and then you'll use the standard xfsprogs which come with Debian.
If you do try this, please let us know the results of your tests.
Best Regards,
--Fio
INSTANCE i-763d0502 ami-6a01291e ec2-79-125-50-34.eu-west-1.compute.amazonaws.com ip-10-224-63-82.
eu-west-1.compute.internal running gsg-keypar 0 m1.small 2009-04-04T05:38:53+0000
eu-west-1a aki-7e0d250a ari-7d0d2509
This AMI is the following Lenny OS:
Linux ip-10-224-63-82 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686
.......
.......
Amazon EC2 Debian 4.0 etch AMI built by Eric Hammond
http://alestic.com
http://ec2debian-group.notlong.com
In my test, I did the following:
ip-10-224-63-82:~# apt-get update
ip-10-224-63-82:~# apt-get upgrade
Then I downloaded and installed the xks.ko module as follows:
ip-10-224-63-82:~# wget
https://s3.amazonaws.com:443/public-linux.2.6.21.7-2.fc8xen-xfs/linux-fs-modules-build-2.6.21.7-
2.fc8xen-xfs.tar.gz
# verify md5 checksum just to be sure the download was ok
ip-10-224-63-82:~# md5sum linux-fs-modules-build-2.6.21.7-2.fc8xen-xfs.tar.gz
70c424e98acdc9b323f3ee6f4a38a26c linux-fs-modules-build-2.6.21.7-2.fc8xen-xfs.tar.gz
ip-10-224-63-82:~# tar xfzo linux-fs-modules-buil-2.6.21.7-2.fc8xen-xfs.tar.gz
ip-10-224-63-82:~# insmod linux-2.6.21.i386/fs/xfs/xfs.ko
ip-10-224-63-82:~# lsmod
Module Size Used by
xfs 497624 0
After loading the updated xfs.ko kernel module, I simply installed the vanilla xfsprogs from the Debian distro (do not install the ones from download_xfs.sh):
ip-10-224-63-82:~# aptitude install xfsprogs
At this point I was able to mkfs an XFS file system, copy files, dismount it, run xfs_repair to check integrity and compare the copied files:
i
p-10-224-63-82:~# mkfs.xfs -f /dev/sda2
ip-10-224-63-82:~# mount /dev/sda2 /mnt
ip-10-224-63-82:~# cp -r linux-fs-modules-build-2.6.21.7-2.fc8xen-xfs.tar.gz linux-2.6.21.i386 /mnt
ip-10-224-63-82:~# umount /dev/sda2
ip-10-224-63-82:~# xfs_repair /dev/sda2
Phase 1 - find and verify superblock...
Phase 2 - using internal log
....
....
Phase 7 - verify and correct link counts...
done
ip-10-224-63-82:~# mount /dev/sda2 /mnt
ip-10-224-63-82:~# df
/dev/sda2 156276168 343460 155932708 1% /mnt
ip-10-224-63-82:~# md5sum linux-fs-modules-build-2.6.21.7-2.fc8xen-xfs.tar.gz
70c424e98acdc9b323f3ee6f4a38a26c linux-fs-modules-build-2.6.21.7-2.fc8xen-xfs.tar.gz
ip-10-224-63-82:~# md5sum /mnt/linux-fs-modules-build-2.6.21.7-2.fc8xen-xfs.tar.gz
70c424e98acdc9b323f3ee6f4a38a26c /mnt/linux-fs-modules-build-2.6.21.7-2.fc8xen-xfs.tar.gz
ip-10-224-63-82:~# ls -la /mnt
total 92364
drwxr-xr-x 4 root root 105 2009-04-04 05:56 .
drwxr-xr-x 20 root root 4096 2009-04-04 05:40 ..
drwxr-xr-x 4 root root 89 2009-04-04 05:56 linux-2.6.21.i386
-rw-r--r-- 1 root root 94576200 2009-04-04 05:56 linux-fs-modules-build-2.6.21.7-2.fc8xen-xfs.tar.gz
drwxr-xr-x 2 root root 6 2009-04-04 05:57 lost+found
|
|
Posts:
1,134
Registered:
7/7/07
|
|
|
|
Re: Debian Lenny and XFS causes kernel panic: Suggested Workaround
Posted:
Apr 20, 2009 8:37 PM PDT
in response to: fio@AWS
|
|
|
Thanks Fio!
The most recent release (2009-04-18) of the affected Ubuntu and Debian images on
http://alestic.com include the XFS kernel module for the 2.6.21fc8 kernel.
In my testing, this corrects the problem that I was seeing with XFS on Ubuntu Intrepid, Ubuntu Jaunty, Debian Lenny, (and probably Debian Squeeze though I'm not sure anybody is running that).
I encourage folks to do their own testing with these images and report back if you find any problems with XFS.
--
Eric Hammond
http://twitter.com/esh
|
|
Posts:
16
Registered:
11/6/07
|
|
|
|
Re: Debian Lenny and XFS causes kernel panic: Suggested Workaround
Posted:
May 12, 2009 9:44 AM PDT
in response to: Lenny
|
|
|
Hi Flo,
It looks like this module only supports i386. Are there any intentions of releasing a 64-bit version?
Thanks,
Martin
|
|
Posts:
104
Registered:
5/14/08
|
|
|
|
Re: Debian Lenny and XFS causes kernel panic: Suggested Workaround
Posted:
May 12, 2009 11:14 AM PDT
in response to: ermal14
|
|
|
Yes we can certainly do that. I won't be able to get to it immediately though.
I will post on this forum once it's available.
Thanks,
--Fio
|
|
Posts:
104
Registered:
5/14/08
|
|
|
|
Re: Debian Lenny and XFS causes kernel panic: Suggested Workaround
Posted:
May 29, 2009 2:41 PM PDT
in response to: ermal14
|
|
|
Greetings,
We now have available a 64 bits build for the xfs kernel module for the following AKI:
IMAGE aki-9800e5f1 ec2-public-images/vmlinuz-2.6.18-xenU-ec2-v1.0.x86_64.aki.manifest.xml amazon available public x86_64 kernel
The xfs.ko kernel module is made available on an as-is basis, as we do not have the resources to do proper testing for all possible combinations.
You are more than welcome to try it out and we'd like to provide us feedback if you have any.
NOTE: we've only tested this on ami-cd52b6a4, although it should work on any ami which uses the AKI shown above.
How to install:
* download the script
https://s3.amazonaws.com:443/public-linux.2.6.18-xenu-ec2-v1.0-x64-xfs/download_xfs_64.sh
* the script will download a tarball file and extract it under ~root/xfs_x64
* the tarball MD5 checksum is 661452c55015fb0934292dd866221a24
* load the xfs.ko kernel module
* install xfsprogs
Example:
RESERVATION r-a74125ce 716644959191 default
INSTANCE i-0528686c ami-cd52b6a4 ec2-174-129-158-5.compute-1.amazonaws.com domU-12-31-39-02-DA-52.compute-1.internal running gsg-keypar 0 m1.large 2009-05-27T02:24:37+0000 us-east-1a aki-9800e5f1
IMAGE ami-cd52b6a4 rightscale-us/CentOS5_2_X86_64_V4_1_10.manifest.xml 411009282317 available public x86_64 machine aki-9800e5f1
cattaneo@cattaneo-2.desktop.amazon.com # ssh -i $EC2_GSG_PRIVATE_KEY root@ec2-174-129-158-5.compute-1.amazonaws.com
.....
.....
Last login: Fri May 29 15:36:34 2009 from 72.21.196.64
[root@domU-12-31-39-02-DA-52 ~]# wget
https://s3.amazonaws.com:443/public-linux.2.6.18-xenu-ec2-v1.0-x64-xfs/download_xfs_64.sh
[root@domU-12-31-39-02-DA-52 ~]# chmod 755 download_xfs_64.sh
[root@domU-12-31-39-02-DA-52 ~]# ./download_xfs_64.sh
./download_xfs_64.sh: downloading linux_fs x64 (may take quite a while, please wait) ....
./download_xfs_64.sh: MD5 checksum for linux-2.6.18-xenU-x64-xfs.tgz ok: 661452c55015fb0934292dd866221a24
./download_xfs_64.sh: unpacking linux_fs tarball ...
./download_xfs_64.sh: x86_64 xfs.ko kernel module is at /root/xfs_x64/build-linux-2.6.18-xenU_x86_64/fs/xfs/xfs.ko
-rw-r--r-- 1 root root 720434 May 29 14:35 /root/xfs_x64/build-linux-2.6.18-xenU_x86_64/fs/xfs/xfs.ko
./download_xfs_64.sh: NOTE: use the standard xfsprogs for mkfs.xfs, xfs_repair and all the other xfs user tools
[root@domU-12-31-39-02-DA-52 ~]# insmod /root/xfs_x64/build-linux-2.6.18-xenU_x86_64/fs/xfs/xfs.ko
[root@domU-12-31-39-02-DA-52 ~]# tail /var/log/messages
.....
.....
May 29 16:54:45 domU-12-31-39-02-DA-52 kernel: [9634199.007913] SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled
May 29 16:54:45 domU-12-31-39-02-DA-52 kernel: [9634199.008388] SGI XFS Quota Management subsystem
[root@domU-12-31-39-02-DA-52 ~]# yum install xfsprogs -y
.....
.....
Installed: xfsprogs.x86_64 0:2.9.4-1.el5.centos
Complete!
[root@domU-12-31-39-02-DA-52 ~]# mkfs.xfs -f /dev/sdj
meta-data=/dev/sdj isize=256 agcount=16, agsize=4096000 blks
= sectsz=512 attr=0
data = bsize=4096 blocks=65536000, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal log bsize=4096 blocks=32000, version=1
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
[root@domU-12-31-39-02-DA-52 ~]# mount /dev/sdj /kernel-build/
[root@domU-12-31-39-02-DA-52 xen-3.1.0]# cp /mnt/xen-3.1.0-src-ec2-v1.0.tgz /kernel-build/
[root@domU-12-31-39-02-DA-52 xen-3.1.0]# cd /kernel-build/
[root@domU-12-31-39-02-DA-52 kernel-build]# tar xfoz xen-3.1.0-src-ec2-v1.0.tgz
[root@domU-12-31-39-02-DA-52 xen-3.1.0-src-ec2-v1.0]# make linux-2.6-xenU-build
make -f buildconfigs/mk.linux-2.6-xenU build
....
....
KSYM .tmp_kallsyms2.S
AS .tmp_kallsyms2.o
LD .tmp_vmlinux3
KSYM .tmp_kallsyms3.S
AS .tmp_kallsyms3.o
LD vmlinux
SYSMAP System.map
SYSMAP .tmp_System.map
OBJCOPY vmlinux-stripped
GZIP vmlinuz
make[2]: Leaving directory `/kernel-build/xen-3.1.0-src-ec2-v1.0/build-linux-2.6.18-xenU_x86_64'
....
....
[root@domU-12-31-39-02-DA-52 xen-3.1.0-src-ec2-v1.0]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 10321208 2236188 7560732 23% /
/dev/sdb 433455904 999848 410437752 1% /mnt
none 3932228 0 3932228 0% /dev/shm
/dev/sdj 262016000 581928 261434072 1% /kernel-build
[root@domU-12-31-39-02-DA-52 xen-3.1.0-src-ec2-v1.0]# cd
[root@domU-12-31-39-02-DA-52 ~]# umount /dev/sdj
[root@domU-12-31-39-02-DA-52 ~]# xfs_repair /dev/sdj
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
Phase 5 - rebuild AG headers and trees...
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done
|
|
|
|
Sign in to the AWS Management Console
Create an AWS Account
