(Changing status back to resolved...sorry again...bugzilla does that by itself
if you forget to set status...p.s. no need to help us, we built custom
2.6.20.15-grml for remaster)

The time testing and discussion was beta cycle, not now.  This "one single
regression" is affecting literally thousands of users + dozens of devices and
Google confirms that to my satisfaction.

Use of 2.6.20 is not as arbitrary.  It has known workaround.  That is far more
sensible at this stage than playing with Linus's latest experiments when Google
says they're broken anyway.

One way grml could handle the problem is to put an alternate kernel/module set
in the 1.0 repository.  Then users have a choice.

* Mark <bts@bts.grml.org> [20070706 00:15]:

[...]

Bug isolation as in "let's see whether the regression is still
present"....

[...]
> We need kernel modules - nVidia, kqemu, truecrypt, etc.  A stand-alone kernel is
> useless.  So we have to compile no matter what, and 2.6.20/n was/is safest right
> now.  We'll see about 2.6.22.

*grml* You shouldn't use the "stand-alone kernel" for production use
but just boot it once and see whether the USB issue got fixed.  But
I wrote that too many times already...

> But regardless...grml will keep USB suspend no matter what.  So 2.6.21 TEST IS
> USELESS FOR GRML BECAUSE IT HAS NO EFFECT ON GRML DESIGN.  No matter what
> happens with 2.6.21 or 2.6.22 or 2.7.0 or even 3.0.0, grml will keep USB suspend
> and grml will use the latest kernel.org.  There is no use in any kernel++ test.

No. Not "no matter what", but I don't want to disable a usefull
feature just because there's one single regression in the kernel.
Instead of disabling that feature, the right way would be to find
the real source of the problem. And if the bug is still present in
2.6.21 we could test the current version of 2.6.22-rc to make sure
it can be fixed for final 2.6.22. But I wrote that as well too many
times already...

> About the flag.  Correct, it is 2.6.20 code - which means the flag is broken. 
> Newer suspend code in the kernel is breaking things.  The compile flag just
> switches it on/off.  Saying the flag has always worked is misleading, because
> the code that it switches has changed.  So sure, it's 2.6.20 regression.  And
> that means the flag is now switching bad code on/off.  We plan to turn off the
> bad code.

You could try
http://bugzilla.kernel.org/attachment.cgi?id=10313&action=view
if you plan to recompile 2.6.20 *with* CONFIG_USB_SUSPEND=y.

Anyway, according to some googling around the kernels behind
2.6.21-rc4 should bring updated code in regard to USB suspend...
But as you insist on wasting your time with 2.6.20 instead...

regards,
-mika-

FYI.
2.6.22 rc 4 Debian:  same trouble, same fix.
http://bugzilla.kernel.org/show_bug.cgi?id=8606
"Disabling CONFIG_USB_SUSPEND fixes this."
- Andreas Kloeckner  2007-06-24

Over and out.  -Mark

> end of discussion now

Well then, for the record only:

Mika said on grml-list that 2.6.21 was for BUG ISOLATION only.  For BUG
ISOLATION, 2.6.20/n is MORE VALID than 2.6.21/y because ONLY ONE THING HAS
CHANGED.  That is how to perform bug isolation - change one thing at a time.  In
this case, a compile flag.

If grml needs such testing, please respond during beta cycle.

> No. What would you gain from CONFIG_USB_SUSPEND=n on 2.6.20?

BUG ISOLATION.  And, a known workaround, as reviewed by Ubuntu.
http://usbsuspendbackport.wikispaces.com/

We need kernel modules - nVidia, kqemu, truecrypt, etc.  A stand-alone kernel is
useless.  So we have to compile no matter what, and 2.6.20/n was/is safest right
now.  We'll see about 2.6.22.

But regardless...grml will keep USB suspend no matter what.  So 2.6.21 TEST IS
USELESS FOR GRML BECAUSE IT HAS NO EFFECT ON GRML DESIGN.  No matter what
happens with 2.6.21 or 2.6.22 or 2.7.0 or even 3.0.0, grml will keep USB suspend
and grml will use the latest kernel.org.  There is no use in any kernel++ test.

About the flag.  Correct, it is 2.6.20 code - which means the flag is broken. 
Newer suspend code in the kernel is breaking things.  The compile flag just
switches it on/off.  Saying the flag has always worked is misleading, because
the code that it switches has changed.  So sure, it's 2.6.20 regression.  And
that means the flag is now switching bad code on/off.  We plan to turn off the
bad code.

USB devices have worked since ages, and new suspend code is breaking drivers. 
The kernel devs say that, not me.  But we acknowledge grml design choices, and
that's fine, we can recompile.

It's definitely end of discussion for us, too.  Let's hope 2.6.22 brings some
good news.

* Mark <bts@bts.grml.org> [20070705 04:15]:

> Closing.  The problem is the scanner and CONFIG_USB_SUSPEND as reported in
> Ubuntu 2.6.20.

> https://launchpad.net/ubuntu/+source/sane-backends/+bug/85488

> It's not the modules, it's not the hub, it's not the PC hardware, it's Linus
> getting too excited about laptop features.

> Linus should not have released this buggy flag.  These scanners all worked fine
> for a long time in Linux.  Vendor hardware is buggy to an extent, but as Linus
> likes to say about his kernels, perfection is impossible and that's the real
> world, so deal with it.

Again: we are using CONFIG_USB_SUSPEND since ages and it never
caused serious failures until 2.6.20. It's a *regression* in 2.6.20.

> Testing 2.6.21 was never useful to us at all.  The right test was to recompile
> 2.6.20 with the flag off.  That's been my focus as time allows.  But GCC keeps
> segfaulting even with 1.5 GB of RAM.  We're still running grml 0.9.

No. What would you gain from CONFIG_USB_SUSPEND=n on 2.6.20?
Nothing grml related. Because grml will continue in using
CONFIG_USB_SUSPEND=y.  So testing a newer kernel (to see whether the
regression is fixed or not) is what's important. That's what I said
more than once and I offered you a ready-to-go Debian package of
2.6.21-grml.  But feel free to do whatever you think is right for
you, for me it's end of discussion now.

regards,
-mika-

(Changing status back to resolved, my mistake, sorry.)

Suspend is a mess according to several kernel developers.  There is no 'big
picture' architecture yet.  Many drivers are breaking.  So, it is not just a
scanner problem.  FYI.
http://lkml.org/lkml/2007/7/4/268

Closing.  The problem is the scanner and CONFIG_USB_SUSPEND as reported in
Ubuntu 2.6.20.

https://launchpad.net/ubuntu/+source/sane-backends/+bug/85488

It's not the modules, it's not the hub, it's not the PC hardware, it's Linus
getting too excited about laptop features.

Linus should not have released this buggy flag.  These scanners all worked fine
for a long time in Linux.  Vendor hardware is buggy to an extent, but as Linus
likes to say about his kernels, perfection is impossible and that's the real
world, so deal with it.

Testing 2.6.21 was never useful to us at all.  The right test was to recompile
2.6.20 with the flag off.  That's been my focus as time allows.  But GCC keeps
segfaulting even with 1.5 GB of RAM.  We're still running grml 0.9.

For us the fix is to remaster grml 1.0 with the flag off, if GCC will cooperate.
 We're a little worried about some other XFS bugs reported in the late 2.6.22
rc's so we are going to watch kernel things more carefully in the future.

* Mark <bts@bts.grml.org> [20070623 08:15]:

> Test results:  the modules are not the problem.  We blacklisted each one in the
> diff list, but all tests gave identical results.  The USB disk was not detected
> properly.

Ok.

> But good news, a different test isolated better.  Here is the full USB system
> that grml 1.0 CD-ROM botches:

> PC --> USB 2.0 Port A --> USB 2.0 Disk
>      --> USB 2.0 Port B --> USB 2.0 Hub --> USB Scanner

> Removing the USB cable to the hub makes grml 1.0 CD-ROM recognize the USB Disk
> (with udev on, no special boot options).

> So the bug happens only with the hub plugged in.  Then grml 1.0 fails to see the
> USB Disk.  If the hub is unplugged, udev works, and grml sees the USB Disk.

Ok, thanks.

> grml 2.6.18 handles everything plugged in, very reliably.  So it is not a USB
> hardware glitch.  In fact the hub is practically brand new.

JFTR: Because hardware is new it does not mean that it's ok. ;)
You can have brand new hardware which just ignores parts of the
specification and therefore does not work on operating systems
having some more checks inside the code than others. (Yes, I
explored such issues already on my own.)

> Let us know what other data you want.  We're pretty convinved it's a udev
> hotplugging issue - maybe the timeout is too short? - udev in 2.6.20.11 is
> having problems with USB hubs.

> We can try the 'grml nousb nousb2' test tomorrow if it will help.

As I've written already: this might be a regression in the kernel as
well. So give 2.6.21-grml a shot or wait until a live grml with
2.6.22-grml is available. If we have the USB hub problem with
>=2.6.21 on your system as well we can continue debugging it in further
detail.

thx && regards,
-mika-

Tests today produced more info.  To simplify, we removed the external hub
(although of course, the motherboard still has hub(s) on-board).  We tried grml
1.0 CD-ROM with USB devices in different combinations.

Udev is not handling the USB scanner.

WITHOUT SCANNER:
We can boot grml 1.0 CD with 1, 2, or 3 USB disks, and all of them work.  Fstab
and /mnt are fine.

WITH SCANNER:
All the USB disks fail to show up in /mnt or fstab.  Syslog shows them all
detected, but they don't get udev events (?).

HARDWARE DATA:
A brand-new Canon LiDE scanner that works perfectly with XSane and USB disks in
grml 0.9.  It is bus-powered, so could be "asleep" at boot time.  Asleep or not,
it should not cause udev failure with USB disks that are awake.

RESULT:
Our best guess right now is a udev rule problem.   What is grml's general udev
situation for USB devices like cameras, printers, scanners, tablets, keyboards,
mice, etc. (things that are not hard disks)?

Test results:  the modules are not the problem.  We blacklisted each one in the
diff list, but all tests gave identical results.  The USB disk was not detected
properly.

But good news, a different test isolated better.  Here is the full USB system
that grml 1.0 CD-ROM botches:

PC --> USB 2.0 Port A --> USB 2.0 Disk
     --> USB 2.0 Port B --> USB 2.0 Hub --> USB Scanner

Removing the USB cable to the hub makes grml 1.0 CD-ROM recognize the USB Disk
(with udev on, no special boot options).

So the bug happens only with the hub plugged in.  Then grml 1.0 fails to see the
USB Disk.  If the hub is unplugged, udev works, and grml sees the USB Disk.

grml 2.6.18 handles everything plugged in, very reliably.  So it is not a USB
hardware glitch.  In fact the hub is practically brand new.

Let us know what other data you want.  We're pretty convinved it's a udev
hotplugging issue - maybe the timeout is too short? - udev in 2.6.20.11 is
having problems with USB hubs.

We can try the 'grml nousb nousb2' test tomorrow if it will help.

* Mark <bts@bts.grml.org> [20070623 01:15]:

> One possibility is that udev loads the right modules in the wrong order.  In
> other words none of the modules are bad, but udev loads them badly.  However I
> will do blacklist tests too.

USB is asynchron in the kernel, there might be some race conditions,
yes. (Some usb modules are loaded in the initrd already, that's why
I suggested use of nousb/nousb2 in my previous mail.)

regards,
-mika-

* Mark <bts@bts.grml.org> [20070623 01:15]:

> Yes, working on that.

Thanks!

> No, I never ran modprobe at all.  Grml loaded all modules by
> itself.

Ok, could you please try booting with 'grml nousb nousb2' then?
Does this help? (Would be really strange, but who knows. ;))
If not: does 'grml noudev nousb nousb2' and running
'modprobe usb-storage' work, right?

regards,
-mika-

One possibility is that udev loads the right modules in the wrong order.  In
other words none of the modules are bad, but udev loads them badly.  However I
will do blacklist tests too.

Yes, working on that.  No, I never ran modprobe at all.  Grml loaded all modules
by itself.

* Mark <bts@bts.grml.org> [20070623 00:01]:

> Don't forget 'sort' before diff.  Here is another file: lsmod of (working)
> 2.6.18 USB boot disk.

> If you have any test suggestions let me know.  The spec sheet details the chipsets.

Thanks.

The notable difference is that the following modules are present
when udev is used but missing when booting with 'noudev':

ac97_bus
af_packet
agpgart
forcedeth
i2c_nforce2
nvidia_agp
parport
parport_pc
pci_hotplug
pcspkr
rtc
shpchp
snd*
soundcore
tsdev

Any chance to test booting with 'grml blacklist=$MODULE' for (some
of) the above modules?  I assume the most promissing candidates are
pci_hotplug, shpchp, parport_pc and i2c_nforce2.

BTW: Did you manually run 'modprobe usb-storage' when booting with
'noudev' or were the usb modules present without any manual
interaction as well?

thx && regards,
-mika-

Don't forget 'sort' before diff.  Here is another file: lsmod of (working)
2.6.18 USB boot disk.

If you have any test suggestions let me know.  The spec sheet details the chipsets.

Here's the two lsmod files, you can diff them.  The 1.0 final CD-ROM boot lines were

fb1280x1024 fbcon=rotate:3
fb1280x1024 fbcon=rotate:3 noudev

EHCI was in both results.

* Mark <bts@bts.grml.org> [20070621 09:45]:

> Booting grml 1.0 final CD-ROM with "noudev" made USB behave normally.  So: the
> bug may be udev rules rather than a kernel regression.

Oh, interesting. And the same usb modules are loaded? Or is
something like ehci-hcd not present?

Any chance to get a diff from lsmod of normal boot and
lsmod of "grml noudev"?

thx && regards,
-mika-

Booting grml 1.0 final CD-ROM with "noudev" made USB behave normally.  So: the
bug may be udev rules rather than a kernel regression.

Mark, did you have the time and possibility to test http://dufo.tugraz.at/
~prokop/grml-kernel/2.6.21-grml/ with your system where you have the usb 
problems?

regards,
-mika-

Well, other distros might not have CONFIG_USB_SUSPEND enabled, so this might be 
the problem. Get the kernel 2.6.21-grml (2.6.20-grml is the one used at grml 
1.0) from http://dufo.tugraz.at/~prokop/grml-kernel/2.6.21-grml/ and give it a 
try. If that still does *not* fix your issue we can do some further 
investigation.

No.  Three reasons why that diagnosis fails.

One, the hardware has worked with all previous grmls.

Two, the same bug cropped up before 0.9-7 with certain dev releases.  There is
some kind of grml design issue involved that reappears now and then.  The
hardware has not changed.

Three, other Linux distros using the same kernel versions have no problems.

The problem could be in udev rules or similar.  I don't know.

Yeah, problems with your usb controller/hub. Nothing I can do for you, sorry. 
Maybe an upcoming kernel version fixes your problem.

1.0rc2 fails the same way.  USB is essentially useless.
Let me know if you need any more data from 1.0rc2.

Try USB hubs as well.  Here's chipset data under working grml:

xyz@testpc ~ % grml-version
0.9 Release Codename Dioptrienotto [2006-12-06]
xyz@testpc ~ % lspci
00:00.0 Host bridge: nVidia Corporation nForce2 AGP (different version?) (rev c1)
00:00.1 RAM memory: nVidia Corporation nForce2 Memory Controller 1 (rev c1)
00:00.2 RAM memory: nVidia Corporation nForce2 Memory Controller 4 (rev c1)
00:00.3 RAM memory: nVidia Corporation nForce2 Memory Controller 3 (rev c1)
00:00.4 RAM memory: nVidia Corporation nForce2 Memory Controller 2 (rev c1)
00:00.5 RAM memory: nVidia Corporation nForce2 Memory Controller 5 (rev c1)
00:01.0 ISA bridge: nVidia Corporation nForce2 ISA Bridge (rev a4)
00:01.1 SMBus: nVidia Corporation nForce2 SMBus (MCP) (rev a2)
00:02.0 USB Controller: nVidia Corporation nForce2 USB Controller (rev a4)
00:02.1 USB Controller: nVidia Corporation nForce2 USB Controller (rev a4)
00:02.2 USB Controller: nVidia Corporation nForce2 USB Controller (rev a4)
00:04.0 Ethernet controller: nVidia Corporation nForce2 Ethernet Controller (rev a1)
00:06.0 Multimedia audio controller: nVidia Corporation nForce2 AC97 Audio
Controler (MCP) (rev a1)
00:08.0 PCI bridge: nVidia Corporation nForce2 External PCI Bridge (rev a3)
00:09.0 IDE interface: nVidia Corporation nForce2 IDE (rev a2)
00:1e.0 PCI bridge: nVidia Corporation nForce2 AGP (rev c1)
01:07.0 Multimedia audio controller: Creative Labs SB Live! EMU10k1 (rev 07)
01:07.1 Input device controller: Creative Labs SB Live! Game Port (rev 07)
02:00.0 VGA compatible controller: nVidia Corporation NV20 [GeForce3] (rev a3)

Thanks for all the information, Mark.

| A similar problem happened once before on a previous grml devel release. 
| (Don't you have any cheap external USB disks for testing?)

For sure, works without any problems here, so seems to be a problem with the 
kernel<->chipset.

I'll test the new grml ISO (providing 2.6.20-grml based on 2.6.20.11, will 
become the grml 1.0-rc1 release) with all the usb disks I can find, stay tuned.

OK.  Here are test results.

This is not a partition or disk problem.  This is a scsi/usb/grml problem.  The
disk is fine.  The partition table is fine.  The filesystems are fine.  The
0.9-7 CD is fine.

A similar problem happened once before on a previous grml devel release.  (Don't
you have any cheap external USB disks for testing?)

On the test PC there are 3 IDE hard drives and 1 external USB drive.  There is
no SCSI device nor any SCSI controller.  Completely SCSI-free.  Just USB 2.

Booting CD-ROM 0.9-7 with no cheatcodes gives this /proc/partitions:

major minor  #blocks  name

   7     0     704712 loop0
   3     0   30018240 hda
   3     1   30018208 hda1
   3    64    8257032 hdb
  22     0   30018240 hdc
  22     1   30018208 hdc1

Booting CD-ROM 0.9-7 with cheatcode "scsi=probe" gives this /proc/partitions:

major minor  #blocks  name

   7     0     704712 loop0
   3     0   30018240 hda
   3     1   30018208 hda1
   3    64    8257032 hdb
  22     0   30018240 hdc
  22     1   30018208 hdc1
   8     0       8192 sda

But cfdisk fails to recognize /dev/sda properly.  Something's wrong with how
grml configures the disk.  Cfdisk works fine under 2.6.18, shows all the
partitions correctly.  I am running grml off the disk right now to type this note.

Anyway, USB should not depend on SCSI probing.  There should always be USB
probing by default.

> Did not check /proc/partitions, but disk shows as sda<1-N> in prior grml

Please check that, as long as it does not appear in /proc/partitions I can't do 
anything about it. I'll upload release-candidate 1 of upcoming grml 1.0 in the 
next ~2-3 days, maybe it works then. But if you have any chance to get output 
of /proc/partitions it would be great, and please check that the ISO and the 
boot medium itself are ok.

thx && regards,
-mika-

Answers
- Did not check /proc/partitions, but disk shows as sda<1-N> in prior grml
- The usb disk has worked with all prior grml versions
- toram is a CD-ROM boot cheatcode
- the disk is basically not present at all in 0.9-7
- partition schema = MS-DOS created by cfdisk

Mark:

I assume the partition does not appear in /proc/partitions as well?

Regarding "I am reporting from the exact same USB disk under grml 2.6.18" -> so 
does the usb disk work with older grml versions, right?

What do you mean with "Same failures running 'toram' and normally."?

"The disk did show up as ATA/IDE under udev disk-by-id.  Obviously that's 
wrong." - well, disk-by-id is just one of several methods udev provides. So it 
would be interesting whether the disk is present at all.

Which partition schema do you use on the disk?

thx && regards,
-mika-

0.9-7 beta CD fails to configure external USB disk.  No /dev/sd* entry.  No
fstab entry.  The udev rules also fail (such as disk-by-label).  Could not run
cfdisk.

Same failures running 'toram' and normally.

The disk did show up as ATA/IDE under udev disk-by-id.  Obviously that's wrong.

The USB disk works fine.  I am reporting from the exact same USB disk under grml
2.6.18.  The test was a CD-ROM boot procedure of course.