============================================= [ INFO: possible recursive locking detected ] 2.6.27.21 #6 --------------------------------------------- dlm_controld/3536 is trying to acquire lock: (&sb->s_type->i_mutex_key#8/2){--..}, at: [] configfs_attach_group+0x43/0x18f [configfs] but task is already holding lock: (&sb->s_type->i_mutex_key#8/2){--..}, at: [] configfs_attach_group+0x43/0x18f [configfs] other info that might help us debug this: 2 locks held by dlm_controld/3536: #0: (&sb->s_type->i_mutex_key#7/1){--..}, at: [] lookup_create+0x1b/0x85 #1: (&sb->s_type->i_mutex_key#8/2){--..}, at: [] configfs_attach_group+0x43/0x18f [configfs] stack backtrace: Pid: 3536, comm: dlm_controld Not tainted 2.6.27.21 #6 [] validate_chain+0x9bd/0xf35 [] __lock_acquire+0x27b/0x932 [] lock_acquire+0x5f/0x77 [] configfs_attach_group+0x43/0x18f [configfs] [] mutex_lock_nested+0xa1/0x25c [] configfs_attach_group+0x43/0x18f [configfs] [] configfs_attach_group+0x43/0x18f [configfs] [] configfs_attach_group+0x43/0x18f [configfs] [] configfs_attach_group+0xe5/0x18f [configfs] [] configfs_mkdir+0x1fb/0x36f [configfs] [] vfs_mkdir+0x9c/0xdf [] _spin_unlock+0x14/0x1c [] sys_mkdirat+0xc5/0xd2 [] trace_hardirqs_on_thunk+0xc/0x10 [] sys_mkdir+0x1f/0x23 [] sysenter_do_call+0x12/0x35 ======================= aoe: e1.0: setting 8704 byte data frames on eth1:0030486543a9 aoe: e2.0: setting 8704 byte data frames on eth1:003048654382 aoe: e2.0: setting 8704 byte data frames on eth1:003048654383 aoe: 0030486543a9 e1.0 vace0 has 2930382333 sectors aoe: 003048654382 e2.0 vace0 has 2930382333 sectors etherd/e1.0: unknown partition table etherd/e2.0: unknown partition table dlm: Using TCP for communications dlm: got connection from 1 dlm: got connection from 5 dlm: connecting to 2 dlm: got connection from 4 dlm: got connection from 2 Trying to join cluster "lock_dlm", "kfki:home" Joined cluster. Now mounting FS... GFS: fsid=kfki:home.3: jid=3: Trying to acquire journal lock... GFS: fsid=kfki:home.3: jid=3: Looking at journal... GFS: fsid=kfki:home.3: jid=3: Done Trying to join cluster "lock_dlm", "kfki:services" Joined cluster. Now mounting FS... GFS: fsid=kfki:services.3: jid=3: Trying to acquire journal lock... GFS: fsid=kfki:services.3: jid=3: Looking at journal... GFS: fsid=kfki:services.3: jid=3: Done GFS: fsid=kfki:home.3: fast statfs start time = 1238196765 GFS: fsid=kfki:services.3: fast statfs start time = 1238196770 SysRq : Show State task PC stack pid father init S f7061b14 0 1 0 f705f110 00000086 00000002 f7061b14 f7061b08 00000000 00000002 c045f0c0 c0461d00 c0461d00 c0461d00 f7061b18 f705f264 c373ed00 f6a8dd00 00000000 ffff9d54 00000000 00000246 c0318668 ffffffff 00000002 00000000 00000000 Call Trace: [] mutex_lock_nested+0x18c/0x25c [] schedule_timeout+0x69/0xa1 [] inotify_poll+0x45/0x4b [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] net_rx_action+0x85/0x1db [] net_rx_action+0x9d/0x1db [] __do_softirq+0xa2/0xf9 [] number+0x25b/0x264 [] restore_nocheck_notrace+0x0/0xe [] lockdep_stats_show+0x299/0x4f0 [] param_set_short+0x9/0x36 [] number+0x2/0x264 [] vsnprintf+0x2dc/0x581 [] validate_chain+0x395/0xf35 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] _spin_lock_irqsave+0x41/0x49 [] do_wait+0x31e/0x342 [] sys_select+0x3f/0x19e [] sysenter_do_call+0x12/0x35 ======================= kthreadd S f7063fac 0 2 0 f705e8a0 00000086 00000002 f7063fac f7063fa0 00000000 f705e8a0 c045f0c0 c0461d00 c0461d00 c0461d00 f7063fb0 f705e9f4 c373ed00 f7896d00 00000246 ffff989b 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] kthreadd+0x14a/0x14f [] kthreadd+0x0/0x14f [] kernel_thread_helper+0x7/0x1c ======================= migration/0 R running 0 3 2 f705e030 00000082 00000086 00000002 00000000 c3734d10 f705e030 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f705e184 c3734d00 f7967800 00000046 00000000 00000002 00000001 00000000 c011cefe 00000046 00000046 c3734d00 Call Trace: [] migration_thread+0x50/0x2cc [] migration_thread+0x147/0x2cc [] migration_thread+0x0/0x2cc [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ksoftirqd/0 S f706bfa8 0 4 2 f7069190 00000092 00000002 f706bfa8 f706bf9c 00000000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 f706bfac f70692e4 c3734d00 f7801800 c040caa0 ffffef69 00000000 c045f100 c0124797 ffffffff c0461b80 00000000 00000000 Call Trace: [] __do_softirq+0xa2/0xf9 [] ksoftirqd+0xee/0xf3 [] ksoftirqd+0x0/0xf3 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= migration/1 S 00000002 0 5 2 f7068920 00000082 00000086 00000002 00000000 c373ed10 f7068920 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f7068a74 c373ed00 f64a8580 00000046 00000000 00000002 00000001 00000000 c011cefe 00000046 00000046 c373ed00 Call Trace: [] migration_thread+0x50/0x2cc [] migration_thread+0x147/0x2cc [] migration_thread+0x0/0x2cc [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ksoftirqd/1 S f7071fa8 0 6 2 f70680b0 00000092 00000002 f7071fa8 f7071f9c 00000000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 f7071fac f7068204 c373ed00 f7c60800 c040caa0 ffffef7b 00000000 c045f100 c0124797 ffffffff c0461b80 00000000 00000000 Call Trace: [] __do_softirq+0xa2/0xf9 [] ksoftirqd+0xee/0xf3 [] ksoftirqd+0x0/0xf3 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= events/0 R running 0 7 2 f70729a0 00000082 00000002 f7079f94 f7079f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f7079f98 f7072af4 c3734d00 f78a5080 00000286 ffffef43 00000000 c0319cc1 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= events/1 R running 0 8 2 f7072130 00000082 00013362 c373baa0 00000002 f707bf90 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f707bf98 f7072284 c373ed00 f7c60800 00000286 ffffef7e 00000000 c0319cc1 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= khelper S 00000001 0 9 2 f707d290 00000082 f707d610 00000001 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f707d3e4 c3734d00 f781a080 00000286 00000001 f702b39c c0319cc1 00000000 f707d290 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kblockd/0 S 00000000 0 47 2 f707ca20 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f707cb74 c3734d00 f7c60300 00000286 00000001 f70a651c c0319cc1 00000000 00000046 00000046 f70a6524 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kblockd/1 S f70cbf94 0 48 2 f707c1b0 00000082 00000002 f70cbf94 f70cbf88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f70cbf98 f707c304 c373ed00 f781a300 00000286 ffff9376 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kacpid S f70cff94 0 49 2 f7083390 00000082 00000002 f70cff94 f70cff88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f70cff98 f70834e4 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kacpi_notify S f70d1f94 0 50 2 f7082b20 00000082 00000002 f70d1f94 f70d1f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f70d1f98 f7082c74 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ata/0 S 00000001 0 130 2 f7120da0 00000082 f7121120 00000001 00000002 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7120ef4 c3734d00 c03e81a0 00000286 00000001 f710dd1c c0319cc1 00000000 f7120da0 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ata/1 S f7125f94 0 131 2 f7121610 00000082 00000002 f7125f94 f7125f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f7125f98 f7121764 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ata_aux S f712df94 0 132 2 f711a4b0 00000082 00000002 f712df94 f712df88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f712df98 f711a604 c3734d00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ksuspend_usbd S 00000001 0 133 2 f711ad20 00000082 f711b0a0 00000001 00000002 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f711ae74 c3734d00 c03e81a0 00000286 00000001 f710da1c c0319cc1 00000000 f711ad20 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= khubd S c373baa0 0 139 2 f711d690 00000082 00000e14 c373baa0 00000002 f7153f0c 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f7153f14 f711d7e4 c373ed00 f6b40300 00000286 ffff8ceb 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] hub_thread+0x87f/0xcef [] trace_hardirqs_on_caller+0x9d/0x113 [] _spin_unlock_irq+0x20/0x23 [] finish_task_switch+0x58/0xab [] schedule+0x228/0x708 [] autoremove_wake_function+0x0/0x35 [] hub_thread+0x0/0xcef [] trace_hardirqs_on_caller+0x9d/0x113 [] hub_thread+0x0/0xcef [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kseriod S c3731aa0 0 142 2 f711ce20 00000082 00000400 c3731aa0 00000002 f7157f80 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f7157f88 f711cf74 c3734d00 f781a580 00000286 ffff8ca9 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] serio_thread+0xa4/0x2c6 [] autoremove_wake_function+0x0/0x35 [] serio_thread+0x0/0x2c6 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= pdflush S 00000002 0 181 2 f7158920 00000082 00000092 00000002 00000000 c03edd8c f7158920 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f7158a74 c3734d00 c03e81a0 00000046 00000000 00000002 00000001 00000000 c0154020 f7158920 c0319c5a c0153fc2 Call Trace: [] pdflush+0x5e/0x19c [] _spin_unlock_irq+0x20/0x23 [] pdflush+0x0/0x19c [] trace_hardirqs_on_caller+0x9d/0x113 [] pdflush+0x0/0x19c [] pdflush+0xb2/0x19c [] trace_hardirqs_on_caller+0x9d/0x113 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= pdflush R running 0 182 2 f7159190 00000082 00000002 f71c3f94 f71c3f88 00000000 f7159190 c045f0c0 c0461d00 c0461d00 c0461d00 f71c3f98 f71592e4 c3734d00 f7c60300 00000046 ffffee63 00000000 00000001 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] pdflush+0x0/0x19c [] pdflush+0xb2/0x19c [] wb_kupdate+0x0/0xe3 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kswapd0 S f71e7f28 0 183 2 f7130030 00000096 00000002 f71e7f28 f71e7f1c 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f71e7f2c f7130184 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] kswapd+0x3fd/0x420 [] schedule+0x228/0x708 [] autoremove_wake_function+0x0/0x35 [] _spin_unlock_irqrestore+0x34/0x39 [] kswapd+0x0/0x420 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= aio/0 S 00000001 0 184 2 f71308a0 00000082 f7130c20 00000001 00000002 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f71309f4 c3734d00 c03e81a0 00000286 00000001 f730dc1c c0319cc1 00000000 f71308a0 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= aio/1 S f732ff94 0 185 2 f7131110 00000082 00000002 f732ff94 f732ff88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f732ff98 f7131264 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kpsmoused S f6addf94 0 335 2 f6aa8a20 00000082 00000002 f6addf94 f6addf88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f6addf98 f6aa8b74 c373ed00 c03e81a0 00000286 ffff8bde 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kstriped S f6b03f94 0 339 2 f6a780b0 00000082 00000002 f6b03f94 f6b03f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f6b03f98 f6a78204 c373ed00 c03e81a0 00000286 ffff8be4 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= edac-poller S f6b0df94 0 343 2 f6aa09a0 00000082 00000002 f6b0df94 f6b0df88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f6b0df98 f6aa0af4 c3734d00 c03e81a0 00000286 ffff8be3 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= scsi_eh_0 S f79f7f68 0 1224 2 f7999410 00000096 00000002 f79f7f68 f79f7f5c 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f79f7f6c f7999564 c3734d00 f7903a80 f780f840 ffff8d11 00000000 f780f800 00000000 ffffffff 00000286 00000000 00000000 Call Trace: [] scsi_error_handler+0x0/0x2de [] scsi_error_handler+0x4b/0x2de [] __wake_up_common+0x46/0x68 [] _spin_unlock_irqrestore+0x34/0x39 [] scsi_error_handler+0x0/0x2de [] trace_hardirqs_on_caller+0x9d/0x113 [] scsi_error_handler+0x0/0x2de [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= scsi_eh_1 S f7c4bf68 0 1225 2 f7a0a330 00000096 00000002 f7c4bf68 f7c4bf5c 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f7c4bf6c f7a0a484 c3734d00 f78a5800 f7c53840 ffff8d7c 00000000 f7c53800 00000000 ffffffff 00000286 00000000 00000000 Call Trace: [] scsi_error_handler+0x0/0x2de [] scsi_error_handler+0x4b/0x2de [] __wake_up_common+0x46/0x68 [] _spin_unlock_irqrestore+0x34/0x39 [] scsi_error_handler+0x0/0x2de [] trace_hardirqs_on_caller+0x9d/0x113 [] scsi_error_handler+0x0/0x2de [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= scsi_eh_2 S f7c53000 0 1226 2 f7a1b510 00000096 f7c53050 f7c53000 00000000 00000046 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f7a1b510 f7a1b664 c373ed00 f7982800 f7c53040 00000286 00000286 f7c53000 00000000 c025645c 00000286 00000282 f7c53000 Call Trace: [] __scsi_iterate_devices+0x49/0x62 [] scsi_error_handler+0x0/0x2de [] scsi_error_handler+0x0/0x2de [] scsi_error_handler+0x4b/0x2de [] __wake_up_common+0x46/0x68 [] _spin_unlock_irqrestore+0x34/0x39 [] scsi_error_handler+0x0/0x2de [] trace_hardirqs_on_caller+0x9d/0x113 [] scsi_error_handler+0x0/0x2de [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= scsi_eh_3 S f78fd800 0 1227 2 f6a79190 00000096 f78fd850 f78fd800 00000000 00000046 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f6a79190 f6a792e4 c373ed00 f6b40300 f78fd840 00000286 00000286 f78fd800 00000000 c025645c 00000286 00000282 f78fd800 Call Trace: [] __scsi_iterate_devices+0x49/0x62 [] scsi_error_handler+0x0/0x2de [] scsi_error_handler+0x0/0x2de [] scsi_error_handler+0x4b/0x2de [] __wake_up_common+0x46/0x68 [] _spin_unlock_irqrestore+0x34/0x39 [] scsi_error_handler+0x0/0x2de [] trace_hardirqs_on_caller+0x9d/0x113 [] scsi_error_handler+0x0/0x2de [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= md2_raid1 S f7c6ff68 0 1391 2 f7a96b20 00000096 00000002 f7c6ff68 f7c6ff5c 00000000 f29c2230 c045f0c0 c0461d00 c0461d00 c0461d00 f7c6ff6c f7a96c74 c3734d00 f7967080 c01f4716 ffffef43 00000000 f79d7a18 00000001 ffffffff 00000046 00000000 00000000 Call Trace: [] __delay+0x6/0x7 [] md_thread+0x0/0xda [] schedule_timeout+0x69/0xa1 [] md_thread+0xb0/0xda [] autoremove_wake_function+0x0/0x35 [] md_thread+0x0/0xda [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= md2_resync R running 0 1392 2 f7a74130 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a74284 c3734d00 f781a300 c04b9e00 00000286 c04b9e00 00000286 00000286 f7d57e28 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] delay_tsc+0x2a/0x47 [] process_timeout+0x0/0x5 [] msleep_interruptible+0x22/0x2f [] sync_request+0x5c7/0x795 [] is_mddev_idle+0x103/0x10f [] is_mddev_idle+0x9/0x10f [] sync_request+0x0/0x795 [] md_do_sync+0x320/0xdb7 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] _spin_unlock_irq+0x20/0x23 [] md_thread+0x0/0xda [] trace_hardirqs_on_caller+0x9d/0x113 [] md_thread+0x0/0xda [] md_thread+0x22/0xda [] _spin_unlock_irqrestore+0x34/0x39 [] md_thread+0x0/0xda [] trace_hardirqs_on_caller+0x9d/0x113 [] md_thread+0x0/0xda [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= md1_raid1 S f7445f68 0 1401 2 f79b1510 00000096 00000002 f7445f68 f7445f5c 00000000 f7073210 c045f0c0 c0461d00 c0461d00 c0461d00 f7445f6c f79b1664 c373ed00 f64d7d00 c01f4716 ffffd981 00000000 f7463418 00000001 ffffffff 00000046 00000000 00000000 Call Trace: [] __delay+0x6/0x7 [] md_thread+0x0/0xda [] schedule_timeout+0x69/0xa1 [] md_thread+0xb0/0xda [] autoremove_wake_function+0x0/0x35 [] md_thread+0x0/0xda [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= md0_raid1 S f7957f68 0 1409 2 f79f21b0 00000096 00000002 f7957f68 f7957f5c 00000000 f7073210 c045f0c0 c0461d00 c0461d00 c0461d00 f7957f6c f79f2304 c373ed00 f64d7d00 c01f4716 ffffd981 00000000 f7490d98 00000001 ffffffff 00000046 00000000 00000000 Call Trace: [] __delay+0x6/0x7 [] md_thread+0x0/0xda [] schedule_timeout+0x69/0xa1 [] md_thread+0xb0/0xda [] autoremove_wake_function+0x0/0x35 [] md_thread+0x0/0xda [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kdmflush S 00000001 0 1420 2 f79d0630 00000082 f79d09b0 00000001 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79d0784 c373ed00 f7801d00 00000286 00000001 f788629c c0319cc1 00000000 f79d0630 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kdmflush S 00000001 0 1426 2 f71d0aa0 00000082 f71d0e20 00000001 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f71d0bf4 c373ed00 f7801d00 00000286 00000001 f7c5e51c c0319cc1 00000000 f71d0aa0 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kdmflush S 00000001 0 1432 2 f7a75210 00000082 f7a75590 00000001 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a75364 c373ed00 f6b40a80 00000286 00000001 f74cf79c c0319cc1 00000000 f7a75210 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kjournald R running 0 1514 2 f7a264b0 00000082 00000046 f78c84d8 00000001 f78c8510 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a26604 c3734d00 f7967080 00000286 00000001 f78c8510 c0319cc1 00000000 00000046 00000246 00000246 f78c8414 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] kjournald+0x1bc/0x1c5 [] autoremove_wake_function+0x0/0x35 [] kjournald+0x0/0x1c5 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= udevd S f759db14 0 1675 1 f7a43110 00000086 00000002 f759db14 f759db08 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f759db18 f7a43264 c373ed00 f7896d00 f78d8800 ffff989b 00000000 00000000 00000002 ffffffff f78d8800 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] pipe_poll+0x24/0x86 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] vsnprintf+0x2dc/0x581 [] number+0x25b/0x264 [] add_to_page_cache_locked+0x46/0x8f [] vsnprintf+0x2dc/0x581 [] _spin_lock+0x31/0x3c [] handle_mm_fault+0x4ea/0x8a4 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] _spin_unlock+0x14/0x1c [] do_wp_page+0x2a2/0x537 [] _spin_lock_irqsave+0x41/0x49 [] do_wait+0x1e7/0x342 [] do_page_fault+0xa5/0x848 [] sys_select+0x3f/0x19e [] sysenter_do_call+0x12/0x35 ======================= gfs2_scand R running 0 2420 2 f7a83290 00000092 00000046 00000000 00000002 00000001 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a833e4 c373ed00 f64a8a80 f706e000 00000282 f706e000 00000282 00000282 f75cbfac f706e000 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] gfs2_scand+0x0/0x63 [gfs2] [] schedule_timeout+0x44/0xa1 [] _read_unlock+0x14/0x1c [] process_timeout+0x0/0x5 [] gfs2_scand+0x56/0x63 [gfs2] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= glock_workque S c3731aa0 0 2422 2 f7a82a20 00000082 00000afb c3731aa0 00000002 f7585f90 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f7585f98 f7a82b74 c3734d00 f64d7d00 00000286 ffff9049 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= glock_workque S 00000001 0 2423 2 f7120530 00000082 f71208b0 00000001 00000002 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7120684 c373ed00 f78ff080 00000286 00000001 f501be9c c0319cc1 00000000 f7120530 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ksnapd S 00000001 0 2536 2 f79db110 00000082 f79db490 00000001 00000002 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79db264 c373ed00 f7963a80 00000286 00000001 f501b89c c0319cc1 00000000 f79db110 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kjournald S f6533f84 0 2623 2 f6a48ca0 00000082 00000002 f6533f84 f6533f78 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f6533f88 f6a48df4 c3734d00 f64a8a80 f6a48ca0 ffff9450 00000000 f5065d10 c013d220 ffffffff 00000246 00000000 00000000 Call Trace: [] mark_held_locks+0x62/0x73 [] kjournald+0x1bc/0x1c5 [] autoremove_wake_function+0x0/0x35 [] kjournald+0x0/0x1c5 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kjournald S f50b7f84 0 2624 2 f6a70030 00000082 00000002 f50b7f84 f50b7f78 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f50b7f88 f6a70184 c3734d00 f7c60a80 00000286 ffffeba1 00000000 c0319cc1 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] kjournald+0x1bc/0x1c5 [] autoremove_wake_function+0x0/0x35 [] kjournald+0x0/0x1c5 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kjournald R running 0 2625 2 f71e1410 00000082 00000046 f506bcd8 00000001 f506bd10 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f71e1564 c3734d00 f7967080 00000286 00000001 f506bd10 c0319cc1 00000000 00000046 00000246 00000246 f506bc14 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] kjournald+0x1bc/0x1c5 [] autoremove_wake_function+0x0/0x35 [] kjournald+0x0/0x1c5 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= getty S f50e3eb4 0 2962 1 f6ab8b20 00000082 00000002 f50e3eb4 f50e3ea8 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f50e3eb8 f6ab8c74 c373ed00 f78a5300 00000000 ffff9238 00000000 00000046 f78e797c ffffffff f7538000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] mark_held_locks+0x62/0x73 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= getty S 0038a000 0 2963 1 f7a6b190 00000082 00000001 0038a000 00000000 c013eb96 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a6b2e4 c373ed00 f7982800 00000000 f7a6b190 c03182b9 00000046 f65b897c f65b8800 f744d000 00000046 00000000 Call Trace: [] __lock_acquire+0x27b/0x932 [] mutex_lock_interruptible_nested+0x192/0x2b7 [] schedule_timeout+0x69/0xa1 [] mark_held_locks+0x62/0x73 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= login S 00000002 0 2965 1 f7a2eda0 00000082 f7a2eda0 00000002 00000000 c040c990 f7a2eda0 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f7a2eef4 c3734d00 f64d7d00 00000000 00000002 00000002 00000000 00000000 c01228e1 00000246 00000246 c040c980 Call Trace: [] do_wait+0x96/0x342 [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= getty S 0038a000 0 2966 1 f7a26d20 00000082 00000001 0038a000 00000000 c013eb96 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a26e74 c3734d00 f7967d00 00000000 f7a26d20 c03182b9 00000046 f748897c f7488800 f6461000 00000046 00000000 Call Trace: [] __lock_acquire+0x27b/0x932 [] mutex_lock_interruptible_nested+0x192/0x2b7 [] schedule_timeout+0x69/0xa1 [] mark_held_locks+0x62/0x73 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= login S 00000002 0 2969 1 f7a02b20 00000082 f7a02b20 00000002 00000000 c040c990 f7a02b20 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f7a02c74 c3734d00 f7801300 00000000 00000002 00000002 00000000 00000000 c01228e1 00000246 00000246 c040c980 Call Trace: [] do_wait+0x96/0x342 [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= getty S 0038a000 0 2970 1 f78e45b0 00000082 00000001 0038a000 00000000 c013eb96 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f78e4704 c3734d00 f6505d00 00000000 f78e45b0 c03182b9 00000046 f7c0b17c f7c0b000 f6460000 00000046 00000000 Call Trace: [] __lock_acquire+0x27b/0x932 [] mutex_lock_interruptible_nested+0x192/0x2b7 [] schedule_timeout+0x69/0xa1 [] mark_held_locks+0x62/0x73 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= syslogd R running 0 3016 1 f71d8b20 00000086 f75be31c f71d8b20 00000002 00000000 f6d175ac c045f0c0 c0461d00 c0461d00 c0461d00 00000020 f71d8c74 c3734d00 f7896300 00000000 00000000 c030b41c 00000246 00000246 f75be30c f75be30c 00000000 00000020 Call Trace: [] unix_peer_get+0x11/0x2b [] _spin_unlock+0x14/0x1c [] schedule_timeout+0x69/0xa1 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] __ext3_get_inode_loc+0xba/0x307 [] __ext3_journal_dirty_metadata+0x16/0x3b [] _spin_lock_irqsave+0x41/0x49 [] journal_stop+0x134/0x1c3 [] ext3_ordered_write_end+0xf0/0x14a [] generic_file_buffered_write+0x1a6/0x677 [] mnt_drop_write+0x1d/0xd8 [] _spin_lock+0x31/0x3c [] pipe_write+0x5c/0x450 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] pipe_write+0x0/0x450 [] do_sync_readv_writev+0xcb/0x107 [] mntput_no_expire+0x18/0xef [] autoremove_wake_function+0x0/0x35 [] copy_from_user+0x2d/0x59 [] rw_copy_check_uvector+0x64/0xf4 [] sys_select+0x3f/0x19e [] sigprocmask+0x68/0xc3 [] sys_rt_sigprocmask+0xbf/0xd4 [] sysenter_do_call+0x12/0x35 ======================= dd S f6489f04 0 3049 1 f6a605b0 00000086 00000002 f6489f04 f6489ef8 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f6489f08 f6a60704 c373ed00 f78ff580 00000282 ffff93bc 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] do_syslog+0x278/0x399 [] autoremove_wake_function+0x0/0x35 [] kmsg_read+0x0/0x36 [] proc_reg_read+0x58/0x79 [] proc_reg_read+0x0/0x79 [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= klogd S 00000000 0 3050 1 f6aa9290 00000092 f6aa9628 00000000 f6aa9610 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f6aa93e4 c373ed00 f7967580 f6aa9290 00000000 00000000 f6aa9290 c0318467 f75e8fd8 c013d34e f75e8fd4 f75e8ff4 Call Trace: [] __mutex_unlock_slowpath+0x89/0xf9 [] trace_hardirqs_on_caller+0x9d/0x113 [] pipe_wait+0x50/0x72 [] autoremove_wake_function+0x0/0x35 [] pipe_read+0xe3/0x3fe [] do_sync_read+0xd2/0x10e [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] do_sync_read+0x0/0x10e [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= mysqld_safe S f5dcff24 0 3150 1 f6ab9390 00000082 00000002 f5dcff24 f5dcff18 00000000 f6ab9390 c045f0c0 c0461d00 c0461d00 c0461d00 f5dcff28 f6ab94e4 c3734d00 f7832a80 00000000 ffff9273 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= mysqld S f50dfb14 0 3192 3150 f6aa1210 00000086 00000002 f50dfb14 00000002 00000000 f540132c c045f0c0 c0461d00 c0461d00 c0461d00 f50dfb18 f6aa1364 c3734d00 f781a300 00000000 00000046 00000000 00000002 00000001 f540131c 00000282 f7854280 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] schedule_timeout+0x69/0xa1 [] unix_poll+0x1a/0xab [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] __getblk+0x36/0x2a7 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] ext3_getblk+0xb9/0x1c5 [] ll_rw_block+0x42/0x10e [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] ip_setsockopt+0x113/0xbad [] local_bh_enable_ip+0x79/0xb5 [] ip_setsockopt+0x113/0xbad [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] enqueue_task_fair+0xa2/0xb4 [] wake_up_new_task+0x87/0xa0 [] do_fork+0xce/0x247 [] sys_select+0x3f/0x19e [] sysenter_do_call+0x12/0x35 ======================= mysqld S f5de7df8 0 3200 3150 f7a42030 00000096 00000002 f5de7df8 f5de7dec 00000000 f7a42030 c045f0c0 c0461d00 c0461d00 c0461d00 f5de7dfc f7a42184 c373ed00 f781a300 f5de7e3c ffff9298 00000000 08e3d9f8 c0319cc1 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] futex_wait+0x318/0x396 [] __lock_acquire+0x27b/0x932 [] __lock_acquire+0x27b/0x932 [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] enqueue_task_fair+0xa2/0xb4 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] rwsem_wake+0x4d/0x116 [] call_rwsem_wake+0xa/0xc [] up_read+0x26/0x2a [] sys_futex+0x86/0x111 [] trace_hardirqs_on_thunk+0xc/0x10 [] trace_hardirqs_on_caller+0x9d/0x113 [] sysenter_do_call+0x12/0x35 ======================= mysqld S 00000002 0 3201 3150 f7a27590 00000096 f7a27590 00000002 00000000 c082d374 f7a27590 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f7a276e4 c3734d00 f781a300 f5081e3c 00000282 00000000 08e3da68 c0319cc1 00000046 00000046 f5081e3c 00000282 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] futex_wait+0x318/0x396 [] try_to_wake_up+0xa8/0x137 [] _spin_unlock+0x14/0x1c [] futex_requeue+0xdb/0x251 [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] find_get_pages_tag+0x11/0x140 [] ext3_write_inode+0x0/0x34 [] mapping_tagged+0x58/0x60 [] pagevec_lookup_tag+0x24/0x2c [] wait_on_page_writeback_range+0x5f/0x104 [] _spin_unlock+0x14/0x1c [] sync_inode+0x25/0x2a [] sys_futex+0x86/0x111 [] do_fsync+0x73/0x92 [] sysenter_do_call+0x12/0x35 ======================= mysqld S f79e8d20 0 3202 3150 f79e89a0 00000096 f79e89a0 f79e8d20 00000001 c082c6e8 f79e89a0 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f79e8af4 c3734d00 f781a300 f75dde3c 00000282 00000000 08e3dad8 c0319cc1 f79e89a0 c0319d8f 00000000 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] futex_wait+0x318/0x396 [] __alloc_pages_internal+0xb6/0x41f [] validate_chain+0x395/0xf35 [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] __lock_acquire+0x27b/0x932 [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] sys_futex+0x86/0x111 [] vfs_read+0xe3/0x11b [] trace_hardirqs_on_thunk+0xc/0x10 [] trace_hardirqs_on_caller+0x9d/0x113 [] sysenter_do_call+0x12/0x35 ======================= mysqld S 00000002 0 3203 3150 f7ac7610 00000096 f7ac7610 00000002 00000000 c082cea4 f7ac7610 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f7ac7764 c3734d00 f781a300 f64e3e3c 00000282 00000000 08e3db48 c0319cc1 00000046 00000046 f64e3e3c 00000282 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] futex_wait+0x318/0x396 [] generic_file_aio_write+0x55/0xd8 [] generic_file_aio_write+0x73/0xd8 [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] sys_futex+0x86/0x111 [] vfs_write+0xe5/0x118 [] sysenter_do_call+0x12/0x35 ======================= mysqld R running 0 3205 3150 f79e9210 00000086 00000002 f7985b14 f7985b08 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f7985b18 f79e9364 c3734d00 f781a300 c04b9e00 ffffef43 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __lock_acquire+0x27b/0x932 [] validate_chain+0x395/0xf35 [] mark_held_locks+0x62/0x73 [] get_page_from_freelist+0x304/0x48d [] trace_hardirqs_on_caller+0x9d/0x113 [] __pollwait+0x0/0xba [] __lock_acquire+0x27b/0x932 [] enqueue_task_fair+0xa2/0xb4 [] try_to_wake_up+0xa8/0x137 [] _spin_lock_irqsave+0x41/0x49 [] del_timer+0x4b/0x53 [] scsi_delete_timer+0xb/0x1b [] queue_work_on+0x34/0x43 [] enqueue_task_fair+0xa2/0xb4 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] _spin_lock_irqsave+0x41/0x49 [] _spin_lock_irqsave+0x41/0x49 [] scsi_next_command+0x25/0x2f [] scsi_end_request+0x52/0x75 [] scsi_io_completion+0x87/0x3de [] _spin_lock+0x31/0x3c [] sys_select+0xd6/0x19e [] sysenter_do_call+0x12/0x35 ======================= mysqld R running 0 3206 3150 f7a1a430 00000086 00000002 f6583b14 f6583b08 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f6583b18 f7a1a584 c3734d00 f781a300 c04b9e00 ffffeed6 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] try_to_wake_up+0xa8/0x137 [] autoremove_wake_function+0x14/0x35 [] __wake_up_common+0x46/0x68 [] queue_work_on+0x34/0x43 [] tcp_rcv_established+0x341/0x70b [] tcp_v4_do_rcv+0x94/0x1af [] _spin_unlock+0x14/0x1c [] tcp_v4_rcv+0x498/0x5be [] ip_local_deliver_finish+0x10d/0x15f [] ip_local_deliver_finish+0x3f/0x15f [] nommu_map_single+0x0/0x63 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] e1000_clean_rx_irq+0x301/0x41a [e1000] [] e1000_clean_rx_irq+0x0/0x41a [e1000] [] e1000_clean+0x450/0x5f9 [e1000] [] _spin_lock_irqsave+0x41/0x49 [] scsi_end_request+0x52/0x75 [] net_rx_action+0x85/0x1db [] net_rx_action+0x9d/0x1db [] sys_select+0xd6/0x19e [] sysenter_do_call+0x12/0x35 ======================= mysqld S 00000002 0 3207 3150 f6a61690 00000096 f6a61690 00000002 00000000 c082d348 f6a61690 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f6a617e4 c3734d00 f781a300 f50d7e3c 00000282 00000000 08af80b0 c0319cc1 00000046 00000046 f50d7e3c 00000282 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] futex_wait+0x318/0x396 [] _spin_lock_irqsave+0x41/0x49 [] futex_wake+0x52/0xe1 [] _spin_unlock+0x14/0x1c [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] scsi_next_command+0x25/0x2f [] scsi_io_completion+0x87/0x3de [] _spin_lock+0x31/0x3c [] sys_futex+0x86/0x111 [] sysenter_do_call+0x12/0x35 ======================= mysqld S f5117ecc 0 3208 3150 f79f2a20 00000082 00000002 f5117ecc f5117ec0 00000000 f7073210 c045f0c0 c0461d00 c0461d00 c0461d00 f5117ed0 f79f2b74 c373ed00 f781a300 00000000 ffffec7b 00000000 c373bf30 f79f2a20 ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] dequeue_signal+0x3c/0x12a [] sys_rt_sigtimedwait+0x87/0x285 [] sys_rt_sigtimedwait+0x145/0x285 [] sigprocmask+0x1f/0xc3 [] _spin_unlock_irq+0x20/0x23 [] sigprocmask+0x68/0xc3 [] sys_rt_sigprocmask+0xbf/0xd4 [] sysenter_do_call+0x12/0x35 ======================= mysqld S f651fd94 0 3209 3150 f79a43b0 00000082 00000002 f651fd94 f651fd88 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f651fd98 f79a4504 c3734d00 f781a300 00000002 ffffb5a6 00000000 f5dcd100 f5dcd12c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] sk_wait_data+0x54/0xa1 [] sk_wait_data+0x69/0xa1 [] autoremove_wake_function+0x0/0x35 [] tcp_recvmsg+0x42c/0x7ff [] sock_common_recvmsg+0x3e/0x54 [] sock_aio_read+0xdb/0xef [] do_sync_read+0xd2/0x10e [] autoremove_wake_function+0x0/0x35 [] vfs_read+0x114/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= mysqld S 00000002 0 3210 3150 f79b84b0 00000096 f79b84b0 00000002 00000000 c082da80 f79b84b0 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f79b8604 c3734d00 f781a300 f5017e3c 00000282 00000000 08edccf4 c0319cc1 00000046 00000046 f5017e3c 00000282 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] futex_wait+0x318/0x396 [] __generic_file_aio_write_nolock+0x285/0x4df [] generic_file_aio_write+0x55/0xd8 [] generic_file_aio_write+0x73/0xd8 [] futex_wake+0x52/0xe1 [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] sys_futex+0x86/0x111 [] sys_write+0x68/0x6a [] sysenter_do_call+0x12/0x35 ======================= mysqld S 00000002 0 4053 3150 f6a49510 00000096 f6a49510 00000002 00000000 c082c6bc f6a49510 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f6a49664 c3734d00 f781a300 f209fe3c 00000282 00000000 0875bba0 c0319cc1 00000046 00000046 f209fe3c 00000282 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] futex_wait+0x318/0x396 [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] autoremove_wake_function+0x0/0x35 [] sys_futex+0x86/0x111 [] sys_read+0x68/0x6a [] sysenter_do_call+0x12/0x35 ======================= logger S 00000000 0 3193 3150 f6ab0230 00000092 f6ab05c8 00000000 f6ab05b0 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f6ab0384 c3734d00 f7896580 f6ab0230 00000000 00000000 f6ab0230 c0318467 f6ddd768 c013d34e f6ddd764 f6ddd784 Call Trace: [] __mutex_unlock_slowpath+0x89/0xf9 [] trace_hardirqs_on_caller+0x9d/0x113 [] pipe_wait+0x50/0x72 [] autoremove_wake_function+0x0/0x35 [] pipe_read+0xe3/0x3fe [] __lock_acquire+0x27b/0x932 [] do_sync_read+0xd2/0x10e [] autoremove_wake_function+0x0/0x35 [] finish_task_switch+0x26/0xab [] schedule+0x228/0x708 [] do_sync_read+0x0/0x10e [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= master R running 0 3362 1 f6aa0130 00000086 00000002 f5173f1c f5173f10 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f5173f20 f6aa0284 c3734d00 f6a8d800 c04b9e00 ffffef68 00000000 00000282 00000282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= mdadm R running 0 3403 1 f79fd310 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79fd464 c3734d00 f7801d00 c04b9e00 00000286 c04b9e00 00000286 00000286 f7499b38 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] mark_held_locks+0x62/0x73 [] get_page_from_freelist+0x304/0x48d [] trace_hardirqs_on_caller+0x9d/0x113 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] enqueue_task_fair+0xa2/0xb4 [] _spin_lock_irqsave+0x41/0x49 [] md_wakeup_thread+0x25/0x29 [] md_ioctl+0x10c/0x1082 [] __d_lookup+0xbf/0x14e [] __d_lookup+0xe9/0x14e [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] do_open+0xc5/0x2b7 [] blkdev_open+0x0/0x53 [] blkdev_open+0x25/0x53 [] __dentry_open+0x176/0x20a [] nameidata_to_filp+0x35/0x3f [] mutex_lock_nested+0x18c/0x25c [] file_kill+0x13/0x2d [] sys_select+0xd6/0x19e [] filp_close+0x3e/0x62 [] sysenter_do_call+0x12/0x35 ======================= ccsd S f6509b14 0 3461 1 f79fcaa0 00000086 00000002 f6509b14 00000002 00000000 f55ef0ac c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79fcbf4 c373ed00 f756c300 00000000 00000046 00000000 00000002 00000001 f55ef09c 00000282 f7910280 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] schedule_timeout+0x69/0xa1 [] unix_poll+0x1a/0xab [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] sched_clock_cpu+0x10f/0x158 [] update_curr+0x40/0x52 [] try_to_wake_up+0xa8/0x137 [] autoremove_wake_function+0x14/0x35 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] _spin_unlock_irq+0x20/0x23 [] finish_task_switch+0x58/0xab [] finish_task_switch+0x26/0xab [] schedule+0x228/0x708 [] sys_select+0x3f/0x19e [] filp_close+0x3e/0x62 [] sysenter_do_call+0x12/0x35 ======================= ccsd S f5105b14 0 3462 1 f79d0ea0 00000086 00000002 f5105b14 f5105b08 00000000 f55f7aac c045f0c0 c0461d00 c0461d00 c0461d00 f5105b18 f79d0ff4 c3734d00 f756c300 00000000 ffff9486 00000000 00000002 00000001 ffffffff 00000282 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] unix_poll+0x1a/0xab [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __lock_acquire+0x27b/0x932 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] validate_chain+0x395/0xf35 [] check_noncircular+0xd/0xe7 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] sched_clock_cpu+0x10f/0x158 [] update_curr+0x40/0x52 [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] __lock_acquire+0x27b/0x932 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] validate_chain+0x395/0xf35 [] sock_aio_read+0xdb/0xef [] validate_chain+0x395/0xf35 [] sockfd_lookup_light+0x24/0x41 [] __lock_acquire+0x27b/0x932 [] _spin_unlock_irq+0x20/0x23 [] trace_hardirqs_on_caller+0x9d/0x113 [] _spin_unlock_irq+0x20/0x23 [] finish_task_switch+0x58/0xab [] finish_task_switch+0x26/0xab [] schedule+0x228/0x708 [] dnotify_parent+0x1f/0x5f [] sys_select+0x3f/0x19e [] trace_hardirqs_on_thunk+0xc/0x10 [] trace_hardirqs_on_caller+0x9d/0x113 [] sysenter_do_call+0x12/0x35 ======================= ntpd S 00000000 0 3475 1 f711c5b0 00200082 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f711c704 c3734d00 f78ffa80 c04b9e00 00200282 c04b9e00 00200282 00200282 f5025c18 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] udpv6_recvmsg+0x1ac/0x335 [ipv6] [] local_bh_enable_ip+0x79/0xb5 [] udpv6_recvmsg+0x1ac/0x335 [ipv6] [] enqueue_task_fair+0xa2/0xb4 [] _spin_lock_irqsave+0x41/0x49 [] _spin_lock_irqsave+0x41/0x49 [] scsi_next_command+0x25/0x2f [] scsi_end_request+0x52/0x75 [] scsi_io_completion+0x87/0x3de [] _spin_lock+0x31/0x3c [] dnotify_parent+0x1f/0x5f [] vfs_write+0xe5/0x118 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= ntpd S 00000000 0 3476 1 f6a708a0 00200082 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f6a709f4 c3734d00 f64d7800 c04b9e00 00200282 c04b9e00 00200282 00200282 f7457c18 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] do_wp_page+0x2a2/0x537 [] handle_mm_fault+0x1b3/0x8a4 [] _spin_lock+0x31/0x3c [] handle_mm_fault+0x4ea/0x8a4 [] task_rq_lock+0x2e/0x55 [] _spin_lock+0x31/0x3c [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] _read_unlock+0x14/0x1c [] unix_stream_sendmsg+0x1c3/0x32b [] sock_aio_write+0xcb/0xe8 [] find_lock_page+0x25/0x55 [] sock_aio_write+0x0/0xe8 [] do_sync_readv_writev+0xcb/0x107 [] __do_fault+0x192/0x3f0 [] dnotify_parent+0x1f/0x5f [] do_readv_writev+0x118/0x179 [] sock_aio_write+0x0/0xe8 [] up_read+0x14/0x2a [] do_page_fault+0x156/0x848 [] vfs_writev+0x3c/0x50 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= sshd S 00000000 0 3492 1 f79da030 00200086 00000042 00000000 00000042 00000002 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 00000032 f79da184 c373ed00 f756c580 f791b980 00000020 00200046 00000000 00000002 00000001 f5afd59c 00200282 f791b980 Call Trace: [] schedule_timeout+0x69/0xa1 [] tcp_poll+0x17/0x11e [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] ndisc_recv_ns+0x226/0x563 [ipv6] [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] __lock_acquire+0x27b/0x932 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] mark_held_locks+0x62/0x73 [] get_page_from_freelist+0x304/0x48d [] trace_hardirqs_on_caller+0x9d/0x113 [] __lock_acquire+0x27b/0x932 [] mark_held_locks+0x62/0x73 [] _spin_unlock_irq+0x20/0x23 [] trace_hardirqs_on_caller+0x9d/0x113 [] _spin_unlock_irq+0x20/0x23 [] add_to_page_cache_locked+0x70/0x8f [] kmap_atomic+0x1c/0x21 [] shmem_getpage+0x4e9/0x661 [] kmap_atomic+0x1c/0x21 [] __lock_acquire+0x27b/0x932 [] __lock_acquire+0x27b/0x932 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] __do_fault+0x2bc/0x3f0 [] __do_fault+0x192/0x3f0 [] trace_hardirqs_on_caller+0x9d/0x113 [] handle_mm_fault+0x176/0x8a4 [] do_page_fault+0xa5/0x848 [] sys_select+0x3f/0x19e [] trace_hardirqs_on_thunk+0xc/0x10 [] do_page_fault+0x0/0x848 [] trace_hardirqs_on_caller+0x9d/0x113 [] sysenter_do_call+0x12/0x35 ======================= pickup S f6581f1c 0 3503 3362 f79e40b0 00200086 00000002 f6581f1c f6581f10 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f6581f20 f79e4204 c3734d00 f78ffd00 c04b9e00 ffffd9c4 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= qmgr R running 0 3504 3362 f71d1310 00200086 00000002 f513ff1c f513ff10 00000000 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f513ff20 f71d1464 c373ed00 f781a580 f706e000 ffffef69 00000000 00200282 00200282 ffffffff f706e000 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] vfs_ioctl+0x1f/0x6d [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= aisexec R running 0 3510 1 f79a4c20 00000082 00000400 c373baa0 00000002 f645fbf0 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f645fbf8 f79a4d74 c373ed00 f64a8a80 f706e000 ffffef78 00000000 00000282 00000282 ffffffff f706e000 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] read_tsc+0x6/0x28 [] getnstimeofday+0x44/0x100 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f750fbf4 0 3511 1 f7aa4c20 00000082 00000002 f750fbf4 f750fbe8 00000000 f7aa4c20 c045f0c0 c0461d00 c0461d00 c0461d00 f750fbf8 f7aa4d74 c373ed00 f64a8a80 00000080 ffff93a4 00000000 00000002 c0555fa0 ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] mark_held_locks+0x62/0x73 [] get_page_from_freelist+0x304/0x48d [] trace_hardirqs_on_caller+0x9d/0x113 [] __lock_acquire+0x27b/0x932 [] validate_chain+0x395/0xf35 [] mark_held_locks+0x62/0x73 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] find_usage_backwards+0xc/0xce [] find_usage_backwards+0xc/0xce [] find_usage_backwards+0xc/0xce [] __lock_acquire+0x27b/0x932 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] __do_fault+0x2bc/0x3f0 [] __do_fault+0x192/0x3f0 [] handle_mm_fault+0x176/0x8a4 [] do_page_fault+0xa5/0x848 [] up_read+0x14/0x2a [] do_page_fault+0x156/0x848 [] trace_hardirqs_on_thunk+0xc/0x10 [] do_page_fault+0x0/0x848 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f4835df8 0 3513 1 f6a51590 00000096 00000002 f4835df8 f4835dec 00000000 f6a51590 c045f0c0 c0461d00 c0461d00 c0461d00 f4835dfc f6a516e4 c373ed00 f64a8a80 f4835e3c ffff9468 00000000 093b44c0 c0319cc1 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] futex_wait+0x318/0x396 [] __generic_file_aio_write_nolock+0x285/0x4df [] generic_file_aio_write+0x73/0xd8 [] futex_wake+0x52/0xe1 [] validate_chain+0x395/0xf35 [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] __lock_acquire+0x27b/0x932 [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] sys_futex+0x86/0x111 [] trace_hardirqs_on_thunk+0xc/0x10 [] trace_hardirqs_on_caller+0x9d/0x113 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f510c998 0 3529 1 f79b9590 00000086 00000246 f510c998 00000000 f510e000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f79b96e4 c373ed00 f64a8a80 00000000 f510c988 00000000 00000000 f510e000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] schedule+0x228/0x708 [] futex_wait+0xb6/0x396 [] _spin_unlock+0x14/0x1c [] futex_wait+0xeb/0x396 [] futex_wake+0x52/0xe1 [] _spin_unlock+0x14/0x1c [] futex_wake+0xc0/0xe1 [] do_futex+0x7a/0x951 [] __lock_acquire+0x27b/0x932 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f510c898 0 3541 1 f6a71110 00000086 00000246 f510c898 00000000 f5106000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f6a71264 c3734d00 f64a8a80 00000000 f510c888 00000001 00000000 f5106000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] copy_from_user+0x2d/0x59 [] copy_from_user+0x2d/0x59 [] verify_iovec+0x2a/0x8a [] sys_sendmsg+0x232/0x237 [] sys_socketcall+0xb7/0x2a9 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f5027d7c 0 3542 1 f7998330 00000086 00000002 f5027d7c f5027d70 00000000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 f5027d80 f7998484 c373ed00 f64a8a80 00000000 ffff985c 00000000 00000000 f5026000 ffffffff 00000246 00000000 00000000 Call Trace: [] sys_semtimedop+0x5a8/0x7a4 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f510c798 0 3556 1 f7a03390 00000086 00000246 f510c798 00000000 f48ac000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f7a034e4 c373ed00 f64a8a80 00000000 f510c788 00000003 00000000 f48ac000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] copy_from_user+0x2d/0x59 [] copy_from_user+0x2d/0x59 [] verify_iovec+0x2a/0x8a [] sys_sendmsg+0x232/0x237 [] sys_socketcall+0xb7/0x2a9 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f4884a98 0 3653 1 f6a60e20 00000086 00000246 f4884a98 00000000 f48d6000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f6a60f74 c3734d00 f64a8a80 00000000 f4884a88 00000004 00000000 f48d6000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] copy_from_user+0x2d/0x59 [] copy_from_user+0x2d/0x59 [] verify_iovec+0x2a/0x8a [] sys_sendmsg+0x232/0x237 [] sys_socketcall+0xb7/0x2a9 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f75a5d98 0 3686 1 f79c2530 00000086 00000246 f75a5d98 00000000 f4878000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f79c2684 c373ed00 f64a8a80 00000000 f75a5d88 00000005 00000000 f4878000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] copy_from_user+0x2d/0x59 [] copy_from_user+0x2d/0x59 [] verify_iovec+0x2a/0x8a [] sys_sendmsg+0x232/0x237 [] __dequeue_entity+0x45/0xaa [] update_curr+0x40/0x52 [] set_next_entity+0x14/0x38 [] pick_next_task_fair+0x8f/0xa8 [] sys_socketcall+0xb7/0x2a9 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f75a5e98 0 3697 1 f79d1710 00000086 00000246 f75a5e98 00000000 f4898000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f79d1864 c373ed00 f64a8a80 00000000 f75a5e88 00000006 00000000 f4898000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] copy_from_user+0x2d/0x59 [] copy_from_user+0x2d/0x59 [] verify_iovec+0x2a/0x8a [] sys_sendmsg+0x232/0x237 [] sys_socketcall+0xb7/0x2a9 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f64f4998 0 3710 1 f7155710 00000086 00000246 f64f4998 00000000 f50f4000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f7155864 c373ed00 f64a8a80 00000000 f64f4988 00000007 00000000 f50f4000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] __dequeue_entity+0x45/0xaa [] set_next_entity+0x14/0x38 [] pick_next_task_fair+0x8f/0xa8 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f64f4a98 0 3714 1 f711b590 00000086 00000246 f64f4a98 00000000 f75d6000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f711b6e4 c3734d00 f64a8a80 00000000 f64f4a88 00000008 00000000 f75d6000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= groupd S f6465bf4 0 3527 1 f6a59610 00000082 00000002 f6465bf4 00000002 00000000 f7abc4b0 c045f0c0 c0461d00 c0461d00 c0461d00 f6465bf8 f6a59764 c373ed00 f7963580 f6465efc 00000046 00000000 00000002 00000001 f4ca931c 00000286 f78c2080 f6465efc Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] schedule_timeout+0x69/0xa1 [] unix_poll+0x1a/0xab [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] _read_unlock+0x14/0x1c [] unix_stream_sendmsg+0x1c3/0x32b [] sock_aio_write+0xcb/0xe8 [] sockfd_lookup_light+0x24/0x41 [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] vfs_write+0xe5/0x118 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= fenced S f5177bf4 0 3532 1 f6a48430 00000082 00000002 f5177bf4 f5177be8 00000000 f55f982c c045f0c0 c0461d00 c0461d00 c0461d00 f5177bf8 f6a48584 c373ed00 f78a5800 f5177ebc ffff95a0 00000000 00000002 00000001 ffffffff 00000286 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] unix_poll+0x1a/0xab [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] sched_clock_cpu+0x10f/0x158 [] update_curr+0x40/0x52 [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] unix_stream_recvmsg+0x3a0/0x4d8 [] sock_aio_read+0xdb/0xef [] sockfd_lookup_light+0x24/0x41 [] skb_dequeue+0x50/0x56 [] dput+0x97/0x145 [] mntput_no_expire+0x18/0xef [] filp_close+0x3e/0x62 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= dlm_controld S c373baa0 0 3536 1 f7abc4b0 00000082 000007fb c373baa0 00000002 00000000 f55faaac c045f0c0 c0461d00 c0461d00 c0461d00 f4837bf8 f7abc604 c373ed00 f756c800 f4837ebc 00000046 00000000 00000002 00000001 f55faa9c 00000286 f79de080 f4837ebc Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] schedule_timeout+0x69/0xa1 [] unix_poll+0x1a/0xab [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] _spin_lock_irqsave+0x41/0x49 [] netlink_recvmsg+0x1d1/0x294 [] __wake_up_common+0x46/0x68 [] sock_recvmsg+0xcf/0xf3 [] autoremove_wake_function+0x0/0x35 [] sock_aio_read+0xdb/0xef [] sockfd_lookup_light+0x24/0x41 [] do_sync_read+0xd2/0x10e [] autoremove_wake_function+0x0/0x35 [] sys_recv+0x37/0x3b [] sys_socketcall+0x1a9/0x2a9 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= gfs_controld S f75b1bf4 0 3540 1 f7ac6da0 00000082 00000002 f75b1bf4 f75b1be8 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f75b1bf8 f7ac6ef4 c3734d00 f78a5080 f7ac6da0 ffffef62 00000000 f7b98a80 f75b1ed4 ffffffff f8939698 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] _spin_unlock+0x14/0x1c [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] ipc_lock+0x60/0xb8 [] _spin_lock+0x31/0x3c [] sys_semtimedop+0x2bf/0x7a4 [] autoremove_wake_function+0x0/0x35 [] _spin_lock_irq+0x37/0x42 [] futex_wake+0x52/0xe1 [] do_futex+0x7a/0x951 [] net_rx_action+0x9d/0x1db [] do_page_fault+0xa5/0x848 [] sys_futex+0x86/0x111 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= clvmd S f50fbb14 0 3644 1 f71c81b0 00000086 00000002 f50fbb14 f50fbb08 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f50fbb18 f71c8304 c3734d00 f64a8300 c04b9e00 ffffde5e 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] _spin_unlock_irq+0x20/0x23 [] finish_task_switch+0x58/0xab [] finish_task_switch+0x26/0xab [] schedule+0x228/0x708 [] sched_clock_cpu+0x10f/0x158 [] update_curr+0x40/0x52 [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] unix_stream_recvmsg+0x3a0/0x4d8 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] sock_aio_read+0xdb/0xef [] sockfd_lookup_light+0x24/0x41 [] sys_recvfrom+0x11f/0x121 [] _spin_unlock_irq+0x20/0x23 [] finish_task_switch+0x58/0xab [] finish_task_switch+0x26/0xab [] schedule+0x228/0x708 [] dnotify_parent+0x1f/0x5f [] sys_select+0xd6/0x19e [] sysenter_do_call+0x12/0x35 ======================= clvmd S 00000002 0 3655 1 f7aa5490 00000086 f7aa5490 00000002 00000000 f75e02a0 f7aa5490 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f7aa55e4 c373ed00 f64a8300 f75e02d8 00000282 f75e02d8 b7dc3368 c0319cc1 00000046 00000246 00000246 f75e0290 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] device_read+0x12d/0x347 [dlm] [] default_wake_function+0x0/0x8 [] device_read+0x0/0x347 [dlm] [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= clvmd S 00000002 0 3656 1 f7ab3510 00000096 f7ab3510 00000002 00000000 c082d818 f7ab3510 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f7ab3664 c3734d00 f64a8300 f4913e3c 00000282 00000000 080b1fa4 c0319cc1 00000046 00000046 f4913e3c 00000282 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] futex_wait+0x318/0x396 [] futex_wake+0x52/0xe1 [] _spin_unlock+0x14/0x1c [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] sys_futex+0x86/0x111 [] up_write+0x14/0x28 [] sysenter_do_call+0x12/0x35 ======================= dlm_astd S 00000002 0 3645 2 f7aa43b0 00000092 00000246 00000002 00000000 f8939428 f7aa43b0 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f7aa4504 c3734d00 f7801800 00000246 00000000 00000002 00000046 00000046 f8939440 00000000 f8939440 f8939460 Call Trace: [] gdlm_bast+0x0/0xa4 [gfs] [] dlm_astd+0x62/0x155 [dlm] [] dlm_astd+0x0/0x155 [dlm] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_scand R running 0 3646 2 f7abcd20 00000092 00000002 f50e7f88 f50e7f7c 00000000 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f50e7f8c f7abce74 c373ed00 f64a8a80 f706e000 ffffef4b 00000000 00000282 00000282 ffffffff f706e000 00000000 00000000 Call Trace: [] dlm_scand+0x0/0x77 [dlm] [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] dlm_scand+0x69/0x77 [dlm] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_recv/0 R running 0 3647 2 f78e5690 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f78e57e4 c3734d00 f7801800 00000286 00000001 f75e071c c0319cc1 00000000 00000046 00000046 f75e0724 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_recv/1 S 00000000 0 3648 2 f6a58da0 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f6a58ef4 c373ed00 f756cd00 00000286 00000001 f75e069c c0319cc1 00000000 00000046 00000046 f75e06a4 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_send S f4823f94 0 3649 2 f6ab0aa0 00000082 00000002 f4823f94 f4823f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f4823f98 f6ab0bf4 c3734d00 f7801800 00000286 ffffef69 00000000 c0319cc1 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_recoverd S f5dc1f80 0 3650 2 f79f3290 00000086 00000002 f5dc1f80 f5dc1f74 00000000 00002bd4 c045f0c0 c0461d00 c0461d00 c0461d00 f5dc1f84 f79f33e4 c3734d00 f64a8a80 00000002 ffff97ae 00000000 f79f3290 00000046 ffffffff f7001280 00000000 00000000 Call Trace: [] dlm_recoverd+0x163/0x55a [dlm] [] __wake_up_common+0x46/0x68 [] dlm_recoverd+0x0/0x55a [dlm] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kdmflush S 00000000 0 3663 2 f6ab82b0 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f6ab8404 c373ed00 f7c60d00 00000286 00000001 f482bd9c c0319cc1 00000000 00000046 00000046 f482bda4 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kdmflush S f6047f94 0 3669 2 f71d0230 00000082 00000002 f6047f94 f6047f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f6047f98 f71d0384 c373ed00 f756c800 00000286 ffff980b 00000000 c0319cc1 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= lock_dlm1 S f5195f50 0 3690 2 f79c2da0 00000086 00000002 f5195f50 f5195f44 00000000 f79c2da0 c045f0c0 c0461d00 c0461d00 c0461d00 f5195f54 f79c2ef4 c373ed00 f7896300 00000246 ffffa9f2 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] gdlm_thread+0x112/0x74a [gfs] [] autoremove_wake_function+0x0/0x35 [] gdlm_thread1+0x0/0xa [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= lock_dlm2 S 00000002 0 3694 2 f7a022b0 00000086 c040dc00 00000002 00000000 f78d2b64 f7a022b0 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f7a02404 c3734d00 f78a5580 00000246 00000000 00000002 00000000 00000000 f8a45d59 00000246 00000246 f78d2b54 Call Trace: [] gdlm_thread+0x12d/0x74a [gfs] [] gdlm_thread+0x112/0x74a [gfs] [] autoremove_wake_function+0x0/0x35 [] gdlm_thread2+0x0/0x7 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_recoverd S f8939428 0 3695 2 f7a428a0 00000086 00000000 f8939428 f7a428a0 f5052bf4 00002bd4 c045f0c0 c0461d00 c0461d00 c0461d00 f5181800 f7a429f4 c373ed00 f64a8a80 00000002 00000000 00000000 f7a428a0 00000046 00000000 f7001280 f704b000 f7001280 Call Trace: [] dlm_recoverd+0x163/0x55a [dlm] [] __wake_up_common+0x46/0x68 [] dlm_recoverd+0x0/0x55a [dlm] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_scand R running 0 3701 2 f7154ea0 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7154ff4 c3734d00 f78a5080 c04b9e00 00000286 c04b9e00 00000286 00000286 f4827fa4 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] gfs_scand+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_scand+0x39/0x48 [gfs] [] gfs_scand+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_glockd S f4897f94 0 3702 2 f7abd590 00000082 00000002 f4897f94 f4897f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f4897f98 f7abd6e4 c373ed00 f64a8a80 00000286 ffffef4b 00000000 c0319cc1 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] gfs_glockd+0x80/0xb4 [gfs] [] autoremove_wake_function+0x0/0x35 [] gfs_glockd+0x0/0xb4 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_recoverd S f5161f80 0 3704 2 f7a17490 00000086 00000002 f5161f80 f5161f74 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f5161f84 f7a175e4 c3734d00 f78a5580 c04b9e00 ffffde92 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_recoverd+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_recoverd+0x39/0x48 [gfs] [] gfs_recoverd+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_logd R running 0 3705 2 f7154630 00000082 00000400 c373baa0 00000002 f4893f2c f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f4893f34 f7154784 c373ed00 f781a580 f706e000 ffffef77 00000000 00000286 00000286 ffffffff f706e000 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_logd+0x23/0xb7 [gfs] [] process_timeout+0x0/0x5 [] gfs_logd+0x41/0xb7 [gfs] [] gfs_logd+0x0/0xb7 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_quotad R running 0 3706 2 f71e0ba0 00000086 00000002 f6039f64 f6039f58 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f6039f68 f71e0cf4 c3734d00 f7c60300 c04b9e00 ffffee49 00000000 00000282 00000282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_quotad+0x88/0x16d [gfs] [] process_timeout+0x0/0x5 [] gfs_quotad+0xa6/0x16d [gfs] [] gfs_quotad+0x0/0x16d [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_inoded R running 0 3707 2 f71e0330 00000086 00000002 f6151f80 f6151f74 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f6151f84 f71e0484 c3734d00 f78a5580 c04b9e00 ffffea63 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_inoded+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_inoded+0x39/0x48 [gfs] [] gfs_inoded+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= lock_dlm1 S 00000002 0 3711 2 f71c8a20 00000086 00000000 00000002 00000000 f71bb764 f71c8a20 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f71c8b74 c3734d00 f7801800 00000246 00000000 00000002 00000000 00000000 f8a45d59 00000246 00000246 f71bb754 Call Trace: [] gdlm_thread+0x12d/0x74a [gfs] [] gdlm_thread+0x112/0x74a [gfs] [] autoremove_wake_function+0x0/0x35 [] gdlm_thread1+0x0/0xa [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= lock_dlm2 S f6157f50 0 3712 2 f6a68ea0 00000086 00000002 f6157f50 f6157f44 00000000 f6a68ea0 c045f0c0 c0461d00 c0461d00 c0461d00 f6157f54 f6a68ff4 c373ed00 f781a580 00000246 ffffef73 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] gdlm_thread+0x112/0x74a [gfs] [] autoremove_wake_function+0x0/0x35 [] gdlm_thread2+0x0/0x7 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_recoverd S f8939428 0 3713 2 f6a69710 00000086 00000000 f8939428 f6a69710 f518ebf4 00002bd4 c045f0c0 c0461d00 c0461d00 c0461d00 f64b5800 f6a69864 c373ed00 f7801580 00000002 00000000 00000000 f6a69710 00000046 00000000 f7001280 f704b000 f7001280 Call Trace: [] dlm_recoverd+0x163/0x55a [dlm] [] __wake_up_common+0x46/0x68 [] dlm_recoverd+0x0/0x55a [dlm] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_scand R running 0 3719 2 f71580b0 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7158204 c3734d00 f78a5a80 c04b9e00 00000286 c04b9e00 00000286 00000286 f74b5fa4 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] gfs_scand+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_scand+0x39/0x48 [gfs] [] gfs_scand+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_glockd S f219112c 0 3720 2 f6ab1310 00000082 f2191158 f219112c 00000246 f8b44994 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f6ab1464 c3734d00 f7801800 00000286 00000001 f74b7fbc c0319cc1 00000000 00000046 00000046 f8b83504 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] gfs_glockd+0x80/0xb4 [gfs] [] autoremove_wake_function+0x0/0x35 [] gfs_glockd+0x0/0xb4 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_recoverd S f4b9bf80 0 3722 2 f79b0ca0 00000086 00000002 f4b9bf80 f4b9bf74 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f4b9bf84 f79b0df4 c3734d00 f64a8a80 c04b9e00 ffffdeeb 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_recoverd+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_recoverd+0x39/0x48 [gfs] [] gfs_recoverd+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_logd R running 0 3723 2 f79b0430 00000082 00000002 f4b9df30 f4b9df24 00000000 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f4b9df34 f79b0584 c373ed00 f64a8a80 f706e000 ffffef4b 00000000 00000286 00000286 ffffffff f706e000 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_logd+0x23/0xb7 [gfs] [] process_timeout+0x0/0x5 [] gfs_logd+0x41/0xb7 [gfs] [] gfs_logd+0x0/0xb7 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_quotad R running 0 3724 2 f6a58530 00000086 00000002 f4b9ff64 f4b9ff58 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f4b9ff68 f6a58684 c3734d00 f7c60300 c04b9e00 ffffee87 00000000 00000282 00000282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_quotad+0x88/0x16d [gfs] [] process_timeout+0x0/0x5 [] gfs_quotad+0xa6/0x16d [gfs] [] gfs_quotad+0x0/0x16d [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_inoded R running 0 3725 2 f7a749a0 00000086 00000002 f4ba1f80 f4ba1f74 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f4ba1f84 f7a74af4 c3734d00 f781a300 c04b9e00 ffffeab3 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_inoded+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_inoded+0x39/0x48 [gfs] [] gfs_inoded+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= lighttpd R running 0 3859 1 f79a5490 00200082 00000002 f2889bf4 f2889be8 00000000 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f2889bf8 f79a55e4 c373ed00 f7c60a80 f706e000 ffffef2f 00000000 00200282 00200282 ffffffff f706e000 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] __qdisc_run+0x7f/0x1c5 [] _spin_lock+0x31/0x3c [] __qdisc_run+0x7f/0x1c5 [] _spin_lock_irqsave+0x41/0x49 [] _spin_lock_irqsave+0x41/0x49 [] scsi_next_command+0x25/0x2f [] scsi_end_request+0x52/0x75 [] scsi_io_completion+0x87/0x3de [] _spin_lock+0x31/0x3c [] blk_done_softirq+0x5b/0x68 [] __do_softirq+0xa2/0xf9 [] _local_bh_enable+0x44/0xa0 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= atd S f3de9f2c 0 3866 1 f79ca5b0 00000082 00000002 f3de9f2c f3de9f20 00000000 f6455ddc c045f0c0 c0461d00 c0461d00 c0461d00 f3de9f30 f79ca704 c3734d00 f7832800 f3de9f60 ffff9c98 00000000 0000000a c045ef20 ffffffff 00000350 00000000 00000000 Call Trace: [] do_nanosleep+0x6d/0x96 [] hrtimer_nanosleep+0x50/0xad [] hrtimer_wakeup+0x0/0x18 [] sys_nanosleep+0x58/0x5c [] sysenter_do_call+0x12/0x35 ======================= cron S f29dff2c 0 3877 1 f6a504b0 00000082 00000002 f29dff2c f29dff20 00000000 c04782ec c045f0c0 c0461d00 c0461d00 c0461d00 f29dff30 f6a50604 c3734d00 f7963080 f29dff60 ffffe623 00000000 00000036 c045ef20 ffffffff 00000044 00000000 00000000 Call Trace: [] do_nanosleep+0x6d/0x96 [] hrtimer_nanosleep+0x50/0xad [] hrtimer_wakeup+0x0/0x18 [] sys_nanosleep+0x58/0x5c [] sysenter_do_call+0x12/0x35 ======================= S99phplog R running 0 3887 1 f79cae20 00000082 00000002 f29eff2c f29eff20 00000000 f64189dc c045f0c0 c0461d00 c0461d00 c0461d00 f29eff30 f79caf74 c373ed00 f78ff800 f29eff60 ffffe5ac 00000000 00000036 c045ef20 ffffffff 0000003d 00000000 00000000 Call Trace: [] do_nanosleep+0x6d/0x96 [] hrtimer_nanosleep+0x50/0xad [] hrtimer_wakeup+0x0/0x18 [] sys_nanosleep+0x58/0x5c [] sysenter_do_call+0x12/0x35 ======================= lighttpd R running 0 3937 1 f71d82b0 00200082 00000002 f2825bf4 f2825be8 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f2825bf8 f71d8404 c3734d00 f78a5580 c04b9e00 ffffef43 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] e1000_xmit_frame+0x7a5/0xb7b [e1000e] [] local_bh_enable_ip+0x79/0xb5 [] ipt_do_table+0x1fe/0x496 [ip_tables] [] __qdisc_run+0x7f/0x1c5 [] _spin_lock+0x31/0x3c [] __qdisc_run+0x7f/0x1c5 [] dev_queue_xmit+0xa2/0x509 [] dev_queue_xmit+0xee/0x509 [] local_bh_enable+0x7f/0xc8 [] dev_queue_xmit+0xee/0x509 [] dev_queue_xmit+0x36/0x509 [] ip_finish_output+0x112/0x26b [] ip_local_out+0x15/0x17 [] ip_queue_xmit+0x195/0x327 [] do_IRQ+0x40/0x72 [] __generic_file_aio_write_nolock+0x285/0x4df [] trace_hardirqs_on_thunk+0xc/0x10 [] restore_nocheck_notrace+0x0/0xe [] tcp_v4_send_check+0x3b/0xc8 [] tcp_transmit_skb+0x363/0x659 [] generic_file_aio_write+0x73/0xd8 [] dput+0x97/0x145 [] mntput_no_expire+0x18/0xef [] filp_close+0x3e/0x62 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 00000002 0 3944 1 f7ac6530 00000082 f7ac6530 00000002 00000000 c040c990 f7ac6530 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f7ac6684 c3734d00 f78ff300 00000000 00000002 00000002 00000000 00000000 c01228e1 00000246 00000246 c040c980 Call Trace: [] do_wait+0x96/0x342 [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 00000002 0 3956 1 f7a821b0 00000082 f7a821b0 00000002 00000000 c040c990 f7a821b0 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f7a82304 c3734d00 f78ff080 00000000 00000002 00000002 00000000 00000000 c01228e1 00000246 00000246 c040c980 Call Trace: [] do_wait+0x96/0x342 [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f2847f24 0 3968 1 f79c3610 00000082 00000002 f2847f24 f2847f18 00000000 f79c3610 c045f0c0 c0461d00 c0461d00 c0461d00 f2847f28 f79c3764 c3734d00 f7967300 00000000 ffff9d3f 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f41a9f24 0 3980 1 f6a50d20 00000082 00000002 f41a9f24 f41a9f18 00000000 f6a50d20 c045f0c0 c0461d00 c0461d00 c0461d00 f41a9f28 f6a50e74 c373ed00 f6b40800 00000000 ffff9d57 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f4117f24 0 3992 1 f7a0aba0 00000082 00000002 f4117f24 f4117f18 00000000 f7a0aba0 c045f0c0 c0461d00 c0461d00 c0461d00 f4117f28 f7a0acf4 c3734d00 f6b40300 00000000 ffff9d5f 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f402bf24 0 4005 1 f70822b0 00000082 00000002 f402bf24 f402bf18 00000000 f70822b0 c045f0c0 c0461d00 c0461d00 c0461d00 f402bf28 f7082404 c3734d00 f7801a80 00000000 ffff9d3f 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 00000002 0 4017 1 f6aa81b0 00000082 f6aa81b0 00000002 00000000 c040c990 f6aa81b0 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f6aa8304 c3734d00 f7982d00 00000000 00000002 00000002 00000000 00000000 c01228e1 00000246 00000246 c040c980 Call Trace: [] do_wait+0x96/0x342 [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c373baa0 0 4019 3956 f7a2e530 00000082 00000400 c373baa0 00000002 f29cbe20 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f29cbe28 f7a2e684 c373ed00 f7982a80 f759e45c ffff9d55 00000000 00000001 f759e44c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4020 3956 f7998ba0 00000082 f7998ba0 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f7998ba0 f7998cf4 c373ed00 f7982580 f759e45c 00000046 f4ca9d2c 00000001 f759e44c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 0000001f 0 4021 3956 f79e8130 00000082 c0405c80 0000001f 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f79e8130 f79e8284 c373ed00 f6505080 f759e45c 00000046 f4ca9d2c 00000001 f759e44c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4022 3956 f7a1aca0 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f7a1aca0 f7a1adf4 c373ed00 f6505a80 f759e45c 00000046 f4ca9d2c 00000001 f759e44c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4023 3944 f7a2f610 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f7a2f610 f7a2f764 c373ed00 f6505580 f782b75c 00000046 f36f932c 00000001 f782b74c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f2927e24 0 4024 3944 f6a78920 00000082 00000002 f2927e24 f2927e18 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f2927e28 f6a78a74 c3734d00 f6505800 f782b75c ffffb5a6 00000000 00000001 f782b74c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] skb_dequeue+0x50/0x56 [] dput+0x97/0x145 [] sys_socketcall+0x23e/0x2a9 [] filp_close+0x3e/0x62 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c373baa0 0 4025 3944 f79da8a0 00000082 00000400 c373baa0 00000002 f2907e20 000800d0 c045f0c0 c0461d00 c0461d00 c0461d00 f2907e28 f79da9f4 c373ed00 f781a080 f782b75c ffff9d56 00000000 00000001 f782b74c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c373baa0 0 4026 3944 f7a16c20 00000082 00000400 c373baa0 00000002 f28ade20 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f28ade28 f7a16d74 c373ed00 f781a800 f782b75c ffff9d56 00000000 00000001 f782b74c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c01512af 0 4027 3968 f7ab2ca0 00000082 c3423b1c c01512af 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f7ab2ca0 f7ab2df4 c3734d00 f781aa80 f782ba5c 00000046 f36f90ac 00000001 f782ba4c 00000046 00000000 00000002 00000001 Call Trace: [] rmqueue_bulk+0x67/0x71 [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4028 4005 f7a962b0 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f7a962b0 f7a96404 c373ed00 f6b40a80 f759ea5c 00000046 f4ca90ac 00000001 f759ea4c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S ffffffff 0 4029 4017 f79b8d20 00000082 00000001 ffffffff 00000000 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f79b8d20 f79b8e74 c3734d00 f781ad00 f7855a5c 00000046 f241e82c 00000001 f7855a4c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] skb_dequeue+0x50/0x56 [] dput+0x97/0x145 [] sys_socketcall+0x23e/0x2a9 [] filp_close+0x3e/0x62 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f7054bb4 0 4030 3968 f7a6a920 00000082 00000000 f7054bb4 f7a6a920 f7a6a920 000800d0 c045f0c0 c0461d00 c0461d00 c0461d00 f7a6a920 f7a6aa74 c3734d00 f7896a80 f782ba5c 00000046 f36f90ac 00000001 f782ba4c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f2dcbe24 0 4031 4017 f71d9390 00000082 00000002 f2dcbe24 f2dcbe18 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f2dcbe28 f71d94e4 c3734d00 f7896080 f7855a5c ffffa048 00000000 00000001 f7855a4c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] skb_dequeue+0x50/0x56 [] dput+0x97/0x145 [] sys_socketcall+0x23e/0x2a9 [] filp_close+0x3e/0x62 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c01512af 0 4032 3968 f7a163b0 00000082 c3424e00 c01512af 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f7a163b0 f7a16504 c3734d00 f7c60d00 f782ba5c 00000046 f36f90ac 00000001 f782ba4c 00000046 00000000 00000002 00000001 Call Trace: [] rmqueue_bulk+0x67/0x71 [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f4177e24 0 4033 4017 f7a6a0b0 00000082 00000002 f4177e24 f4177e18 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f4177e28 f7a6a204 c3734d00 f756ca80 f7855a5c ffffc6d5 00000000 00000001 f7855a4c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] skb_dequeue+0x50/0x56 [] dput+0x97/0x145 [] sys_socketcall+0x23e/0x2a9 [] filp_close+0x3e/0x62 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c01512af 0 4034 3968 f79e5190 00000082 c342544c c01512af 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f79e5190 f79e52e4 c3734d00 f756c080 f782ba5c 00000046 f36f90ac 00000001 f782ba4c 00000046 00000000 00000002 00000001 Call Trace: [] rmqueue_bulk+0x67/0x71 [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f2da3e24 0 4035 4017 f79e4920 00000082 00000002 f2da3e24 f2da3e18 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f2da3e28 f79e4a74 c3734d00 f756cd00 f7855a5c ffffe979 00000000 00000001 f7855a4c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] skb_dequeue+0x50/0x56 [] dput+0x97/0x145 [] sys_socketcall+0x23e/0x2a9 [] filp_close+0x3e/0x62 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c34260ec 0 4036 4005 f7a97390 00000082 00000000 c34260ec 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f7a97390 f7a974e4 c373ed00 f7982080 f759ea5c 00000046 f4ca90ac 00000001 f759ea4c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c3731aa0 0 4037 3980 f7ab2430 00000082 00000400 c3731aa0 00000002 f4007e20 000800d0 c045f0c0 c0461d00 c0461d00 c0461d00 f4007e28 f7ab2584 c3734d00 f6b40580 f783ed5c ffff9d3f 00000001 00000001 f783ed4c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4038 3980 f79fc230 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f79fc230 f79fc384 c3734d00 f7982300 f783ed5c 00000046 f240dd2c 00000001 f783ed4c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c01512af 0 4039 3980 f7a0b410 00000082 c34273c8 c01512af 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f7a0b410 f7a0b564 c373ed00 f64a8800 f783ed5c 00000046 f240dd2c 00000001 f783ed4c 00000046 00000000 00000002 00000001 Call Trace: [] rmqueue_bulk+0x67/0x71 [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c01512af 0 4040 4005 f78e4e20 00000082 c3427a14 c01512af 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f78e4e20 f78e4f74 c3734d00 f6a8d580 f759ea5c 00000046 f4ca90ac 00000001 f759ea4c 00000046 00000000 00000002 00000001 Call Trace: [] rmqueue_bulk+0x67/0x71 [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4041 3980 f71c9290 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f71c9290 f71c93e4 c373ed00 f7967a80 f783ed5c 00000046 f240dd2c 00000001 f783ed4c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4042 4005 f79cb690 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f79cb690 f79cb7e4 c3734d00 f6a8da80 f759ea5c 00000046 f4ca90ac 00000001 f759ea4c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f29b9e24 0 4043 3992 f2d7b210 00000082 00000002 f29b9e24 f29b9e18 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f29b9e28 f2d7b364 c3734d00 f6a8d080 f75ded5c ffffe350 00000000 00000001 f75ded4c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] skb_dequeue+0x50/0x56 [] dput+0x97/0x145 [] sys_socketcall+0x23e/0x2a9 [] filp_close+0x3e/0x62 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4044 3992 f2d7a9a0 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f2d7a9a0 f2d7aaf4 c3734d00 f6a8d300 f75ded5c 00000046 f240daac 00000001 f75ded4c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f5009e24 0 4045 3992 f2d7a130 00000082 00000002 f5009e24 f5009e18 00000000 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f5009e28 f2d7a284 c373ed00 f28a9d00 f75ded5c ffff9d5f 00000000 00000001 f75ded4c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c01512af 0 4046 3992 f29e1290 00000082 c34112c0 c01512af 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f29e1290 f29e13e4 c3734d00 f28a9a80 f75ded5c 00000046 f240daac 00000001 f75ded4c 00000046 00000000 00000002 00000001 Call Trace: [] rmqueue_bulk+0x67/0x71 [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f29e5e24 0 4047 3992 f29e0a20 00000082 00000002 f29e5e24 f29e5e18 00000000 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f29e5e28 f29e0b74 c373ed00 f28a9800 f75ded5c ffff9d60 00000000 00000001 f75ded4c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f7054bb4 0 4048 3992 f29e01b0 00000082 00000000 f7054bb4 f29e01b0 f29e01b0 000800d0 c045f0c0 c0461d00 c0461d00 c0461d00 f29e01b0 f29e0304 c3734d00 f28a9580 f75ded5c 00000046 f240daac 00000001 f75ded4c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f29c5e24 0 4049 3992 f29c3310 00000082 00000002 f29c5e24 f29c5e18 00000000 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f29c5e28 f29c3464 c373ed00 f28a9300 f75ded5c ffff9d60 00000000 00000001 f75ded4c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4050 3992 f29c2aa0 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f29c2aa0 f29c2bf4 c3734d00 f28a9080 f75ded5c 00000046 f240daac 00000001 f75ded4c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= bash S 00000000 0 4055 2969 f6a68630 00000082 00000002 00000000 f7191c3c f6a68630 c03eb0b0 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f6a68784 c3734d00 f7801580 00000000 00000002 00000001 00000046 f719197c f7191800 f7192000 00000046 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] _spin_lock_irqsave+0x41/0x49 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= bash S f1857eb4 0 4078 2965 f2836530 00000082 00000002 f1857eb4 f1857ea8 00000000 c03eb0b0 c045f0c0 c0461d00 c0461d00 c0461d00 f1857eb8 f2836684 c3734d00 f7832300 00000000 ffffe335 00000000 00000046 f748817c ffffffff f651a000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] _spin_lock_irqsave+0x41/0x49 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= mailmanctl S f2017f24 0 4100 1 f2837610 00200082 00000002 f2017f24 f2017f18 00000000 f2837610 c045f0c0 c0461d00 c0461d00 c0461d00 f2017f28 f2837764 c373ed00 f7801080 00000000 ffffedab 00000000 00000000 00000000 ffffffff 00200246 00000000 00000000 Call Trace: [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4101 4100 f29c2230 00000086 00000002 f18bbb14 f18bbb08 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f18bbb18 f29c2384 c3734d00 f7967080 c04b9e00 ffffef43 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] cpupri_set+0x96/0xbc [] __enqueue_rt_entity+0x87/0x151 [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] _read_unlock+0x14/0x1c [] sock_queue_rcv_skb+0x8d/0xc6 [] _spin_unlock+0x14/0x1c [] udp_queue_rcv_skb+0x11b/0x223 [] __udp4_lib_lookup+0xed/0x110 [] __udp4_lib_rcv+0x324/0x7a6 [] ipt_local_in_hook+0x0/0x19 [iptable_filter] [] ip_local_deliver_finish+0x10d/0x15f [] ip_local_deliver_finish+0x3f/0x15f [] nommu_map_single+0x0/0x63 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] e1000_clean_rx_irq+0x301/0x41a [e1000] [] e1000_clean_rx_irq+0x0/0x41a [e1000] [] e1000_clean+0x450/0x5f9 [e1000] [] gfs_readdir+0x329/0x3c2 [gfs] [] filldir_reg_func+0x0/0x1c9 [gfs] [] copy_to_user+0x2f/0x4d [] filldir64+0x0/0xc5 [] net_rx_action+0x85/0x1db [] net_rx_action+0x9d/0x1db [] sys_select+0xd6/0x19e [] _local_bh_enable+0x44/0xa0 [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4102 4100 f203b310 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f203b464 c3734d00 f78a5d00 c04b9e00 00000286 c04b9e00 00000286 00000286 f20a9b38 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] wait_for_common+0x26/0x11b [] find_get_page+0xa2/0xbf [] find_get_page+0xa2/0xbf [] find_get_page+0xd/0xbf [] do_filldir_main+0x28/0x1d1 [gfs] [] compare_dents+0x0/0x65 [gfs] [] wait_for_common+0x26/0x11b [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] filldir64+0x0/0xc5 [] _spin_unlock+0x14/0x1c [] gfs_readdir+0x329/0x3c2 [gfs] [] filldir_reg_func+0x0/0x1c9 [gfs] [] copy_to_user+0x2f/0x4d [] filldir64+0x0/0xc5 [] file_kill+0x13/0x2d [] sys_select+0xd6/0x19e [] copy_to_user+0x2f/0x4d [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4103 4100 f203aaa0 00000086 00000000 c3726014 f203aaa0 f18c5ea0 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 00000002 f203abf4 c3734d00 f7801800 c02a6018 00000246 00000246 f75a0c90 f75a0c90 f18c5e9c c3726004 c0319826 f64f4680 Call Trace: [] dm_get_table+0x11/0x2e [] _read_unlock+0x14/0x1c [] dm_table_unplug_all+0x26/0x35 [] io_schedule+0x1b/0x24 [] sync_page+0x2c/0x39 [] __wait_on_bit+0x42/0x5e [] sync_page+0x0/0x39 [] wait_on_page_bit+0x64/0x6b [] wake_bit_function+0x0/0x3c [] wait_on_page_writeback_range+0x9c/0x104 [] gfs_sync_page_i+0x28/0x5c [gfs] [] gfs_drop_inode+0x55/0x5d [gfs] [] iput+0x44/0x4a [] do_unlinkat+0xc6/0x140 [] do_munmap+0x183/0x1e0 [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4104 4100 00000283 00000283 000000e0 f71a0000 f88ed512 0000002a 00000001 c04b9e00 00000286 f18c9b38 f18c9b18 c0319cc1 f71a0618 0000002a f7874f00 00000046 c04b9e00 00000286 c04b9e00 f71a0000 f7097080 00009b38 00df0000 f71a0580 Call Trace: [] e1000_xmit_frame+0x7a5/0xb7b [e1000e] [] _spin_lock_irqsave+0x41/0x49 [] read_tsc+0x6/0x28 [] sched_slice+0x31/0x37 [] read_tsc+0x6/0x28 [] run_timer_softirq+0x30/0x192 [] _spin_unlock_irq+0x20/0x23 [] run_timer_softirq+0x16a/0x192 [] __do_softirq+0xa2/0xf9 [] _local_bh_enable+0x44/0xa0 [] trace_hardirqs_on_thunk+0xc/0x10 [] restore_nocheck_notrace+0x0/0xe [] native_read_tsc+0x8/0xf [] delay_tsc+0x12/0x47 [] __delay+0x6/0x7 [] _raw_spin_lock+0xe6/0x170 [] igrab+0xd/0x3b [] igrab+0xd/0x3b [] gfs_iget+0x38/0x1e7 [gfs] [] gfs_inode_attr_in+0xb/0x26 [gfs] [] inode_go_unlock+0x20/0x2a [gfs] [] gfs_glock_dq+0xc5/0x157 [gfs] [] gfs_glock_dq_uninit+0x8/0x10 [gfs] [] gfs_permission+0x59/0x63 [gfs] [] inode_permission+0x57/0x95 [] __link_path_walk+0x69/0xbf0 [] core_sys_select+0x22/0x335 [] path_walk+0x37/0x73 [] do_path_lookup+0x63/0x10d [] getname+0x89/0x98 [] user_path_at+0x37/0x68 [] copy_to_user+0x2f/0x4d [] sys_linkat+0x3d/0xf5 [] copy_to_user+0x2f/0x4d [] sys_select+0x183/0x19e [] sys_link+0x2f/0x33 [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4105 4100 f18cb390 00000086 00000002 f18cdb14 f18cdb08 00000000 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f18cdb18 f18cb4e4 c373ed00 f7c60800 f706e000 ffffef7b 00000000 00000286 00000286 ffffffff f706e000 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] wait_for_common+0x26/0x11b [] wait_for_common+0x26/0x11b [] _spin_unlock_irq+0x20/0x23 [] wait_for_common+0xcd/0x11b [] find_get_page+0xa2/0xbf [] gfs_sort+0x80/0x103 [gfs] [] do_filldir_main+0x28/0x1d1 [gfs] [] compare_dents+0x0/0x65 [gfs] [] getbuf+0xfe/0x196 [gfs] [] wait_for_common+0x26/0x11b [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] filldir64+0x0/0xc5 [] _spin_unlock+0x14/0x1c [] gfs_readdir+0x329/0x3c2 [gfs] [] filldir_reg_func+0x0/0x1c9 [gfs] [] copy_to_user+0x2f/0x4d [] filldir64+0x0/0xc5 [] dput+0xa9/0x145 [] sys_select+0xd6/0x19e [] filp_close+0x3e/0x62 [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4106 4100 f18cab20 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f18cac74 c3734d00 f7c60580 c04b9e00 00000286 c04b9e00 00000286 00000286 f18cfb38 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] e1000_clean+0x450/0x5f9 [e1000] [] cpupri_set+0x96/0xbc [] __pollwait+0x0/0xba [] net_rx_action+0x85/0x1db [] net_rx_action+0x9d/0x1db [] __do_softirq+0xa2/0xf9 [] _local_bh_enable+0x44/0xa0 [] find_get_page+0xa2/0xbf [] find_get_page+0xa2/0xbf [] find_get_page+0xd/0xbf [] do_filldir_main+0x28/0x1d1 [gfs] [] compare_dents+0x0/0x65 [gfs] [] wait_for_common+0x26/0x11b [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] filldir64+0x0/0xc5 [] _spin_unlock+0x14/0x1c [] gfs_readdir+0x329/0x3c2 [gfs] [] filldir_reg_func+0x0/0x1c9 [gfs] [] copy_to_user+0x2f/0x4d [] filldir64+0x0/0xc5 [] file_kill+0x13/0x2d [] sys_select+0xd6/0x19e [] copy_to_user+0x2f/0x4d [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4107 4100 f18ca2b0 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f18ca404 c3734d00 f7c60300 c04b9e00 00000286 c04b9e00 00000286 00000286 f5d0fb38 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] _spin_lock_irqsave+0x41/0x49 [] __pollwait+0x0/0xba [] wait_for_common+0x26/0x11b [] wait_for_common+0x26/0x11b [] _spin_unlock_irq+0x20/0x23 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] dput+0x97/0x145 [] gfs_drevalidate+0x1a6/0x1f0 [gfs] [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] copy_to_user+0x2f/0x4d [] cp_new_stat64+0xf9/0x10b [] sys_select+0xd6/0x19e [] sysenter_do_call+0x12/0x35 ======================= python S f2015b14 0 4108 4100 f283b410 00000086 00000002 f2015b14 f2015b08 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f2015b18 f283b564 c3734d00 f64a8080 c04b9e00 ffffeddf 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] e1000_clean+0x450/0x5f9 [e1000] [] __pollwait+0x0/0xba [] wait_for_common+0x26/0x11b [] wait_for_common+0x26/0x11b [] _spin_unlock_irq+0x20/0x23 [] wait_for_common+0xcd/0x11b [] find_get_page+0xa2/0xbf [] gfs_sort+0x80/0x103 [gfs] [] do_filldir_main+0x28/0x1d1 [gfs] [] compare_dents+0x0/0x65 [gfs] [] getbuf+0xfe/0x196 [gfs] [] wait_for_common+0x26/0x11b [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] filldir64+0x0/0xc5 [] _spin_unlock+0x14/0x1c [] gfs_readdir+0x329/0x3c2 [gfs] [] filldir_reg_func+0x0/0x1c9 [gfs] [] filldir64+0x0/0xc5 [] dput+0xa9/0x145 [] sys_select+0xd6/0x19e [] filp_close+0x3e/0x62 [] sysenter_do_call+0x12/0x35 ======================= smtpd R running 0 4109 3362 f283aba0 00200086 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f283acf4 c3734d00 f64a8d00 c04b9e00 00200282 c04b9e00 00200282 00200282 f143bf40 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= proxymap S 00000002 0 4110 3362 f283a330 00200086 c3731f30 00000002 00000000 f1916ed0 f283a330 c045f0c0 c0461d00 c0461d00 c0461d00 00200046 f283a484 c3734d00 f7896800 f1916e90 f283a330 00000002 00000001 00000000 f1916ea0 00200046 f1916e90 f1916e80 Call Trace: [] schedule_timeout+0x69/0xa1 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= trivial-rewri R running 0 4111 3362 f143f490 00200086 00000002 f4851f1c f4851f10 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f4851f20 f143f5e4 c3734d00 f6b40d00 c04b9e00 ffffef29 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= cleanup R running 0 4112 3362 f2836da0 00200086 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f2836ef4 c3734d00 f64a8580 c04b9e00 00200282 c04b9e00 00200282 00200282 f180df40 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= smtp S f19d9f50 0 4113 3362 f21781b0 00200086 00000002 f19d9f50 f19d9f44 00000000 00200046 c045f0c0 c0461d00 c0461d00 c0461d00 f19d9f54 f2178304 c373ed00 f2033d00 00200282 ffffef4b 00000000 c0319cc1 00000000 ffffffff 00200046 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] flock_lock_file_wait+0x177/0x275 [] autoremove_wake_function+0x0/0x35 [] sys_flock+0x11d/0x122 [] sysenter_do_call+0x12/0x35 ======================= smtp S f154ff50 0 4114 3362 f2179290 00200086 00000002 f154ff50 f154ff44 00000000 00200046 c045f0c0 c0461d00 c0461d00 c0461d00 f154ff54 f21793e4 c373ed00 f6b40080 00200282 ffffef66 00000000 c0319cc1 00000000 ffffffff 00200046 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] flock_lock_file_wait+0x177/0x275 [] autoremove_wake_function+0x0/0x35 [] sys_flock+0x11d/0x122 [] sysenter_do_call+0x12/0x35 ======================= smtp S 00000000 0 4115 3362 f2178a20 00200086 00200046 00000000 00000002 00000001 00200046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f2178b74 c3734d00 f7c60080 00200282 00000001 f1027f44 c0319cc1 00000000 00200046 00200046 f1027f44 00200282 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] flock_lock_file_wait+0x177/0x275 [] autoremove_wake_function+0x0/0x35 [] sys_flock+0x11d/0x122 [] sysenter_do_call+0x12/0x35 ======================= smtp R running 0 4116 3362 f1dbe67c f1dbf0c0 c0182513 f1dbe674 c0180230 f1dbe674 f1dbe67c c0181511 00000008 f1dbf0c0 c0171dc1 00000040 00000000 f1dbe674 f7020680 f2091980 f7041500 00000000 f2091980 c016f61c 0000000c f7041500 f7041580 0000000c Call Trace: [] iput+0x1d/0x4a [] d_kill+0x2b/0x44 [] dput+0x78/0x145 [] __fput+0x127/0x17f [] filp_close+0x3e/0x62 [] sys_close+0x5b/0x97 [] sysenter_do_call+0x12/0x35 ======================= smtp S 00000000 0 4117 3362 f1504ca0 00200086 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f1504df4 c3734d00 f7963300 c04b9e00 00200282 c04b9e00 00200282 00200282 f154bf40 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= SysRq : Show backtrace of all active CPUs CPU0: Pid: 4116, comm: smtp Not tainted (2.6.27.21 #6) EIP: 0060:[] EFLAGS: 00200246 CPU: 0 EIP is at native_read_tsc+0xe/0xf EAX: ad617fe3 EBX: 00000000 ECX: 00000000 EDX: 000000f3 ESI: ad617f17 EDI: 00000001 EBP: 00000000 ESP: f1551eec DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 CR0: 8005003b CR2: b7c6a84d CR3: 31011000 CR4: 000006f0 DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 DR6: ffff0ff0 DR7: 00000400 [] delay_tsc+0x2a/0x47 [] __delay+0x6/0x7 [] _raw_spin_lock+0xe6/0x170 [] _atomic_dec_and_lock+0x31/0x58 [] _atomic_dec_and_lock+0x31/0x58 [] iput+0x1d/0x4a [] d_kill+0x2b/0x44 [] dput+0x78/0x145 [] __fput+0x127/0x17f [] filp_close+0x3e/0x62 [] sys_close+0x5b/0x97 [] sysenter_do_call+0x12/0x35 ======================= SysRq : Emergency Sync SysRq : Emergency Sync SysRq : Show Regs Pid: 4116, comm: smtp Not tainted (2.6.27.21 #6) EIP: 0060:[] EFLAGS: 00200246 CPU: 0 EIP is at delay_tsc+0x2a/0x47 EAX: b8030ea1 EBX: 00000000 ECX: 00000000 EDX: 0000010b ESI: b8030dd5 EDI: 00000001 EBP: 00000000 ESP: f1551ef0 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 CR0: 8005003b CR2: b7c6a84d CR3: 31011000 CR4: 000006f0 DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 DR6: ffff0ff0 DR7: 00000400 [] __delay+0x6/0x7 [] _raw_spin_lock+0xe6/0x170 [] _atomic_dec_and_lock+0x31/0x58 [] _atomic_dec_and_lock+0x31/0x58 [] iput+0x1d/0x4a [] d_kill+0x2b/0x44 [] dput+0x78/0x145 [] __fput+0x127/0x17f [] filp_close+0x3e/0x62 [] sys_close+0x5b/0x97 [] sysenter_do_call+0x12/0x35 ======================= SysRq : Show Memory Mem-Info: DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 Normal per-cpu: CPU 0: hi: 186, btch: 31 usd: 133 CPU 1: hi: 186, btch: 31 usd: 70 HighMem per-cpu: CPU 0: hi: 186, btch: 31 usd: 10 CPU 1: hi: 186, btch: 31 usd: 40 Active:53773 inactive:23336 dirty:1098 writeback:0 unstable:0 free:679348 slab:13056 mapped:11237 pagetables:922 bounce:0 DMA free:8804kB min:1140kB low:1424kB high:1708kB active:0kB inactive:0kB present:15788kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 868 3015 3015 Normal free:784680kB min:64392kB low:80488kB high:96588kB active:3792kB inactive:9156kB present:889680kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 17174 17174 HighMem free:1923908kB min:512kB low:40288kB high:80064kB active:211300kB inactive:84188kB present:2198284kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 DMA: 3*4kB 3*8kB 2*16kB 3*32kB 3*64kB 4*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8804kB Normal: 104*4kB 367*8kB 229*16kB 82*32kB 63*64kB 29*128kB 23*256kB 7*512kB 8*1024kB 4*2048kB 181*4096kB = 784616kB HighMem: 1*4kB 0*8kB 1*16kB 2*32kB 1*64kB 1*128kB 0*256kB 1*512kB 2*1024kB 0*2048kB 469*4096kB = 1923860kB 32045 total pagecache pages 0 pages in swap cache Swap cache stats: add 0, delete 0, find 0/0 Free swap = 979832kB Total swap = 979832kB 786176 pages RAM 556800 pages HighMem 12334 pages reserved 58472 pages shared 71957 pages non-shared SysRq : Show Blocked State task PC stack pid father SysRq : Resetting