============================================= [ INFO: possible recursive locking detected ] 2.6.27.21 #6 --------------------------------------------- dlm_controld/3533 is trying to acquire lock: (&sb->s_type->i_mutex_key#8/2){--..}, at: [] configfs_attach_group+0x43/0x18f [configfs] but task is already holding lock: (&sb->s_type->i_mutex_key#8/2){--..}, at: [] configfs_attach_group+0x43/0x18f [configfs] other info that might help us debug this: 2 locks held by dlm_controld/3533: #0: (&sb->s_type->i_mutex_key#7/1){--..}, at: [] lookup_create+0x1b/0x85 #1: (&sb->s_type->i_mutex_key#8/2){--..}, at: [] configfs_attach_group+0x43/0x18f [configfs] stack backtrace: Pid: 3533, comm: dlm_controld Not tainted 2.6.27.21 #6 [] validate_chain+0x9bd/0xf35 [] __lock_acquire+0x27b/0x932 [] lock_acquire+0x5f/0x77 [] configfs_attach_group+0x43/0x18f [configfs] [] mutex_lock_nested+0xa1/0x25c [] configfs_attach_group+0x43/0x18f [configfs] [] configfs_attach_group+0x43/0x18f [configfs] [] configfs_attach_group+0x43/0x18f [configfs] [] configfs_attach_group+0xe5/0x18f [configfs] [] configfs_mkdir+0x1fb/0x36f [configfs] [] vfs_mkdir+0x9c/0xdf [] _spin_unlock+0x14/0x1c [] sys_mkdirat+0xc5/0xd2 [] trace_hardirqs_on_thunk+0xc/0x10 [] sys_mkdir+0x1f/0x23 [] sysenter_do_call+0x12/0x35 ======================= aoe: e1.0: setting 8704 byte data frames on eth1:0030486543a9 aoe: e2.0: setting 8704 byte data frames on eth1:003048654382 aoe: e2.0: setting 8704 byte data frames on eth1:003048654383 aoe: 0030486543a9 e1.0 vace0 has 2930382333 sectors aoe: 003048654382 e2.0 vace0 has 2930382333 sectors etherd/e1.0: unknown partition table etherd/e2.0: unknown partition table dlm: Using TCP for communications dlm: got connection from 1 dlm: got connection from 5 dlm: connecting to 2 dlm: got connection from 2 dlm: connecting to 4 dlm: got connection from 4 Trying to join cluster "lock_dlm", "kfki:home" Joined cluster. Now mounting FS... GFS: fsid=kfki:home.3: jid=3: Trying to acquire journal lock... GFS: fsid=kfki:home.3: jid=3: Looking at journal... GFS: fsid=kfki:home.3: jid=3: Done GFS: fsid=kfki:home.3: Scanning for log elements... GFS: fsid=kfki:home.3: Found 0 unlinked inodes GFS: fsid=kfki:home.3: Found quota changes for 0 IDs GFS: fsid=kfki:home.3: Done Trying to join cluster "lock_dlm", "kfki:services" Joined cluster. Now mounting FS... GFS: fsid=kfki:services.3: jid=3: Trying to acquire journal lock... GFS: fsid=kfki:services.3: jid=3: Looking at journal... GFS: fsid=kfki:services.3: jid=3: Done GFS: fsid=kfki:services.3: Scanning for log elements... GFS: fsid=kfki:services.3: Found 63 unlinked inodes GFS: fsid=kfki:services.3: Found quota changes for 4 IDs GFS: fsid=kfki:services.3: Done GFS: fsid=kfki:home.3: fast statfs start time = 1238197131 GFS: fsid=kfki:services.3: fast statfs start time = 1238197132 SysRq : Show backtrace of all active CPUs CPU0: Pid: 4149, comm: smtp Not tainted (2.6.27.21 #6) EIP: 0060:[] EFLAGS: 00200246 CPU: 0 EIP is at native_read_tsc+0x8/0xf EAX: 2039323b EBX: 00000000 ECX: 00000000 EDX: 000000d2 ESI: 2039316f EDI: 00000001 EBP: 00000000 ESP: f08ddec8 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 CR0: 8005003b CR2: b7c6f000 CR3: 3152c000 CR4: 000006f0 DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 DR6: ffff0ff0 DR7: 00000400 [] delay_tsc+0x2a/0x47 [] __delay+0x6/0x7 [] _raw_spin_lock+0xe6/0x170 [] new_inode+0x24/0x8d [] new_inode+0x24/0x8d [] sock_alloc+0xd/0x4d [] __sock_create+0x3a/0x1e0 [] net_rx_action+0x85/0x1db [] sock_create+0x34/0x3a [] sys_socket+0x35/0x67 [] sys_socketcall+0x2a2/0x2a9 [] _local_bh_enable+0x44/0xa0 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= SysRq : Show State task PC stack pid father init S f7061b14 0 1 0 f705f110 00000086 00000002 f7061b14 f7061b08 00000000 00000002 c045f0c0 c0461d00 c0461d00 c0461d00 f7061b18 f705f264 c3734d00 f6a8cd00 00000000 ffff9fa6 00000000 00000246 c0318668 ffffffff 00000002 00000000 00000000 Call Trace: [] mutex_lock_nested+0x18c/0x25c [] schedule_timeout+0x69/0xa1 [] inotify_poll+0x45/0x4b [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] __lock_acquire+0x27b/0x932 [] _spin_unlock+0x14/0x1c [] number+0x25b/0x264 [] kmem_cache_free+0x4e/0x62 [] posix_cpu_timer_create+0x40/0xd8 [] unix_dgram_sendmsg+0x226/0x4e3 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] vsnprintf+0x2dc/0x581 [] validate_chain+0x395/0xf35 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] _spin_lock_irqsave+0x41/0x49 [] do_wait+0x31e/0x342 [] sys_select+0x3f/0x19e [] sysenter_do_call+0x12/0x35 ======================= kthreadd S 00000002 0 2 0 f705e8a0 00000086 00000086 00000002 00000000 c03ecbd8 f705e8a0 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f705e9f4 c3734d00 f6417080 00000246 00000000 00000002 00000000 00000000 c0131483 00000246 00000246 c03ecbc8 Call Trace: [] kthreadd+0xea/0x14f [] kthreadd+0x14a/0x14f [] kthreadd+0x0/0x14f [] kernel_thread_helper+0x7/0x1c ======================= migration/0 R running 0 3 2 f705e030 00000082 00000002 f7067f84 f7067f78 00000000 f705e030 c045f0c0 c0461d00 c0461d00 c0461d00 f7067f88 f705e184 c3734d00 f6417080 00000046 ffffdda7 00000000 00000001 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] migration_thread+0x147/0x2cc [] _spin_unlock_irqrestore+0x34/0x39 [] migration_thread+0x0/0x2cc [] migration_thread+0x0/0x2cc [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ksoftirqd/0 S f706bfa8 0 4 2 f7069190 00000092 00000002 f706bfa8 f706bf9c 00000000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 f706bfac f70692e4 c3734d00 f7969a80 c040caa0 ffffe1fd 00000000 c045f100 c0124797 ffffffff c0461b80 00000000 00000000 Call Trace: [] __do_softirq+0xa2/0xf9 [] ksoftirqd+0xee/0xf3 [] ksoftirqd+0x0/0xf3 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= migration/1 S f706df84 0 5 2 f7068920 00000082 00000002 f706df84 f706df78 00000000 f7068920 c045f0c0 c0461d00 c0461d00 c0461d00 f706df88 f7068a74 c373ed00 f0866080 00000046 ffffe102 00000000 00000001 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] migration_thread+0x147/0x2cc [] _spin_unlock_irqrestore+0x34/0x39 [] migration_thread+0x0/0x2cc [] migration_thread+0x0/0x2cc [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ksoftirqd/1 S f7071fa8 0 6 2 f70680b0 00000092 00000002 f7071fa8 f7071f9c 00000000 f7018a80 c045f0c0 c0461d00 c0461d00 c0461d00 f7071fac f7068204 c373ed00 f0866d00 c040caa0 ffffe204 00000000 c045f100 c0124797 ffffffff c0461b80 00000000 00000000 Call Trace: [] __do_softirq+0xa2/0xf9 [] ksoftirqd+0xee/0xf3 [] ksoftirqd+0x0/0xf3 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= events/0 R running 0 7 2 f70729a0 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7072af4 c3734d00 f7969a80 00000286 00000001 f702b51c c0319cc1 00000000 00000046 00000046 f702b524 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= events/1 R running 0 8 2 f7072130 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7072284 c373ed00 f7854300 00000286 00000001 f702b49c c0319cc1 00000000 00000046 00000046 f702b4a4 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= khelper S 00000001 0 9 2 f707d290 00000082 f707d610 00000001 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f707d3e4 c373ed00 f7c37d00 00000286 00000001 f702b39c c0319cc1 00000000 f707d290 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kblockd/0 S 00000000 0 47 2 f707ca20 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f707cb74 c3734d00 f7843080 00000286 00000001 f70a651c c0319cc1 00000000 00000046 00000046 f70a6524 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kblockd/1 S f70cbf94 0 48 2 f707c1b0 00000082 00000002 f70cbf94 f70cbf88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f70cbf98 f707c304 c373ed00 f7854300 00000286 ffffdccf 00000000 c0319cc1 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kacpid S c373baa0 0 49 2 f70cf390 00000082 00000818 c373baa0 00000002 f70d1f90 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f70d1f98 f70cf4e4 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kacpi_notify S f70d3f94 0 50 2 f70ceb20 00000082 00000002 f70d3f94 f70d3f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f70d3f98 f70cec74 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ata/0 S 00000001 0 130 2 f711eda0 00000082 f711f120 00000001 00000002 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f711eef4 c3734d00 c03e81a0 00000286 00000001 f710dd1c c0319cc1 00000000 f711eda0 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ata/1 S f7125f94 0 131 2 f71144b0 00000082 00000002 f7125f94 f7125f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f7125f98 f7114604 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ata_aux S f712df94 0 132 2 f711f610 00000082 00000002 f712df94 f712df88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f712df98 f711f764 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ksuspend_usbd S f712ff94 0 133 2 f7114d20 00000082 00000002 f712ff94 f712ff88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f712ff98 f7114e74 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= khubd S 00000001 0 139 2 f7121690 00000082 f7121a10 00000001 00000000 c03fad50 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f71217e4 c3734d00 f7955300 00000286 00000001 00000000 c0319cc1 00000000 f7121690 c0319d8f f792ac00 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] hub_thread+0x87f/0xcef [] trace_hardirqs_on_caller+0x9d/0x113 [] _spin_unlock_irq+0x20/0x23 [] finish_task_switch+0x58/0xab [] schedule+0x228/0x708 [] autoremove_wake_function+0x0/0x35 [] hub_thread+0x0/0xcef [] trace_hardirqs_on_caller+0x9d/0x113 [] hub_thread+0x0/0xcef [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kseriod S f7151f84 0 142 2 f7120e20 00000082 00000002 f7151f84 f7151f78 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f7151f88 f7120f74 c3734d00 f6aff080 00000286 ffff8c9a 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] serio_thread+0xa4/0x2c6 [] autoremove_wake_function+0x0/0x35 [] serio_thread+0x0/0x2c6 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= pdflush S 00000002 0 181 2 f71588a0 00000082 00000092 00000002 00000000 c03edd8c f71588a0 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f71589f4 c3734d00 c03e81a0 00000046 00000000 00000002 00000001 00000000 c0154020 f71588a0 c0319c5a c0153fc2 Call Trace: [] pdflush+0x5e/0x19c [] _spin_unlock_irq+0x20/0x23 [] pdflush+0x0/0x19c [] trace_hardirqs_on_caller+0x9d/0x113 [] pdflush+0x0/0x19c [] pdflush+0xb2/0x19c [] trace_hardirqs_on_caller+0x9d/0x113 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= pdflush R running 0 182 2 f7158030 00000082 00000002 f71c1f94 f71c1f88 00000000 f7158030 c045f0c0 c0461d00 c0461d00 c0461d00 f71c1f98 f7158184 c3734d00 f7843080 00000046 ffffe0cd 00000000 00000001 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] pdflush+0x0/0x19c [] pdflush+0xb2/0x19c [] wb_kupdate+0x0/0xe3 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kswapd0 S f71e5f28 0 183 2 f7159110 00000096 00000002 f71e5f28 f71e5f1c 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f71e5f2c f7159264 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] kswapd+0x3fd/0x420 [] schedule+0x228/0x708 [] autoremove_wake_function+0x0/0x35 [] _spin_unlock_irqrestore+0x34/0x39 [] kswapd+0x0/0x420 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= aio/0 S 00000001 0 184 2 f7154630 00000082 f71549b0 00000001 00000002 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7154784 c3734d00 c03e81a0 00000286 00000001 f730ac1c c0319cc1 00000000 f7154630 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= aio/1 S f732df94 0 185 2 f7154ea0 00000082 00000002 f732df94 f732df88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f732df98 f7154ff4 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kpsmoused S f6ac3f94 0 335 2 f7155710 00000082 00000002 f6ac3f94 f6ac3f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f6ac3f98 f7155864 c373ed00 c03e81a0 00000286 ffff8bde 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kstriped S f6ad3f94 0 339 2 f71d5310 00000082 00000002 f6ad3f94 f6ad3f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f6ad3f98 f71d5464 c373ed00 c03e81a0 00000286 ffff8be4 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= edac-poller S f6adbf94 0 343 2 f71d4230 00000082 00000002 f6adbf94 f6adbf88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f6adbf98 f71d4384 c3734d00 c03e81a0 00000286 ffff8be3 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= scsi_eh_0 S f7c03f68 0 1229 2 f79f81b0 00000096 00000002 f7c03f68 f7c03f5c 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f7c03f6c f79f8304 c3734d00 f6aff080 f7849840 ffff8d81 00000000 f7849800 00000000 ffffffff 00000286 00000000 00000000 Call Trace: [] scsi_error_handler+0x0/0x2de [] scsi_error_handler+0x4b/0x2de [] __wake_up_common+0x46/0x68 [] _spin_unlock_irqrestore+0x34/0x39 [] scsi_error_handler+0x0/0x2de [] trace_hardirqs_on_caller+0x9d/0x113 [] scsi_error_handler+0x0/0x2de [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= scsi_eh_1 S f78e5f68 0 1230 2 f79bd690 00000096 00000002 f78e5f68 f78e5f5c 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f78e5f6c f79bd7e4 c3734d00 f7c37a80 f7c7c840 ffff8dd9 00000000 f7c7c800 00000000 ffffffff 00000286 00000000 00000000 Call Trace: [] scsi_error_handler+0x0/0x2de [] scsi_error_handler+0x4b/0x2de [] __wake_up_common+0x46/0x68 [] _spin_unlock_irqrestore+0x34/0x39 [] scsi_error_handler+0x0/0x2de [] trace_hardirqs_on_caller+0x9d/0x113 [] scsi_error_handler+0x0/0x2de [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= scsi_eh_2 S f79adf68 0 1231 2 f79dc030 00000096 00000002 f79adf68 f79adf5c 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f79adf6c f79dc184 c3734d00 f6a8ca80 f7c7c040 ffff8dea 00000000 f7c7c000 00000000 ffffffff 00000286 00000000 00000000 Call Trace: [] scsi_error_handler+0x0/0x2de [] scsi_error_handler+0x4b/0x2de [] __wake_up_common+0x46/0x68 [] _spin_unlock_irqrestore+0x34/0x39 [] scsi_error_handler+0x0/0x2de [] trace_hardirqs_on_caller+0x9d/0x113 [] scsi_error_handler+0x0/0x2de [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= scsi_eh_3 S f7c9a800 0 1232 2 f7afca20 00000096 f7c9a850 f7c9a800 00000000 00000046 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f7afca20 f7afcb74 c373ed00 f7c37300 f7c9a840 00000286 00000286 f7c9a800 00000000 c025645c 00000286 00000282 f7c9a800 Call Trace: [] __scsi_iterate_devices+0x49/0x62 [] scsi_error_handler+0x0/0x2de [] scsi_error_handler+0x0/0x2de [] scsi_error_handler+0x4b/0x2de [] __wake_up_common+0x46/0x68 [] _spin_unlock_irqrestore+0x34/0x39 [] scsi_error_handler+0x0/0x2de [] trace_hardirqs_on_caller+0x9d/0x113 [] scsi_error_handler+0x0/0x2de [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= md2_raid1 R running 0 1388 2 f79cd710 00000096 00000002 f79cbf68 f79cbf5c 00000000 f7073210 c045f0c0 c0461d00 c0461d00 c0461d00 f79cbf6c f79cd864 c373ed00 f0866d00 c0124699 ffffe204 00000000 f781be18 00000001 ffffffff 00000046 00000000 00000000 Call Trace: [] _local_bh_enable+0x44/0xa0 [] md_thread+0x0/0xda [] schedule_timeout+0x69/0xa1 [] md_thread+0xb0/0xda [] autoremove_wake_function+0x0/0x35 [] md_thread+0x0/0xda [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= md2_resync R running 0 1391 2 f70ce2b0 00000086 00000002 f7965e04 f7965df8 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f7965e08 f70ce404 c3734d00 f7969a80 c04b9e00 ffffe1ba 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] msleep_interruptible+0x22/0x2f [] sync_request+0x5c7/0x795 [] sync_request+0x178/0x795 [] sync_request+0x0/0x795 [] md_do_sync+0x320/0xdb7 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] _spin_unlock_irq+0x20/0x23 [] md_thread+0x0/0xda [] trace_hardirqs_on_caller+0x9d/0x113 [] md_thread+0x0/0xda [] md_thread+0x22/0xda [] _spin_unlock_irqrestore+0x34/0x39 [] md_thread+0x0/0xda [] trace_hardirqs_on_caller+0x9d/0x113 [] md_thread+0x0/0xda [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= md1_raid1 S f744df68 0 1398 2 f7aa68a0 00000096 00000002 f744df68 f744df5c 00000000 f715d190 c045f0c0 c0461d00 c0461d00 c0461d00 f7469e18 f7aa69f4 c3734d00 f7951800 f7aa68a0 ffffc6fd 00000046 f7469e18 00000001 f7469e08 00000046 00000000 00000002 Call Trace: [] md_thread+0x0/0xda [] schedule_timeout+0x69/0xa1 [] md_thread+0xb0/0xda [] autoremove_wake_function+0x0/0x35 [] md_thread+0x0/0xda [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= md0_raid1 S f74ebf68 0 1407 2 f7afc1b0 00000096 00000002 f74ebf68 f74ebf5c 00000000 f7aa68a0 c045f0c0 c0461d00 c0461d00 c0461d00 f748c998 f7afc304 c3734d00 f7951800 f7afc1b0 ffff9c67 00000046 f748c998 00000001 f748c988 00000046 00000000 00000002 Call Trace: [] md_thread+0x0/0xda [] schedule_timeout+0x69/0xa1 [] md_thread+0xb0/0xda [] autoremove_wake_function+0x0/0x35 [] md_thread+0x0/0xda [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kdmflush S 00000001 0 1413 2 f7a91610 00000082 f7a91990 00000001 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a91764 c373ed00 f6a8ca80 00000286 00000001 f746a29c c0319cc1 00000000 f7a91610 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kdmflush S 00000001 0 1419 2 f7a262b0 00000082 f7a26630 00000001 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a26404 c373ed00 f6a8c080 00000286 00000001 f74dc51c c0319cc1 00000000 f7a262b0 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kdmflush S f7421f94 0 1427 2 f79f8a20 00000082 00000002 f7421f94 f7421f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f7421f98 f79f8b74 c373ed00 f7955a80 00000286 ffff8e20 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kjournald R running 0 1507 2 f79e3190 00000082 00000002 f74c7f84 f74c7f78 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f74c7f88 f79e32e4 c3734d00 f7951080 00000286 ffffdf84 00000000 c0319cc1 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] kjournald+0x1bc/0x1c5 [] autoremove_wake_function+0x0/0x35 [] kjournald+0x0/0x1c5 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= udevd S f7927b14 0 1668 1 f7a5f510 00000086 00000002 f7927b14 f7927b08 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f7927b18 f7a5f664 c3734d00 f7854a80 f78c8400 ffff9ce6 00000000 00000000 00000002 ffffffff f78c8400 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] pipe_poll+0x24/0x86 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] number+0x25b/0x264 [] aoecmd_ata_rsp+0x432/0x79c [aoe] [] vsnprintf+0x2dc/0x581 [] _spin_lock+0x31/0x3c [] e1000_clean_rx_irq+0x0/0x41a [e1000] [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] _spin_unlock+0x14/0x1c [] do_wp_page+0x2a2/0x537 [] _spin_lock_irqsave+0x41/0x49 [] do_wait+0x1e7/0x342 [] do_page_fault+0xa5/0x848 [] sys_select+0x3f/0x19e [] sysenter_do_call+0x12/0x35 ======================= gfs2_scand R running 0 2409 2 f7a26b20 00000092 00000046 00000000 00000002 00000001 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a26c74 c373ed00 f65d6580 f706e000 00000282 f706e000 00000282 00000282 f7521fac f706e000 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] gfs2_scand+0x0/0x63 [gfs2] [] schedule_timeout+0x44/0xa1 [] _read_unlock+0x14/0x1c [] process_timeout+0x0/0x5 [] gfs2_scand+0x56/0x63 [gfs2] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= glock_workque S 00000001 0 2416 2 f71cc1b0 00000082 f71cc530 00000001 00000002 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f71cc304 c3734d00 f78a9800 00000286 00000001 f653619c c0319cc1 00000000 f71cc1b0 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= glock_workque S 00000001 0 2417 2 f78ef590 00000082 f78ef910 00000001 00000002 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f78ef6e4 c373ed00 f7951300 00000286 00000001 f653611c c0319cc1 00000000 f78ef590 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ksnapd S 00000001 0 2525 2 f7b04230 00000082 f7b045b0 00000001 00000002 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7b04384 c373ed00 f7843080 00000286 00000001 f65a991c c0319cc1 00000000 f7b04230 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ntpd S f44c5bf4 0 2702 1 f7b41590 00200082 00000002 f44c5bf4 f44c5be8 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f44c5bf8 f7b416e4 c3734d00 f6bdfa80 c04b9e00 ffffc01e 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] udpv6_recvmsg+0x1ac/0x335 [ipv6] [] local_bh_enable_ip+0x79/0xb5 [] udpv6_recvmsg+0x1ac/0x335 [ipv6] [] sock_common_recvmsg+0x3e/0x54 [] sock_recvmsg+0xcf/0xf3 [] sock_sendmsg+0xbc/0xde [] autoremove_wake_function+0x0/0x35 [] sys_recvfrom+0xb8/0x121 [] _spin_unlock_irq+0x20/0x23 [] finish_task_switch+0x58/0xab [] dnotify_parent+0x1f/0x5f [] vfs_write+0xe5/0x118 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= ntpd R running 0 2703 1 f787eca0 00200082 00000002 f463bbf4 f463bbe8 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f463bbf8 f787edf4 c3734d00 f7969800 c04b9e00 ffffdcab 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] do_wp_page+0x2a2/0x537 [] handle_mm_fault+0x1b3/0x8a4 [] _spin_lock+0x31/0x3c [] handle_mm_fault+0x4ea/0x8a4 [] do_page_fault+0xa5/0x848 [] up_read+0x14/0x2a [] do_page_fault+0x156/0x848 [] sock_def_readable+0x12/0x61 [] _read_lock+0x31/0x3c [] _read_unlock+0x14/0x1c [] unix_stream_sendmsg+0x1c3/0x32b [] sock_aio_write+0xcb/0xe8 [] find_lock_page+0x25/0x55 [] sock_aio_write+0x0/0xe8 [] do_sync_readv_writev+0xcb/0x107 [] __do_fault+0x192/0x3f0 [] dnotify_parent+0x1f/0x5f [] do_readv_writev+0x118/0x179 [] sock_aio_write+0x0/0xe8 [] up_read+0x14/0x2a [] do_page_fault+0x156/0x848 [] vfs_writev+0x3c/0x50 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= kjournald S f7985f84 0 2736 2 f7aa6030 00000082 00000002 f7985f84 f7985f78 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f7985f88 f7aa6184 c3734d00 f65d6580 f7aa6030 ffff98ef 00000000 f75aad10 c013d220 ffffffff 00000246 00000000 00000000 Call Trace: [] mark_held_locks+0x62/0x73 [] kjournald+0x1bc/0x1c5 [] autoremove_wake_function+0x0/0x35 [] kjournald+0x0/0x1c5 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kjournald S f64c5f84 0 2737 2 f79760b0 00000082 00000002 f64c5f84 f64c5f78 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f64c5f88 f7976204 c3734d00 f7843580 00000286 ffffe04c 00000000 c0319cc1 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] kjournald+0x1bc/0x1c5 [] autoremove_wake_function+0x0/0x35 [] kjournald+0x0/0x1c5 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kjournald R running 0 2738 2 f78763b0 00000082 00000002 f65c1f84 f65c1f78 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f65c1f88 f7876504 c3734d00 f6a8c800 00000286 ffffe1fd 00000000 c0319cc1 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] kjournald+0x1bc/0x1c5 [] autoremove_wake_function+0x0/0x35 [] kjournald+0x0/0x1c5 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= getty S f6407eb4 0 3060 1 f711a430 00000082 00000002 f6407eb4 f6407ea8 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f6407eb8 f711a584 c3734d00 f7854580 00000000 ffff96f1 00000000 00000046 f64d697c ffffffff f756d000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] mark_held_locks+0x62/0x73 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= getty S 00388000 0 3061 1 f7a27390 00000082 00000001 00388000 00000000 c013eb96 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a274e4 c3734d00 f7c37300 00000000 f7a27390 c03182b9 00000046 f44c097c f44c0800 f65f1000 00000046 00000000 Call Trace: [] __lock_acquire+0x27b/0x932 [] mutex_lock_interruptible_nested+0x192/0x2b7 [] schedule_timeout+0x69/0xa1 [] mark_held_locks+0x62/0x73 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= login S 00000002 0 3063 1 f715c0b0 00000082 f715c0b0 00000002 00000000 c040c990 f715c0b0 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f715c204 c3734d00 f649c080 00000000 00000002 00000002 00000000 00000000 c01228e1 00000246 00000246 c040c980 Call Trace: [] do_wait+0x96/0x342 [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= getty S 00388000 0 3065 1 f7acf190 00000082 00000001 00388000 00000000 c013eb96 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7acf2e4 c3734d00 f6417800 00000000 f7acf190 c03182b9 00000046 f44c017c f44c0000 f781c000 00000046 00000000 Call Trace: [] __lock_acquire+0x27b/0x932 [] mutex_lock_interruptible_nested+0x192/0x2b7 [] schedule_timeout+0x69/0xa1 [] mark_held_locks+0x62/0x73 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= login S 00000002 0 3067 1 f78fce20 00000082 f78fce20 00000002 00000000 c040c990 f78fce20 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f78fcf74 c3734d00 f6a8c580 00000000 00000002 00000002 00000000 00000000 c01228e1 00000246 00000246 c040c980 Call Trace: [] do_wait+0x96/0x342 [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= getty S 00388000 0 3068 1 f7ad7210 00000082 00000001 00388000 00000000 c013eb96 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7ad7364 c3734d00 f649c300 00000000 f7ad7210 c03182b9 00000046 f644617c f6446000 f78c3000 00000046 00000000 Call Trace: [] __lock_acquire+0x27b/0x932 [] mutex_lock_interruptible_nested+0x192/0x2b7 [] schedule_timeout+0x69/0xa1 [] mark_held_locks+0x62/0x73 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= syslogd R running 0 3112 1 f7b17410 00000086 f7448c1c f7b17410 00000002 00000000 f6ddc32c c045f0c0 c0461d00 c0461d00 c0461d00 00000020 f7b17564 c3734d00 f6a8c800 00000000 00000000 c030b41c 00000246 00000246 f7448c0c f7448c0c 00000000 00000020 Call Trace: [] unix_peer_get+0x11/0x2b [] _spin_unlock+0x14/0x1c [] schedule_timeout+0x69/0xa1 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] __ext3_get_inode_loc+0xba/0x307 [] __ext3_journal_dirty_metadata+0x16/0x3b [] _spin_lock_irqsave+0x41/0x49 [] journal_stop+0x134/0x1c3 [] ext3_ordered_write_end+0xf0/0x14a [] generic_file_buffered_write+0x1a6/0x677 [] mnt_drop_write+0x1d/0xd8 [] _spin_lock+0x31/0x3c [] pipe_write+0x5c/0x450 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] pipe_write+0x0/0x450 [] do_sync_readv_writev+0xcb/0x107 [] mntput_no_expire+0x18/0xef [] autoremove_wake_function+0x0/0x35 [] copy_from_user+0x2d/0x59 [] rw_copy_check_uvector+0x64/0xf4 [] sys_select+0x3f/0x19e [] sigprocmask+0x68/0xc3 [] sys_rt_sigprocmask+0xbf/0xd4 [] sysenter_do_call+0x12/0x35 ======================= dd S f799df04 0 3132 1 f5e6e630 00000086 00000002 f799df04 f799def8 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f799df08 f5e6e784 c373ed00 f753da80 00000282 ffff96b5 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] do_syslog+0x278/0x399 [] autoremove_wake_function+0x0/0x35 [] kmsg_read+0x0/0x36 [] proc_reg_read+0x58/0x79 [] proc_reg_read+0x0/0x79 [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= klogd S 00000000 0 3133 1 f79e2920 00000092 f79e2cb8 00000000 f79e2ca0 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79e2a74 c3734d00 f7955d00 f79e2920 00000000 00000000 f79e2920 c0318467 f647c488 c013d34e f647c484 f647c4a4 Call Trace: [] __mutex_unlock_slowpath+0x89/0xf9 [] trace_hardirqs_on_caller+0x9d/0x113 [] pipe_wait+0x50/0x72 [] autoremove_wake_function+0x0/0x35 [] pipe_read+0xe3/0x3fe [] do_sync_read+0xd2/0x10e [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] do_sync_read+0x0/0x10e [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= sshd S 00000000 0 3156 1 f71d4aa0 00200086 00000042 00000000 00000042 00000002 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 00000032 f71d4bf4 c373ed00 f7951300 f780b080 00000020 00200046 00000000 00000002 00000001 f6c0581c 00200282 f780b080 Call Trace: [] schedule_timeout+0x69/0xa1 [] tcp_poll+0x17/0x11e [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] __find_get_block+0x8c/0x18c [] trace_hardirqs_on_caller+0x9d/0x113 [] validate_chain+0x395/0xf35 [] __find_get_block+0x99/0x18c [] validate_chain+0x395/0xf35 [] kmap_atomic+0x1c/0x21 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] mark_held_locks+0x62/0x73 [] get_page_from_freelist+0x304/0x48d [] trace_hardirqs_on_caller+0x9d/0x113 [] __lock_acquire+0x27b/0x932 [] mark_held_locks+0x62/0x73 [] _spin_unlock_irq+0x20/0x23 [] trace_hardirqs_on_caller+0x9d/0x113 [] _spin_unlock_irq+0x20/0x23 [] add_to_page_cache_locked+0x70/0x8f [] kmap_atomic+0x1c/0x21 [] shmem_getpage+0x4e9/0x661 [] kmap_atomic+0x1c/0x21 [] __lock_acquire+0x27b/0x932 [] __lock_acquire+0x27b/0x932 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] __do_fault+0x2bc/0x3f0 [] __do_fault+0x192/0x3f0 [] trace_hardirqs_on_caller+0x9d/0x113 [] handle_mm_fault+0x176/0x8a4 [] do_page_fault+0xa5/0x848 [] sys_select+0x3f/0x19e [] trace_hardirqs_on_thunk+0xc/0x10 [] do_page_fault+0x0/0x848 [] trace_hardirqs_on_caller+0x9d/0x113 [] sysenter_do_call+0x12/0x35 ======================= mysqld_safe S f64a9f24 0 3204 1 f5e6eea0 00000082 00000002 f64a9f24 f64a9f18 00000000 f5e6eea0 c045f0c0 c0461d00 c0461d00 c0461d00 f64a9f28 f5e6eff4 c3734d00 f753d080 00000000 ffff970e 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= mysqld S f50e9b14 0 3246 3204 f798e230 00000086 00000002 f50e9b14 00000002 00000000 f545082c c045f0c0 c0461d00 c0461d00 c0461d00 f50e9b18 f798e384 c373ed00 f65d6580 00000000 00000046 00000000 00000002 00000001 f545081c 00000282 f6567180 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] schedule_timeout+0x69/0xa1 [] unix_poll+0x1a/0xab [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] __lock_acquire+0x27b/0x932 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] __lock_acquire+0x27b/0x932 [] mark_held_locks+0x62/0x73 [] kmem_cache_free+0x4e/0x62 [] trace_hardirqs_on_caller+0x9d/0x113 [] journal_stop+0x134/0x1c3 [] validate_chain+0x395/0xf35 [] ext3_ordered_write_end+0xf0/0x14a [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] enqueue_task_fair+0xa2/0xb4 [] wake_up_new_task+0x87/0xa0 [] do_fork+0xce/0x247 [] sys_select+0x3f/0x19e [] sysenter_do_call+0x12/0x35 ======================= mysqld S f649fdf8 0 3249 3204 f7119490 00000096 00000002 f649fdf8 f649fdec 00000000 f7119490 c045f0c0 c0461d00 c0461d00 c0461d00 f649fdfc f71195e4 c373ed00 f65d6580 f649fe3c ffff972b 00000000 09b7a9f8 c0319cc1 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] futex_wait+0x318/0x396 [] __lock_acquire+0x27b/0x932 [] __lock_acquire+0x27b/0x932 [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] enqueue_task_fair+0xa2/0xb4 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] rwsem_wake+0x4d/0x116 [] call_rwsem_wake+0xa/0xc [] up_read+0x26/0x2a [] sys_futex+0x86/0x111 [] trace_hardirqs_on_thunk+0xc/0x10 [] trace_hardirqs_on_caller+0x9d/0x113 [] sysenter_do_call+0x12/0x35 ======================= mysqld S 00000002 0 3250 3204 f7a5eca0 00000096 f7a5eca0 00000002 00000000 c082d424 f7a5eca0 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f7a5edf4 c3734d00 f65d6580 f4c1be3c 00000282 00000000 09b7aa68 c0319cc1 00000046 00000046 f4c1be3c 00000282 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] futex_wait+0x318/0x396 [] try_to_wake_up+0xa8/0x137 [] _spin_unlock+0x14/0x1c [] futex_requeue+0xdb/0x251 [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] find_get_pages_tag+0x11/0x140 [] ext3_write_inode+0x0/0x34 [] mapping_tagged+0x58/0x60 [] pagevec_lookup_tag+0x24/0x2c [] wait_on_page_writeback_range+0x5f/0x104 [] _spin_unlock+0x14/0x1c [] sync_inode+0x25/0x2a [] sys_futex+0x86/0x111 [] do_fsync+0x73/0x92 [] sysenter_do_call+0x12/0x35 ======================= mysqld S f787e7b0 0 3251 3204 f787e430 00000096 f787e430 f787e7b0 00000001 c082c8f8 f787e430 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f787e584 c3734d00 f65d6580 f6573e3c 00000282 00000000 09b7aad8 c0319cc1 f787e430 c0319d8f 00000000 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] futex_wait+0x318/0x396 [] __alloc_pages_internal+0xb6/0x41f [] validate_chain+0x395/0xf35 [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] __lock_acquire+0x27b/0x932 [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] sys_futex+0x86/0x111 [] vfs_read+0xe3/0x11b [] trace_hardirqs_on_thunk+0xc/0x10 [] trace_hardirqs_on_caller+0x9d/0x113 [] sysenter_do_call+0x12/0x35 ======================= mysqld S f4defdf8 0 3252 3204 f798eaa0 00000096 00000002 f4defdf8 f4defdec 00000000 f798eaa0 c045f0c0 c0461d00 c0461d00 c0461d00 f4defdfc f798ebf4 c373ed00 f65d6580 f4defe3c ffff9c9c 00000000 09b7ab48 c0319cc1 ffffffff 00000046 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] futex_wait+0x318/0x396 [] try_to_wake_up+0xa8/0x137 [] _spin_unlock+0x14/0x1c [] futex_requeue+0xdb/0x251 [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] sys_futex+0x86/0x111 [] vfs_write+0xe5/0x118 [] sysenter_do_call+0x12/0x35 ======================= mysqld R running 0 3254 3204 f7afd290 00000086 00000046 00000000 00000002 00000001 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7afd3e4 c373ed00 f65d6580 f706e000 00000286 f706e000 00000286 00000286 f758bb38 f706e000 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __lock_acquire+0x27b/0x932 [] validate_chain+0x395/0xf35 [] mark_held_locks+0x62/0x73 [] get_page_from_freelist+0x304/0x48d [] __pollwait+0x0/0xba [] free_pages_bulk+0x2f/0x2b0 [] _spin_lock+0x31/0x3c [] free_pages_bulk+0xda/0x2b0 [] bictcp_cong_avoid+0x12/0x35b [] tcp_ack+0xa26/0x1cb5 [] __do_fault+0x2fb/0x3f0 [] e1000_clean_rx_irq+0x0/0x41a [e1000] [] e1000_clean+0x450/0x5f9 [e1000] [] net_rx_action+0x85/0x1db [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] e1000_clean_rx_irq+0x0/0x41a [e1000] [] e1000_clean+0x450/0x5f9 [e1000] [] _spin_lock_irqsave+0x41/0x49 [] scsi_end_request+0x52/0x75 [] net_rx_action+0x85/0x1db [] net_rx_action+0x9d/0x1db [] sys_select+0xd6/0x19e [] sysenter_do_call+0x12/0x35 ======================= mysqld R running 0 3255 3204 f7976920 00000086 00000002 f65d9b14 f65d9b08 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f65d9b18 f7976a74 c3734d00 f65d6580 c04b9e00 ffffe172 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __lock_acquire+0x27b/0x932 [] validate_chain+0x395/0xf35 [] trace_hardirqs_on_caller+0x9d/0x113 [] __pollwait+0x0/0xba [] enqueue_task_fair+0xa2/0xb4 [] try_to_wake_up+0xa8/0x137 [] autoremove_wake_function+0x14/0x35 [] __wake_up_common+0x46/0x68 [] enqueue_task_fair+0xa2/0xb4 [] try_to_wake_up+0xa8/0x137 [] autoremove_wake_function+0x14/0x35 [] __wake_up_common+0x46/0x68 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] _spin_lock_irqsave+0x41/0x49 [] _spin_lock_irqsave+0x41/0x49 [] scsi_next_command+0x25/0x2f [] scsi_end_request+0x52/0x75 [] scsi_io_completion+0x87/0x3de [] _spin_lock+0x31/0x3c [] sys_select+0xd6/0x19e [] sysenter_do_call+0x12/0x35 ======================= mysqld S f45c1df8 0 3256 3204 f711e530 00000096 00000002 f45c1df8 f45c1dec 00000000 f711e530 c045f0c0 c0461d00 c0461d00 c0461d00 f45c1dfc f711e684 c3734d00 f65d6580 f45c1e3c ffffde9a 00000000 098350b0 c0319cc1 ffffffff 00000046 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] futex_wait+0x318/0x396 [] futex_wake+0x52/0xe1 [] _spin_unlock+0x14/0x1c [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] __dequeue_entity+0x45/0xaa [] set_next_entity+0x14/0x38 [] _spin_unlock_irq+0x20/0x23 [] schedule+0x301/0x708 [] sys_futex+0x86/0x111 [] sysenter_do_call+0x12/0x35 ======================= mysqld S f4c77ecc 0 3257 3204 f715c920 00000082 00000002 f4c77ecc f4c77ec0 00000000 f7073210 c045f0c0 c0461d00 c0461d00 c0461d00 f4c77ed0 f715ca74 c373ed00 f65d6580 00000000 ffffd39a 00000000 c373bf30 f715c920 ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] dequeue_signal+0x3c/0x12a [] sys_rt_sigtimedwait+0x87/0x285 [] sys_rt_sigtimedwait+0x145/0x285 [] sigprocmask+0x1f/0xc3 [] _spin_unlock_irq+0x20/0x23 [] sigprocmask+0x68/0xc3 [] sys_rt_sigprocmask+0xbf/0xd4 [] sysenter_do_call+0x12/0x35 ======================= mysqld S c013eb96 0 3260 3204 f7118c20 00000082 00000000 c013eb96 00000001 00000169 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f7118c20 f7118d74 c3734d00 f65d6580 00000002 00000246 f74012bc f7401280 f74012ac 00000246 00000000 00000002 00000000 Call Trace: [] __lock_acquire+0x27b/0x932 [] schedule_timeout+0x69/0xa1 [] local_bh_enable_ip+0x79/0xb5 [] trace_hardirqs_on_caller+0x9d/0x113 [] sk_wait_data+0x54/0xa1 [] sk_wait_data+0x69/0xa1 [] autoremove_wake_function+0x0/0x35 [] tcp_recvmsg+0x42c/0x7ff [] sock_common_recvmsg+0x3e/0x54 [] sock_aio_read+0xdb/0xef [] do_sync_read+0xd2/0x10e [] __lock_acquire+0x27b/0x932 [] autoremove_wake_function+0x0/0x35 [] vfs_read+0x114/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= mysqld S f7b166b0 0 3261 3204 f7b16330 00000096 f7b16330 f7b166b0 00000001 c082c664 f7b16330 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f7b16484 c3734d00 f65d6580 f75bde3c 00000282 00000000 09c19cf4 c0319cc1 f7b16330 c0319d8f 00000000 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] futex_wait+0x318/0x396 [] __generic_file_aio_write_nolock+0x285/0x4df [] generic_file_aio_write+0x73/0xd8 [] validate_chain+0x395/0xf35 [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] __lock_acquire+0x27b/0x932 [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] sys_futex+0x86/0x111 [] trace_hardirqs_on_thunk+0xc/0x10 [] trace_hardirqs_on_caller+0x9d/0x113 [] sysenter_do_call+0x12/0x35 ======================= logger S 00000000 0 3247 3204 f7a01310 00000092 f7a016a8 00000000 f7a01690 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a01464 c3734d00 f6aff300 f7a01310 00000000 00000000 f7a01310 c0318467 f59e599c c013d34e f59e5998 f59e59b8 Call Trace: [] __mutex_unlock_slowpath+0x89/0xf9 [] trace_hardirqs_on_caller+0x9d/0x113 [] pipe_wait+0x50/0x72 [] autoremove_wake_function+0x0/0x35 [] pipe_read+0xe3/0x3fe [] __lock_acquire+0x27b/0x932 [] do_sync_read+0xd2/0x10e [] autoremove_wake_function+0x0/0x35 [] finish_task_switch+0x26/0xab [] schedule+0x228/0x708 [] do_sync_read+0x0/0x10e [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= master R running 0 3375 1 f70f2ba0 00000082 00000046 00000046 f78a5b9c 00000000 f78a5b9c c045f0c0 c0461d00 c0461d00 c0461d00 f783e980 f70f2cf8 c3734d00 f7951d00 f78a5b9c f78a5b80 f4cfc530 00000304 fffffff7 00000019 0000001b bfdaec38 c019a8d9 Call Trace: [] ep_ptable_queue_proc+0x0/0x6c [] work_resched+0x5/0x2a ======================= pickup S f6481f1c 0 3394 3375 f7b04aa0 00200086 00000002 f6481f1c f6481f10 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f6481f20 f7b04bf4 c3734d00 f7955080 c04b9e00 ffffde4a 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= qmgr R running 0 3395 3375 f7a9f710 00200086 00200046 00000000 00000002 00000001 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a9f864 c373ed00 f6aff800 f706e000 00200282 f706e000 00200282 00200282 f6409f40 f706e000 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= mdadm S 00000000 0 3422 1 f715d190 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f715d2e4 c3734d00 f7951800 c04b9e00 00000286 c04b9e00 00000286 00000286 f7981b38 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] __rmqueue_smallest+0xcc/0x13e [] validate_chain+0x395/0xf35 [] __rmqueue+0x12/0x1df [] rmqueue_bulk+0x26/0x71 [] mark_held_locks+0x62/0x73 [] __lock_acquire+0x27b/0x932 [] _spin_lock_irqsave+0x41/0x49 [] md_wakeup_thread+0x25/0x29 [] md_ioctl+0x10c/0x1082 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] do_open+0xc5/0x2b7 [] blkdev_open+0x0/0x53 [] blkdev_open+0x25/0x53 [] __dentry_open+0x176/0x20a [] nameidata_to_filp+0x35/0x3f [] mutex_lock_nested+0x18c/0x25c [] file_kill+0x13/0x2d [] sys_select+0xd6/0x19e [] filp_close+0x3e/0x62 [] sysenter_do_call+0x12/0x35 ======================= ccsd S f6451b14 0 3480 1 f79bce20 00000086 00000002 f6451b14 f6451b08 00000000 f54a182c c045f0c0 c0461d00 c0461d00 c0461d00 f6451b18 f79bcf74 c373ed00 f7951580 00000000 ffff9cbf 00000000 00000002 00000001 ffffffff 00000282 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] unix_poll+0x1a/0xab [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] sched_clock_cpu+0x10f/0x158 [] update_curr+0x40/0x52 [] try_to_wake_up+0xa8/0x137 [] sched_clock_cpu+0x10f/0x158 [] update_curr+0x40/0x52 [] try_to_wake_up+0xa8/0x137 [] autoremove_wake_function+0x14/0x35 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] skb_dequeue+0x50/0x56 [] sys_select+0x3f/0x19e [] filp_close+0x3e/0x62 [] sysenter_do_call+0x12/0x35 ======================= ccsd S f459db14 0 3481 1 f794d110 00000086 00000002 f459db14 f459db08 00000000 f567bd2c c045f0c0 c0461d00 c0461d00 c0461d00 f459db18 f794d264 c3734d00 f7951580 00000000 ffff98ac 00000000 00000002 00000001 ffffffff 00000282 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] unix_poll+0x1a/0xab [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] e1000_clean+0x336/0x5cd [e1000e] [] __lock_acquire+0x27b/0x932 [] sched_clock_cpu+0x10f/0x158 [] update_curr+0x40/0x52 [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] __lock_acquire+0x27b/0x932 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] validate_chain+0x395/0xf35 [] sock_aio_read+0xdb/0xef [] validate_chain+0x395/0xf35 [] sockfd_lookup_light+0x24/0x41 [] __lock_acquire+0x27b/0x932 [] _spin_unlock_irq+0x20/0x23 [] trace_hardirqs_on_caller+0x9d/0x113 [] _spin_unlock_irq+0x20/0x23 [] finish_task_switch+0x58/0xab [] finish_task_switch+0x26/0xab [] schedule+0x228/0x708 [] dnotify_parent+0x1f/0x5f [] sys_select+0x3f/0x19e [] trace_hardirqs_on_thunk+0xc/0x10 [] trace_hardirqs_on_caller+0x9d/0x113 [] sysenter_do_call+0x12/0x35 ======================= aisexec R running 0 3493 1 f71cca20 00000082 00000002 f50fdbf4 f50fdbe8 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f50fdbf8 f71ccb74 c3734d00 f7951080 c04b9e00 ffffe1e3 00000000 00000282 00000282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] read_tsc+0x6/0x28 [] getnstimeofday+0x44/0x100 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f65ddbf4 0 3494 1 f79dd110 00000082 00000002 f65ddbf4 f65ddbe8 00000000 f78fd690 c045f0c0 c0461d00 c0461d00 c0461d00 f65ddbf8 f79dd264 c373ed00 f7951080 00000100 ffff985b 00000000 00000002 c0555fa0 ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] __do_fault+0x2bc/0x3f0 [] validate_chain+0x395/0xf35 [] __do_fault+0x2fb/0x3f0 [] __lock_acquire+0x27b/0x932 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] find_usage_backwards+0xc/0xce [] find_usage_backwards+0xc/0xce [] find_usage_backwards+0xc/0xce [] __lock_acquire+0x27b/0x932 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] __do_fault+0x2bc/0x3f0 [] __do_fault+0x192/0x3f0 [] handle_mm_fault+0x176/0x8a4 [] do_page_fault+0xa5/0x848 [] up_read+0x14/0x2a [] do_page_fault+0x156/0x848 [] trace_hardirqs_on_thunk+0xc/0x10 [] do_page_fault+0x0/0x848 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f642ddf8 0 3495 1 f711b510 00000096 00000002 f642ddf8 f642ddec 00000000 f711b510 c045f0c0 c0461d00 c0461d00 c0461d00 f642ddfc f711b664 c373ed00 f7951080 f642de3c ffff98b0 00000000 0a1244c0 c0319cc1 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] futex_wait+0x318/0x396 [] __generic_file_aio_write_nolock+0x285/0x4df [] generic_file_aio_write+0x73/0xd8 [] futex_wake+0x52/0xe1 [] validate_chain+0x395/0xf35 [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] __lock_acquire+0x27b/0x932 [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] sys_futex+0x86/0x111 [] trace_hardirqs_on_thunk+0xc/0x10 [] trace_hardirqs_on_caller+0x9d/0x113 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f45bfd7c 0 3526 1 f78fc5b0 00000086 00000002 f45bfd7c f45bfd70 00000000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 f45bfd80 f78fc704 c373ed00 f7951080 00000000 ffff9cbf 00000000 00000000 f45be000 ffffffff 00000246 00000000 00000000 Call Trace: [] sys_semtimedop+0x5a8/0x7a4 [] copy_from_user+0x2d/0x59 [] copy_from_user+0x2d/0x59 [] verify_iovec+0x2a/0x8a [] sys_sendmsg+0x232/0x237 [] __dequeue_entity+0x45/0xaa [] set_next_entity+0x14/0x38 [] pick_next_task_fair+0x8f/0xa8 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] sys_socketcall+0xb7/0x2a9 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f75dde98 0 3543 1 f7ace0b0 00000086 00000246 f75dde98 00000000 f4dea000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f7ace204 c373ed00 f7951080 00000000 f75dde88 00000001 00000000 f4dea000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] copy_from_user+0x2d/0x59 [] copy_from_user+0x2d/0x59 [] verify_iovec+0x2a/0x8a [] sys_sendmsg+0x232/0x237 [] _spin_lock_irqsave+0x41/0x49 [] scsi_next_command+0x25/0x2f [] scsi_end_request+0x52/0x75 [] scsi_io_completion+0x87/0x3de [] _spin_lock+0x31/0x3c [] sys_socketcall+0xb7/0x2a9 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f4571d7c 0 3547 1 f70f3410 00000086 00000002 f4571d7c f4571d70 00000000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 f4571d80 f70f3564 c373ed00 f7951080 00000000 ffff9cbe 00000000 00000000 f4570000 ffffffff 00000246 00000000 00000000 Call Trace: [] sys_semtimedop+0x5a8/0x7a4 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f7410098 0 3553 1 f5e6f710 00000086 00000246 f7410098 00000000 f4cc6000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f5e6f864 c3734d00 f7951080 00000000 f7410088 00000003 00000000 f4cc6000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] copy_from_user+0x2d/0x59 [] copy_from_user+0x2d/0x59 [] verify_iovec+0x2a/0x8a [] sys_sendmsg+0x232/0x237 [] sys_socketcall+0xb7/0x2a9 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f4dd1d7c 0 3651 1 f71cd290 00000086 00000002 f4dd1d7c f4dd1d70 00000000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 f4dd1d80 f71cd3e4 c373ed00 f7951080 00000000 ffff9be4 00000000 00000000 f4dd0000 ffffffff 00000246 00000000 00000000 Call Trace: [] sys_semtimedop+0x5a8/0x7a4 [] copy_from_user+0x2d/0x59 [] copy_from_user+0x2d/0x59 [] verify_iovec+0x2a/0x8a [] sys_sendmsg+0x232/0x237 [] sys_socketcall+0xb7/0x2a9 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f6470b98 0 3689 1 f7a563b0 00000086 00000246 f6470b98 00000000 f44e2000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f7a56504 c3734d00 f7951080 00000000 f6470b88 00000005 00000000 f44e2000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] copy_from_user+0x2d/0x59 [] copy_from_user+0x2d/0x59 [] verify_iovec+0x2a/0x8a [] sys_sendmsg+0x232/0x237 [] _spin_lock_irqsave+0x41/0x49 [] __dequeue_entity+0x45/0xaa [] update_curr+0x40/0x52 [] set_next_entity+0x14/0x38 [] pick_next_task_fair+0x8f/0xa8 [] sys_socketcall+0xb7/0x2a9 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f4c44c98 0 3698 1 f7a61590 00000086 00000246 f4c44c98 00000000 f4d5a000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f7a616e4 c373ed00 f7951080 00000000 f4c44c88 00000006 00000000 f4d5a000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] enqueue_task_fair+0xa2/0xb4 [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] __dequeue_entity+0x45/0xaa [] update_curr+0x40/0x52 [] set_next_entity+0x14/0x38 [] pick_next_task_fair+0x8f/0xa8 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f7828098 0 3710 1 f71dc2b0 00000086 00000246 f7828098 00000000 f4c04000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f71dc404 c373ed00 f7951080 00000000 f7828088 00000007 00000000 f4c04000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] __dequeue_entity+0x45/0xaa [] set_next_entity+0x14/0x38 [] pick_next_task_fair+0x8f/0xa8 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f75dd998 0 3717 1 f797e130 00000086 00000246 f75dd998 00000000 f48a8000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f797e284 c3734d00 f7951080 00000000 f75dd988 00000008 00000000 f48a8000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= smtpd S 00000000 0 3508 3375 f7ad6130 00200082 00200046 00000000 00000002 00000001 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7ad6284 c373ed00 f7c37580 f706e000 00200282 f706e000 00200282 00200282 f654fc18 f706e000 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] ip_finish_output+0x112/0x26b [] ip_local_out+0x15/0x17 [] ip_queue_xmit+0x195/0x327 [] resched_task+0x56/0x5c [] try_to_wake_up+0xa8/0x137 [] tcp_transmit_skb+0x363/0x659 [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] _read_unlock+0x14/0x1c [] unix_stream_sendmsg+0x1c3/0x32b [] sock_aio_write+0xcb/0xe8 [] do_sync_write+0xd2/0x10e [] cp_new_stat64+0xf9/0x10b [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] read_tsc+0x6/0x28 [] getnstimeofday+0x44/0x100 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= anvil S 00000000 0 3523 3375 f786f410 00200086 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f786f564 c3734d00 f6bdfd00 c04b9e00 00200282 c04b9e00 00200282 00200282 f654bf40 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= groupd S f74b3bf4 0 3524 1 f788e530 00000082 00000002 f74b3bf4 f74b3be8 00000000 f70680b0 c045f0c0 c0461d00 c0461d00 c0461d00 f74b3bf8 f788e684 c373ed00 f7854d00 f74b3efc ffff9cbf 00000000 00000002 00000001 ffffffff 00000286 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] unix_poll+0x1a/0xab [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] _read_unlock+0x14/0x1c [] unix_stream_sendmsg+0x1c3/0x32b [] sock_aio_write+0xcb/0xe8 [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] vfs_write+0xe5/0x118 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= fenced S f456fbf4 0 3529 1 f7b16ba0 00000082 00000002 f456fbf4 f456fbe8 00000000 f55740ac c045f0c0 c0461d00 c0461d00 c0461d00 f456fbf8 f7b16cf4 c3734d00 f78a9d00 f456febc ffff9a0e 00000000 00000002 00000001 ffffffff 00000286 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] unix_poll+0x1a/0xab [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] sched_clock_cpu+0x10f/0x158 [] update_curr+0x40/0x52 [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] unix_stream_recvmsg+0x3a0/0x4d8 [] sock_aio_read+0xdb/0xef [] sockfd_lookup_light+0x24/0x41 [] skb_dequeue+0x50/0x56 [] dput+0x97/0x145 [] mntput_no_expire+0x18/0xef [] filp_close+0x3e/0x62 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= dlm_controld S f65b3bf4 0 3533 1 f71183b0 00000082 00000002 f65b3bf4 f65b3be8 00000000 f4001d2c c045f0c0 c0461d00 c0461d00 c0461d00 f65b3bf8 f7118504 c373ed00 f7854800 f65b3ebc ffff9cf2 00000000 00000002 00000001 ffffffff 00000286 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] unix_poll+0x1a/0xab [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] _spin_lock_irqsave+0x41/0x49 [] netlink_recvmsg+0x1d1/0x294 [] __wake_up_common+0x46/0x68 [] sock_recvmsg+0xcf/0xf3 [] autoremove_wake_function+0x0/0x35 [] sock_aio_read+0xdb/0xef [] sockfd_lookup_light+0x24/0x41 [] do_sync_read+0xd2/0x10e [] autoremove_wake_function+0x0/0x35 [] sys_recv+0x37/0x3b [] sys_socketcall+0x1a9/0x2a9 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= gfs_controld S f750dbf4 0 3537 1 f79bc5b0 00000082 00000002 f750dbf4 f750dbe8 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f750dbf8 f79bc704 c373ed00 f7969580 f79bc5b0 ffffdfb5 00000000 f70fb180 f750ded4 ffffffff f8939698 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] _spin_unlock+0x14/0x1c [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] ipc_lock+0x60/0xb8 [] _spin_lock+0x31/0x3c [] find_get_page+0xa2/0xbf [] _spin_lock_irq+0x37/0x42 [] futex_wake+0x52/0xe1 [] do_futex+0x7a/0x951 [] handle_mm_fault+0x176/0x8a4 [] do_page_fault+0xa5/0x848 [] up_read+0x14/0x2a [] sys_futex+0x86/0x111 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= clvmd R running 0 3644 1 f7115590 00000086 00000002 f65c3b14 f65c3b08 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f65c3b18 f71156e4 c3734d00 f7854080 c04b9e00 ffffcb5b 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] _spin_unlock_irq+0x20/0x23 [] finish_task_switch+0x58/0xab [] finish_task_switch+0x26/0xab [] schedule+0x228/0x708 [] sched_clock_cpu+0x10f/0x158 [] update_curr+0x40/0x52 [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] unix_stream_recvmsg+0x3a0/0x4d8 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] sock_aio_read+0xdb/0xef [] sockfd_lookup_light+0x24/0x41 [] sys_recvfrom+0x11f/0x121 [] _spin_unlock_irq+0x20/0x23 [] finish_task_switch+0x58/0xab [] finish_task_switch+0x26/0xab [] schedule+0x228/0x708 [] dnotify_parent+0x1f/0x5f [] sys_select+0xd6/0x19e [] sysenter_do_call+0x12/0x35 ======================= clvmd S 00000002 0 3655 1 f71205b0 00000086 f71205b0 00000002 00000000 f45e3e20 f71205b0 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f7120704 c3734d00 f7854080 f45e3e58 00000282 f45e3e58 b7df4368 c0319cc1 00000046 00000246 00000246 f45e3e10 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] device_read+0x12d/0x347 [dlm] [] default_wake_function+0x0/0x8 [] device_read+0x0/0x347 [dlm] [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= clvmd S c373baa0 0 3656 1 f7a5e430 00000096 000006e2 c373baa0 00000002 f45e9df4 f7a5e430 c045f0c0 c0461d00 c0461d00 c0461d00 f45e9dfc f7a5e584 c373ed00 f7854080 f45e9e3c ffff9c79 00000000 080b1fa4 c0319cc1 ffffffff 00000046 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] futex_wait+0x318/0x396 [] _spin_unlock+0x14/0x1c [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] _spin_unlock_irq+0x20/0x23 [] finish_task_switch+0x58/0xab [] finish_task_switch+0x26/0xab [] schedule+0x228/0x708 [] sys_futex+0x86/0x111 [] up_write+0x14/0x28 [] sysenter_do_call+0x12/0x35 ======================= dlm_astd S c3731aa0 0 3645 2 f78ee4b0 00000092 00000400 c3731aa0 00000002 f4835fa4 f78ee4b0 c045f0c0 c0461d00 c0461d00 c0461d00 f4835fac f78ee604 c3734d00 f7951d00 00000246 ffffe1fd 00000000 00000046 00000046 ffffffff 00000000 00000000 00000000 Call Trace: [] gdlm_bast+0x0/0xa4 [gfs] [] dlm_astd+0x62/0x155 [dlm] [] dlm_astd+0x0/0x155 [dlm] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_scand R running 0 3646 2 f78fd690 00000092 00000002 f4837f88 f4837f7c 00000000 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f4837f8c f78fd7e4 c373ed00 f7854300 f706e000 ffffe028 00000000 00000282 00000282 ffffffff f706e000 00000000 00000000 Call Trace: [] dlm_scand+0x0/0x77 [dlm] [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] dlm_scand+0x69/0x77 [dlm] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_recv/0 R running 0 3647 2 f7a00230 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a00384 c3734d00 f7951d00 00000286 00000001 f45e341c c0319cc1 00000000 00000046 00000046 f45e3424 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_recv/1 S 00000000 0 3648 2 f711aca0 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f711adf4 c373ed00 f7969d00 00000286 00000001 f45e349c c0319cc1 00000000 00000046 00000046 f45e34a4 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_send S 00000000 0 3649 2 f79e20b0 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79e2204 c3734d00 f7969a80 00000286 00000001 f4562d1c c0319cc1 00000000 00000046 00000046 f4562d24 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_recoverd S f483ff80 0 3650 2 f78eed20 00000086 00000002 f483ff80 f483ff74 00000000 00002bd4 c045f0c0 c0461d00 c0461d00 c0461d00 f483ff84 f78eee74 c3734d00 f7951080 00000002 ffff9c07 00000000 f78eed20 00000046 ffffffff f7001280 00000000 00000000 Call Trace: [] dlm_recoverd+0x163/0x55a [dlm] [] __wake_up_common+0x46/0x68 [] dlm_recoverd+0x0/0x55a [dlm] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kdmflush S 00000000 0 3663 2 f71dcb20 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f71dcc74 c373ed00 f6417d00 00000286 00000001 f50cab9c c0319cc1 00000000 00000046 00000046 f50caba4 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kdmflush S 00000000 0 3669 2 f79dc8a0 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79dc9f4 c373ed00 f6417d00 00000286 00000001 f481761c c0319cc1 00000000 00000046 00000046 f4817624 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= lock_dlm1 S f4cebf50 0 3692 2 f7a56c20 00000086 00000002 f4cebf50 f4cebf44 00000000 f7a56c20 c045f0c0 c0461d00 c0461d00 c0461d00 f4cebf54 f7a56d74 c3734d00 f7951080 00000246 ffffaeb4 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] gdlm_thread+0x112/0x74a [gfs] [] autoremove_wake_function+0x0/0x35 [] gdlm_thread1+0x0/0xa [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= lock_dlm2 S 00000002 0 3693 2 f794c8a0 00000086 00000000 00000002 00000000 f7942764 f794c8a0 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f794c9f4 c3734d00 f7951080 00000246 00000000 00000002 00000000 00000000 f8a45d59 00000246 00000246 f7942754 Call Trace: [] gdlm_thread+0x12d/0x74a [gfs] [] gdlm_thread+0x112/0x74a [gfs] [] autoremove_wake_function+0x0/0x35 [] gdlm_thread2+0x0/0x7 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_recoverd S f8939428 0 3696 2 f7a00aa0 00000086 00000000 f8939428 f7a00aa0 f481ebf4 00002bd4 c045f0c0 c0461d00 c0461d00 c0461d00 f7486000 f7a00bf4 c373ed00 f65d6580 00000002 00000000 00000000 f7a00aa0 00000046 00000000 f7001280 f704b000 f7001280 Call Trace: [] dlm_recoverd+0x163/0x55a [dlm] [] __wake_up_common+0x46/0x68 [] dlm_recoverd+0x0/0x55a [dlm] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_scand R running 0 3701 2 f7a90530 00000086 00000002 f5137f80 f5137f74 00000000 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f5137f84 f7a90684 c373ed00 f0866080 f706e000 ffffe120 00000000 00000286 00000286 ffffffff f706e000 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_scand+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_scand+0x39/0x48 [gfs] [] gfs_scand+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_glockd S f4cd3f94 0 3702 2 f7a57490 00000082 00000002 f4cd3f94 f4cd3f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f4cd3f98 f7a575e4 c3734d00 f7951080 00000286 ffffe11f 00000000 c0319cc1 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] gfs_glockd+0x80/0xb4 [gfs] [] autoremove_wake_function+0x0/0x35 [] gfs_glockd+0x0/0xb4 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_recoverd R running 0 3704 2 f7a9e630 00000086 00000046 00000000 00000002 00000001 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a9e784 c373ed00 f7854080 f706e000 00000286 f706e000 00000286 00000286 f488ffa4 f706e000 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] gfs_recoverd+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_recoverd+0x39/0x48 [gfs] [] gfs_recoverd+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_logd R running 0 3705 2 f788eda0 00000082 00000002 f4891f30 f4891f24 00000000 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f4891f34 f788eef4 c373ed00 f7854300 f706e000 ffffe1d0 00000000 00000286 00000286 ffffffff f706e000 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_logd+0x23/0xb7 [gfs] [] process_timeout+0x0/0x5 [] gfs_logd+0x41/0xb7 [gfs] [] gfs_logd+0x0/0xb7 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_quotad R running 0 3706 2 f788f610 00000086 00000046 00000000 00000002 00000001 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f788f764 c373ed00 f0866080 f706e000 00000282 f706e000 00000282 00000282 f4893f88 f706e000 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] gfs_quotad+0x88/0x16d [gfs] [] process_timeout+0x0/0x5 [] gfs_quotad+0xa6/0x16d [gfs] [] gfs_quotad+0x0/0x16d [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_inoded R running 0 3707 2 f787f510 00000086 00000002 f4895f80 f4895f74 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f4895f84 f787f664 c3734d00 f7951080 c04b9e00 ffffdd30 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_inoded+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_inoded+0x39/0x48 [gfs] [] gfs_inoded+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= lock_dlm1 S 00000002 0 3711 2 f79cc630 00000086 00000400 00000002 00000000 f75f9364 f79cc630 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f79cc784 c3734d00 f7951d00 00000246 00000000 00000002 00000000 00000000 f8a45d59 00000246 00000246 f75f9354 Call Trace: [] gdlm_thread+0x12d/0x74a [gfs] [] gdlm_thread+0x112/0x74a [gfs] [] autoremove_wake_function+0x0/0x35 [] gdlm_thread1+0x0/0xa [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= lock_dlm2 S 00000002 0 3712 2 f71dd390 00000086 c011abae 00000002 00000000 f75f9364 f71dd390 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f71dd4e4 c3734d00 f7951d00 00000246 00000000 00000002 00000000 00000000 f8a45d59 00000246 00000246 f75f9354 Call Trace: [] try_to_wake_up+0xa8/0x137 [] gdlm_thread+0x12d/0x74a [gfs] [] gdlm_thread+0x112/0x74a [gfs] [] autoremove_wake_function+0x0/0x35 [] gdlm_thread2+0x0/0x7 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_recoverd S f8939428 0 3714 2 f786e330 00000086 00000000 f8939428 f786e330 f48a2bf4 00002bd4 c045f0c0 c0461d00 c0461d00 c0461d00 f7486800 f786e484 c3734d00 f65d6580 00000002 00000000 00000000 f786e330 00000046 00000000 f7001280 f700d800 f7001280 Call Trace: [] dlm_recoverd+0x163/0x55a [dlm] [] __wake_up_common+0x46/0x68 [] dlm_recoverd+0x0/0x55a [dlm] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_scand R running 0 3719 2 f70f2330 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f70f2484 c3734d00 f64c8580 c04b9e00 00000286 c04b9e00 00000286 00000286 f489dfa4 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] gfs_scand+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_scand+0x39/0x48 [gfs] [] gfs_scand+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_glockd S f10f8d2c 0 3720 2 f786eba0 00000082 f10f8d58 f10f8d2c 00000246 f8b7ab8c 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f786ecf4 c373ed00 f7854300 00000286 00000001 f48adfbc c0319cc1 00000000 00000046 00000046 f8b83504 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] gfs_glockd+0x80/0xb4 [gfs] [] autoremove_wake_function+0x0/0x35 [] gfs_glockd+0x0/0xb4 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_recoverd R running 0 3722 2 f7b2d510 00000086 00000046 00000000 00000002 00000001 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7b2d664 c373ed00 f7854080 f706e000 00000286 f706e000 00000286 00000286 f3d33fa4 f706e000 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] gfs_recoverd+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_recoverd+0x39/0x48 [gfs] [] gfs_recoverd+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_logd R running 0 3723 2 f7b2c430 00000082 00000002 f3d35f30 f3d35f24 00000000 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f3d35f34 f7b2c584 c373ed00 f65d6580 f706e000 ffffe1b6 00000000 00000286 00000286 ffffffff f706e000 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_logd+0x23/0xb7 [gfs] [] process_timeout+0x0/0x5 [] gfs_logd+0x41/0xb7 [gfs] [] gfs_logd+0x0/0xb7 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_quotad R running 0 3724 2 f7aa7110 00000086 00000046 00000000 00000002 00000001 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7aa7264 c373ed00 f65d6580 f706e000 00000282 f706e000 00000282 00000282 f3d37f88 f706e000 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] gfs_quotad+0x88/0x16d [gfs] [] process_timeout+0x0/0x5 [] gfs_quotad+0xa6/0x16d [gfs] [] gfs_quotad+0x0/0x16d [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_inoded R running 0 3725 2 f79ccea0 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79ccff4 c3734d00 f6bdf580 c04b9e00 00000286 c04b9e00 00000286 00000286 f3d39fa4 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] gfs_inoded+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_inoded+0x39/0x48 [gfs] [] gfs_inoded+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= atd S c373bf78 0 3858 1 f22e4130 00000082 f4cd1efc c373bf78 f65a81e0 f4cd1f60 f65a81dc c045f0c0 c0461d00 c0461d00 c0461d00 c373bf20 f22e4284 c373ed00 f64e4800 f4cd1f60 c0134804 08e28ae0 0000000c c045ef20 399b2ae0 00000352 00000000 c373bf20 Call Trace: [] hrtimer_start+0xd3/0x15e [] do_nanosleep+0x6d/0x96 [] hrtimer_nanosleep+0x50/0xad [] hrtimer_wakeup+0x0/0x18 [] sys_nanosleep+0x58/0x5c [] sysenter_do_call+0x12/0x35 ======================= cron R running 0 3873 1 f22e5210 00000082 00000002 f499bf2c f499bf20 00000000 f7325cdc c045f0c0 c0461d00 c0461d00 c0461d00 f499bf30 f22e5364 c3734d00 f7c37d00 f499bf60 ffffd0bc 00000000 00000029 c045ef20 ffffffff 00000037 00000000 00000000 Call Trace: [] do_nanosleep+0x6d/0x96 [] hrtimer_nanosleep+0x50/0xad [] hrtimer_wakeup+0x0/0x18 [] sys_nanosleep+0x58/0x5c [] sysenter_do_call+0x12/0x35 ======================= lighttpd R running 0 3883 1 f7a9eea0 00200082 00000002 f2121bf4 f2121be8 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f2121bf8 f7a9eff4 c3734d00 f649c580 c04b9e00 ffffe1ba 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] _spin_lock+0x31/0x3c [] rb_insert_color+0xa3/0xc8 [] enqueue_task_fair+0xa2/0xb4 [] _spin_lock_irqsave+0x41/0x49 [] scsi_next_command+0x25/0x2f [] scsi_end_request+0x52/0x75 [] scsi_io_completion+0x87/0x3de [] _spin_lock+0x31/0x3c [] blk_done_softirq+0x5b/0x68 [] __do_softirq+0xa2/0xf9 [] _local_bh_enable+0x44/0xa0 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= S99phplog S f2107f2c 0 3887 1 f7ace920 00000082 00000002 f2107f2c f2107f20 00000000 f65519dc c045f0c0 c0461d00 c0461d00 c0461d00 f2107f30 f7acea74 c373ed00 f6a8c300 f2107f60 ffffdf5e 00000000 00000032 c045ef20 ffffffff 00000039 00000000 00000000 Call Trace: [] do_nanosleep+0x6d/0x96 [] hrtimer_nanosleep+0x50/0xad [] hrtimer_wakeup+0x0/0x18 [] sys_nanosleep+0x58/0x5c [] sysenter_do_call+0x12/0x35 ======================= lighttpd R running 0 3936 1 f7b05310 00200082 00000400 c3731aa0 00000002 f212dbf0 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f212dbf8 f7b05464 c3734d00 f78a9080 c04b9e00 ffffe1fd 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] handle_mm_fault+0x176/0x8a4 [] do_page_fault+0xa5/0x848 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] tcp_sendmsg+0x704/0xa99 [] local_bh_enable_ip+0x79/0xb5 [] tcp_sendmsg+0x704/0xa99 [] _spin_unlock+0x14/0x1c [] rmqueue_bulk+0x67/0x71 [] sock_aio_write+0xcb/0xe8 [] gfs_readpage+0x35/0x14f [gfs] [] do_sync_write+0xd2/0x10e [] __do_page_cache_readahead+0xb8/0x257 [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] vfs_write+0xe5/0x118 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f21b5f24 0 3944 1 f7877490 00000082 00000002 f21b5f24 f21b5f18 00000000 f7877490 c045f0c0 c0461d00 c0461d00 c0461d00 f21b5f28 f78775e4 c3734d00 f78f5580 00000000 ffff9f88 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 00000002 0 3956 1 f22dd190 00000082 f22dd190 00000002 00000000 c040c990 f22dd190 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f22dd2e4 c373ed00 f79c8800 00000000 00000002 00000002 00000000 00000000 c01228e1 00000246 00000246 c040c980 Call Trace: [] do_wait+0x96/0x342 [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f2175f24 0 3968 1 f797e9a0 00000082 00000002 f2175f24 f2175f18 00000000 f797e9a0 c045f0c0 c0461d00 c0461d00 c0461d00 f2175f28 f797eaf4 c373ed00 f7955a80 00000000 ffff9f95 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f216ff24 0 3980 1 f797f210 00000082 00000002 f216ff24 f216ff18 00000000 f797f210 c045f0c0 c0461d00 c0461d00 c0461d00 f216ff28 f797f364 c3734d00 f7955800 00000000 ffff9f88 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f21dff24 0 3992 1 f7977190 00000082 00000002 f21dff24 f21dff18 00000000 f7977190 c045f0c0 c0461d00 c0461d00 c0461d00 f21dff28 f79772e4 c3734d00 f7843d00 00000000 ffff9f88 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c3731aa0 0 4005 1 f7b40d20 00000082 00000400 c3731aa0 00000002 f21cbf20 f7b40d20 c045f0c0 c0461d00 c0461d00 c0461d00 f21cbf28 f7b40e74 c3734d00 f78f5d00 00000000 ffff9f88 00000001 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c373baa0 0 4006 3980 f798f310 00000082 00000400 c373baa0 00000002 f4c1de20 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f4c1de28 f798f464 c373ed00 f753dd00 f44c645c ffff9f95 00000001 00000001 f44c644c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c01512af 0 4007 3980 f2113310 00000082 c3414178 c01512af 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f2113310 f2113464 c373ed00 f78f5300 f44c645c 00000046 f40100ac 00000001 f44c644c 00000046 00000000 00000002 00000001 Call Trace: [] rmqueue_bulk+0x67/0x71 [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c01512af 0 4008 3980 f20628a0 00000082 c34147c4 c01512af 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f20628a0 f20629f4 c3734d00 f7955580 f44c645c 00000046 f40100ac 00000001 f44c644c 00000046 00000000 00000002 00000001 Call Trace: [] rmqueue_bulk+0x67/0x71 [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4009 3980 f22dc920 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f22dc920 f22dca74 c3734d00 f78f5a80 f44c645c 00000046 f40100ac 00000001 f44c644c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 0000001f 0 4010 3956 f7a60d20 00000082 c0405c80 0000001f 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f7a60d20 f7a60e74 c373ed00 f78f5080 f44c675c 00000046 f557482c 00000001 f44c674c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c01e3113 0 4011 4005 f7a2a330 00000082 0000000a c01e3113 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f7a2a330 f7a2a484 c3734d00 f78a9300 f758715c 00000046 f2a0c5ac 00000001 f758714c 00000046 00000000 00000002 00000001 Call Trace: [] blk_done_softirq+0x5b/0x68 [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4013 4005 f2062030 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f2062030 f2062184 c3734d00 f7969300 f758715c 00000046 f2a0c5ac 00000001 f758714c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4014 4005 f794c030 00000082 f794c030 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f794c030 f794c184 c373ed00 f7969080 f758715c 00000046 f2a0c5ac 00000001 f758714c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4018 4005 f2112230 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f2112230 f2112384 c3734d00 f78f5800 f758715c 00000046 f2a0c5ac 00000001 f758714c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4019 3968 f2112aa0 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f2112aa0 f2112bf4 c373ed00 f753d300 f44c6a5c 00000046 f40105ac 00000001 f44c6a4c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4020 3956 f7a90da0 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f7a90da0 f7a90ef4 c373ed00 f753d800 f44c675c 00000046 f557482c 00000001 f44c674c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c01512af 0 4021 3968 f7ad69a0 00000082 c3419354 c01512af 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f7ad69a0 f7ad6af4 c373ed00 f753d580 f44c6a5c 00000046 f40105ac 00000001 f44c6a4c 00000046 00000000 00000002 00000001 Call Trace: [] rmqueue_bulk+0x67/0x71 [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4022 3956 f21eb390 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f21eb390 f21eb4e4 c373ed00 f6affa80 f44c675c 00000046 f557482c 00000001 f44c674c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4023 3968 f21eab20 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f21eab20 f21eac74 c373ed00 f6aff080 f44c6a5c 00000046 f40105ac 00000001 f44c6a4c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 00000001 0 4024 3956 f21ea2b0 00000082 f701bf34 00000001 f7045400 00000046 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f21ea2b0 f21ea404 c373ed00 f6aff580 f44c675c 00000046 f557482c 00000001 f44c674c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f7054bb4 0 4025 3968 f2313410 00000082 00000000 f7054bb4 f2313410 f2313410 000800d0 c045f0c0 c0461d00 c0461d00 c0461d00 f2313410 f2313564 c373ed00 f6affd00 f44c6a5c 00000046 f40105ac 00000001 f44c6a4c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 00000286 0 4026 3944 f2317490 00000082 f6a40364 00000286 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f2317490 f23175e4 c3734d00 f649c800 f513cd5c 00000046 f29815ac 00000001 f513cd4c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4029 3944 f231f510 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f231f510 f231f664 c3734d00 f7843300 f513cd5c 00000046 f29815ac 00000001 f513cd4c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4030 3944 f231eca0 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f231eca0 f231edf4 c3734d00 f649cd00 f513cd5c 00000046 f29815ac 00000001 f513cd4c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f7054bb4 0 4032 3944 f231e430 00000082 00000000 f7054bb4 f231e430 f231e430 000800d0 c045f0c0 c0461d00 c0461d00 c0461d00 f231e430 f231e584 c3734d00 f79c8300 f513cd5c 00000046 f29815ac 00000001 f513cd4c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f2355f24 0 4037 1 f2353610 00000082 00000002 f2355f24 f2355f18 00000000 f2353610 c045f0c0 c0461d00 c0461d00 c0461d00 f2355f28 f2353764 c3734d00 f79c8080 00000000 ffff9f88 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f21a7e24 0 4038 4037 f7b404b0 00000082 00000002 f21a7e24 f21a7e18 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f21a7e28 f7b40604 c3734d00 f79c8d00 f643545c ffffd8d9 00000000 00000001 f643544c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] skb_dequeue+0x50/0x56 [] net_rx_action+0x85/0x1db [] net_rx_action+0x9d/0x1db [] sys_socketcall+0x23e/0x2a9 [] _local_bh_enable+0x44/0xa0 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f48abe24 0 4039 4037 f2063110 00000082 00000002 f48abe24 f48abe18 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f48abe28 f2063264 c3734d00 f79c8a80 f643545c ffffd55b 00000000 00000001 f643544c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] skb_dequeue+0x50/0x56 [] dput+0x97/0x145 [] sys_socketcall+0x23e/0x2a9 [] filp_close+0x3e/0x62 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f2377e24 0 4040 4037 f20dea20 00000082 00000002 f2377e24 f2377e18 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f2377e28 f20deb74 c3734d00 f7951a80 f643545c ffffde30 00000000 00000001 f643544c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] skb_dequeue+0x50/0x56 [] dput+0x97/0x145 [] sys_socketcall+0x23e/0x2a9 [] filp_close+0x3e/0x62 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f2395e24 0 4041 4037 f22dc0b0 00000082 00000002 f2395e24 f2395e18 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f2395e28 f22dc204 c3734d00 f6bdf080 f643545c ffffd1a7 00000000 00000001 f643544c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] skb_dequeue+0x50/0x56 [] dput+0x97/0x145 [] sys_socketcall+0x23e/0x2a9 [] filp_close+0x3e/0x62 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f2397e24 0 4042 3992 f2316c20 00000082 00000002 f2397e24 f2397e18 00000000 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f2397e28 f2316d74 c373ed00 f6bdf800 f44c615c ffff9f95 00000000 00000001 f44c614c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4043 3992 f20de1b0 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f20de1b0 f20de304 c3734d00 f64e4300 f44c615c 00000046 f2a96d2c 00000001 f44c614c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f239de24 0 4044 3992 f2352da0 00000082 00000002 f239de24 f239de18 00000000 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f239de28 f2352ef4 c373ed00 f64e4580 f44c615c ffff9f95 00000000 00000001 f44c614c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S c01512af 0 4045 3992 f2352530 00000082 c340b03c c01512af 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f2352530 f2352684 c3734d00 f64e4a80 f44c615c 00000046 f2a96d2c 00000001 f44c614c 00000046 00000000 00000002 00000001 Call Trace: [] rmqueue_bulk+0x67/0x71 [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f23c1e24 0 4046 3992 f23bf690 00000082 00000002 f23c1e24 f23c1e18 00000000 000800d0 c045f0c0 c0461d00 c0461d00 c0461d00 f23c1e28 f23bf7e4 c373ed00 f64e4080 f44c615c ffff9f95 00000000 00000001 f44c614c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f7054bb4 0 4047 3992 f23bee20 00000082 00000000 f7054bb4 f23bee20 f23bee20 000800d0 c045f0c0 c0461d00 c0461d00 c0461d00 f23bee20 f23bef74 c3734d00 f64c8d00 f44c615c 00000046 f2a96d2c 00000001 f44c614c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S f23c7e24 0 4048 3992 f23be5b0 00000082 00000002 f23c7e24 f23c7e18 00000000 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f23c7e28 f23be704 c373ed00 f64c8300 f44c615c ffff9f95 00000000 00000001 f44c614c ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= php5-cgi S 001200d2 0 4049 3992 f23c9710 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 c0461d00 c0461d00 c0461d00 f23c9710 f23c9864 c3734d00 f64c8080 f44c615c 00000046 f2a96d2c 00000001 f44c614c 00000046 00000000 00000002 00000001 Call Trace: [] schedule_timeout+0x69/0xa1 [] __skb_recv_datagram+0x65/0x22d [] autoremove_wake_function+0x0/0x35 [] skb_recv_datagram+0x22/0x27 [] unix_accept+0x56/0xe3 [] do_accept+0xf0/0x1d4 [] handle_mm_fault+0x176/0x8a4 [] up_read+0x14/0x2a [] do_page_fault+0xa5/0x848 [] do_page_fault+0x156/0x848 [] sys_socketcall+0x23e/0x2a9 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= bash S 00000000 0 4072 3067 f2312330 00000082 00000002 00000000 f7191c3c f2312330 c03eb0b0 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f2312484 c3734d00 f7843580 00000000 00000002 00000001 00000046 f719197c f7191800 f7192000 00000046 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] _spin_lock_irqsave+0x41/0x49 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= bash S f1dcdeb4 0 4095 3063 f7a2aba0 00000082 00000002 f1dcdeb4 f1dcdea8 00000000 c03eb0b0 c045f0c0 c0461d00 c0461d00 c0461d00 f1dcdeb8 f7a2acf4 c3734d00 f65d6d00 00000000 ffffce32 00000000 00000046 f644697c ffffffff f45eb000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] _spin_lock_irqsave+0x41/0x49 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= mailmanctl S f147bf24 0 4115 1 f1dd1410 00200082 00000002 f147bf24 f147bf18 00000000 f1dd1410 c045f0c0 c0461d00 c0461d00 c0461d00 f147bf28 f1dd1564 c373ed00 f65d6800 00000000 ffffd43c 00000000 00000000 00000000 ffffffff 00200246 00000000 00000000 Call Trace: [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4116 4115 f7b2cca0 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7b2cdf4 c3734d00 f7843a80 c04b9e00 00000286 c04b9e00 00000286 00000286 f1467b38 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] wait_for_common+0x26/0x11b [] wait_for_common+0x26/0x11b [] _spin_unlock_irq+0x20/0x23 [] wait_for_common+0xcd/0x11b [] find_get_page+0xa2/0xbf [] gfs_sort+0x80/0x103 [gfs] [] do_filldir_main+0x28/0x1d1 [gfs] [] compare_dents+0x0/0x65 [gfs] [] getbuf+0xfe/0x196 [gfs] [] wait_for_common+0x26/0x11b [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] filldir64+0x0/0xc5 [] _spin_unlock+0x14/0x1c [] gfs_readdir+0x329/0x3c2 [gfs] [] filldir_reg_func+0x0/0x1c9 [gfs] [] copy_to_user+0x2f/0x4d [] filldir64+0x0/0xc5 [] file_kill+0x13/0x2d [] sys_select+0xd6/0x19e [] filp_close+0x3e/0x62 [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4117 4115 f20df290 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f20df3e4 c3734d00 f65d6a80 c04b9e00 00000286 c04b9e00 00000286 00000286 f1d53b38 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] task_rq_lock+0x2e/0x55 [] _spin_lock_irqsave+0x41/0x49 [] __pollwait+0x0/0xba [] wait_for_common+0x26/0x11b [] find_get_page+0xa2/0xbf [] find_get_page+0xa2/0xbf [] find_get_page+0xd/0xbf [] do_filldir_main+0x28/0x1d1 [gfs] [] compare_dents+0x0/0x65 [gfs] [] wait_for_common+0x26/0x11b [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] filldir64+0x0/0xc5 [] _spin_unlock+0x14/0x1c [] gfs_readdir+0x329/0x3c2 [gfs] [] filldir_reg_func+0x0/0x1c9 [gfs] [] copy_to_user+0x2f/0x4d [] filldir64+0x0/0xc5 [] file_kill+0x13/0x2d [] sys_select+0xd6/0x19e [] copy_to_user+0x2f/0x4d [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4118 4115 f7876c20 00000086 00000818 c3731aa0 00000002 f147db10 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f147db18 f7876d74 c3734d00 f6bdf580 c04b9e00 ffffe1c1 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] wait_for_common+0x26/0x11b [] find_get_page+0xa2/0xbf [] find_get_page+0xa2/0xbf [] gfs_sort+0x80/0x103 [gfs] [] do_filldir_main+0x28/0x1d1 [gfs] [] compare_dents+0x0/0x65 [gfs] [] wait_for_common+0x26/0x11b [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] filldir64+0x0/0xc5 [] _spin_unlock+0x14/0x1c [] gfs_readdir+0x329/0x3c2 [gfs] [] filldir_reg_func+0x0/0x1c9 [gfs] [] copy_to_user+0x2f/0x4d [] filldir64+0x0/0xc5 [] file_kill+0x13/0x2d [] sys_select+0xd6/0x19e [] filp_close+0x3e/0x62 [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4119 4115 f22e49a0 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f22e4af4 c3734d00 f7843080 c04b9e00 00000286 c04b9e00 00000286 00000286 f147fb38 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] scsi_next_command+0x25/0x2f [] scsi_io_completion+0x87/0x3de [] _spin_lock+0x31/0x3c [] __pollwait+0x0/0xba [] blk_done_softirq+0x5b/0x68 [] wait_for_common+0x26/0x11b [] wait_for_common+0x26/0x11b [] _spin_unlock_irq+0x20/0x23 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] dput+0x97/0x145 [] gfs_drevalidate+0x1a6/0x1f0 [gfs] [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] copy_to_user+0x2f/0x4d [] cp_new_stat64+0xf9/0x10b [] sys_select+0xd6/0x19e [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4120 4115 f2312ba0 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f2312cf4 c3734d00 f6bdf300 c04b9e00 00000286 c04b9e00 00000286 00000286 f1c59b38 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] _spin_lock+0x31/0x3c [] __pollwait+0x0/0xba [] wait_for_common+0x26/0x11b [] wait_for_common+0x26/0x11b [] _spin_unlock_irq+0x20/0x23 [] wait_for_common+0xcd/0x11b [] find_get_page+0xa2/0xbf [] gfs_sort+0x80/0x103 [gfs] [] do_filldir_main+0x28/0x1d1 [gfs] [] compare_dents+0x0/0x65 [gfs] [] getbuf+0xfe/0x196 [gfs] [] wait_for_common+0x26/0x11b [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] filldir64+0x0/0xc5 [] _spin_unlock+0x14/0x1c [] gfs_readdir+0x329/0x3c2 [gfs] [] filldir_reg_func+0x0/0x1c9 [gfs] [] copy_to_user+0x2f/0x4d [] filldir64+0x0/0xc5 [] dput+0xa9/0x145 [] sys_select+0xd6/0x19e [] filp_close+0x3e/0x62 [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4121 4115 f23c8ea0 00000082 00000e03 00000009 00000009 098d81b8 f1c5a000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f23c8ff8 c373ed00 f7854300 00000009 09b7c8ec 00000e03 00000000 00000246 f7854334 00000009 09b7c8ec 00000e03 Call Trace: [] work_resched+0x5/0x2a ======================= python R running 0 4122 4115 f23c8630 00000086 00000002 f1c15e50 f1c15e44 00000000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 f1c15e54 f23c8784 c3734d00 f7969a80 c02a6018 ffffe1fd 00000000 f754fa90 f754fa90 ffffffff c370ed9c 00000000 00000000 Call Trace: [] dm_get_table+0x11/0x2e [] io_schedule+0x1b/0x24 [] sync_page+0x2c/0x39 [] __wait_on_bit+0x42/0x5e [] sync_page+0x0/0x39 [] wait_on_page_bit+0x64/0x6b [] wake_bit_function+0x0/0x3c [] wait_on_page_writeback_range+0x9c/0x104 [] gfs_sync_page_i+0x28/0x5c [gfs] [] gfs_drop_inode+0x55/0x5d [gfs] [] iput+0x44/0x4a [] do_unlinkat+0xc6/0x140 [] do_munmap+0x183/0x1e0 [] sysenter_do_call+0x12/0x35 ======================= python S 00000000 0 4123 4115 f7a2b410 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a2b564 c3734d00 f79c8580 c04b9e00 00000286 c04b9e00 00000286 00000286 f1dcfb38 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] _spin_lock_irqsave+0x41/0x49 [] __pollwait+0x0/0xba [] wait_for_common+0x26/0x11b [] wait_for_common+0x26/0x11b [] _spin_unlock_irq+0x20/0x23 [] wait_for_common+0xcd/0x11b [] find_get_page+0xa2/0xbf [] gfs_sort+0x80/0x103 [gfs] [] do_filldir_main+0x28/0x1d1 [gfs] [] compare_dents+0x0/0x65 [gfs] [] getbuf+0xfe/0x196 [gfs] [] wait_for_common+0x26/0x11b [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] filldir64+0x0/0xc5 [] _spin_unlock+0x14/0x1c [] gfs_readdir+0x329/0x3c2 [gfs] [] filldir_reg_func+0x0/0x1c9 [gfs] [] filldir64+0x0/0xc5 [] dput+0xa9/0x145 [] sys_select+0xd6/0x19e [] filp_close+0x3e/0x62 [] sysenter_do_call+0x12/0x35 ======================= cleanup R running 0 4127 3375 f1021c0c 00000001 00000000 c01b671d f1021c5c f1021c60 f1021c0c f1dd0ba0 00000001 f19d711c f197c3c0 00000000 008ef800 00000001 f19d7030 00000001 f1021c4c 00000000 00001000 00000001 f2bc231c 00200286 00000000 f1021c18 Call Trace: [] ext3_get_blocks_handle+0x32c/0x9fe [] _spin_lock_irqsave+0x41/0x49 [] free_poll_entry+0xe/0x16 [] poll_freewait+0x2a/0x61 [] do_sys_poll+0x2b7/0x311 [] __pollwait+0x0/0xba [] ext3_get_block+0x6d/0xee [] __block_prepare_write+0x1ef/0x3b8 [] block_write_begin+0x4c/0xe0 [] ext3_get_block+0x0/0xee [] ext3_write_begin+0xcc/0x19e [] ext3_get_block+0x0/0xee [] generic_file_buffered_write+0x113/0x677 [] mnt_drop_write+0x1d/0xd8 [] _spin_lock+0x31/0x3c [] __generic_file_aio_write_nolock+0x285/0x4df [] generic_file_aio_write+0x55/0xd8 [] generic_file_aio_write+0x55/0xd8 [] generic_file_aio_write+0x6a/0xd8 [] sockfd_lookup_light+0x24/0x41 [] ext3_file_write+0x24/0x99 [] do_sync_write+0xd2/0x10e [] cp_new_stat64+0xf9/0x10b [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] do_sync_write+0x0/0x10e [] vfs_write+0x87/0x118 [] sys_write+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= smtp R running 0 4128 3375 f79f9290 00200082 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79f93e4 c3734d00 f64c8a80 c04b9e00 00200282 c04b9e00 00200282 00200282 f1105c18 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] ip_finish_output+0x112/0x26b [] ip_local_out+0x15/0x17 [] ip_queue_xmit+0x195/0x327 [] _spin_lock_irqsave+0x41/0x49 [] tcp_transmit_skb+0x363/0x659 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] tcp_sendmsg+0x704/0xa99 [] local_bh_enable_ip+0x79/0xb5 [] tcp_sendmsg+0x704/0xa99 [] copy_to_user+0x2f/0x4d [] sock_aio_write+0xcb/0xe8 [] do_sync_write+0xd2/0x10e [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] read_tsc+0x6/0x28 [] getnstimeofday+0x44/0x100 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= smtp R running 0 4129 3375 f7a604b0 00200082 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a60604 c3734d00 f7955300 c04b9e00 00200282 c04b9e00 00200282 00200282 f105bc18 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] ip_finish_output+0x112/0x26b [] ip_local_out+0x15/0x17 [] ip_queue_xmit+0x195/0x327 [] tcp_current_mss+0xa2/0xd4 [] tcp_transmit_skb+0x363/0x659 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] tcp_sendmsg+0x704/0xa99 [] local_bh_enable_ip+0x79/0xb5 [] tcp_sendmsg+0x704/0xa99 [] nommu_map_single+0x2c/0x63 [] nommu_map_single+0x0/0x63 [] e1000_clean_rx_irq+0x29b/0x32f [e1000e] [] sock_aio_write+0xcb/0xe8 [] do_sync_write+0xd2/0x10e [] net_rx_action+0xe0/0x1db [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] read_tsc+0x6/0x28 [] getnstimeofday+0x44/0x100 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= smtp R running 0 4130 3375 f23163b0 00200082 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f2316504 c3734d00 f65d6300 c04b9e00 00200282 c04b9e00 00200282 00200282 f1027c18 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] ip_finish_output+0x112/0x26b [] ip_local_out+0x15/0x17 [] ip_queue_xmit+0x195/0x327 [] _spin_lock_irqsave+0x41/0x49 [] tcp_transmit_skb+0x363/0x659 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] tcp_sendmsg+0x704/0xa99 [] local_bh_enable_ip+0x79/0xb5 [] tcp_sendmsg+0x704/0xa99 [] copy_to_user+0x2f/0x4d [] sock_aio_write+0xcb/0xe8 [] do_sync_write+0xd2/0x10e [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] read_tsc+0x6/0x28 [] getnstimeofday+0x44/0x100 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= smtp R running 0 4131 3375 f1ce1490 00200082 0000055d c3731aa0 00000002 f1ddfbf0 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f1ddfbf8 f1ce15e4 c3734d00 f65d6080 c04b9e00 ffffe1e3 00000001 00200282 00200282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] ip_finish_output+0x112/0x26b [] ip_local_out+0x15/0x17 [] ip_queue_xmit+0x195/0x327 [] dev_queue_xmit+0x36/0x509 [] tcp_transmit_skb+0x363/0x659 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] tcp_sendmsg+0x704/0xa99 [] local_bh_enable_ip+0x79/0xb5 [] tcp_sendmsg+0x704/0xa99 [] tcp_recvmsg+0x2cf/0x7ff [] local_bh_enable_ip+0x79/0xb5 [] tcp_recvmsg+0x2cf/0x7ff [] sock_common_recvmsg+0x3e/0x54 [] e1000_clean_rx_irq+0x28b/0x32f [e1000e] [] sock_aio_write+0xcb/0xe8 [] do_sync_write+0xd2/0x10e [] net_rx_action+0x85/0x1db [] net_rx_action+0xe0/0x1db [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] read_tsc+0x6/0x28 [] getnstimeofday+0x44/0x100 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= smtp R running 0 4132 3375 f1ce0c20 00200082 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f1ce0d74 c3734d00 f7c37080 c04b9e00 00200282 c04b9e00 00200282 00200282 f1481c18 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] ip_finish_output+0x112/0x26b [] ip_local_out+0x15/0x17 [] ip_queue_xmit+0x195/0x327 [] dev_queue_xmit+0x36/0x509 [] ip_finish_output+0x112/0x26b [] ip_local_out+0x15/0x17 [] tcp_transmit_skb+0x363/0x659 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] tcp_sendmsg+0x704/0xa99 [] local_bh_enable_ip+0x79/0xb5 [] tcp_sendmsg+0x704/0xa99 [] nommu_map_single+0x2c/0x63 [] nommu_map_single+0x0/0x63 [] e1000_clean_rx_irq+0x29b/0x32f [e1000e] [] sock_aio_write+0xcb/0xe8 [] do_sync_write+0xd2/0x10e [] net_rx_action+0xe0/0x1db [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] read_tsc+0x6/0x28 [] getnstimeofday+0x44/0x100 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= scache R running 0 4133 3375 f1ce03b0 00200086 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f1ce0504 c3734d00 f64c8800 c04b9e00 00200282 c04b9e00 00200282 00200282 f1537f40 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= smtp R running 0 4134 3375 f1571510 00200082 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f1571664 c3734d00 f6a8ca80 c04b9e00 00200282 c04b9e00 00200282 00200282 f1573c18 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] ip_finish_output+0x112/0x26b [] ip_local_out+0x15/0x17 [] ip_queue_xmit+0x195/0x327 [] _read_unlock+0x14/0x1c [] sock_queue_rcv_skb+0x8d/0xc6 [] _spin_unlock+0x14/0x1c [] udp_queue_rcv_skb+0x11b/0x223 [] __udp4_lib_lookup+0xed/0x110 [] tcp_transmit_skb+0x363/0x659 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] tcp_sendmsg+0x704/0xa99 [] local_bh_enable_ip+0x79/0xb5 [] tcp_sendmsg+0x704/0xa99 [] nommu_map_single+0x0/0x63 [] e1000_clean_rx_irq+0x29b/0x32f [e1000e] [] sock_aio_write+0xcb/0xe8 [] do_sync_write+0xd2/0x10e [] net_rx_action+0xe0/0x1db [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] read_tsc+0x6/0x28 [] getnstimeofday+0x44/0x100 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= smtp R running 0 4135 3375 f1570ca0 00200082 00200046 00000000 00000002 00000001 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f1570df4 c373ed00 f6a8c080 f706e000 00200282 f706e000 00200282 00200282 f1511c18 f706e000 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] ip_finish_output+0x112/0x26b [] ip_local_out+0x15/0x17 [] ip_queue_xmit+0x195/0x327 [] _spin_lock_irqsave+0x41/0x49 [] tcp_transmit_skb+0x363/0x659 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] tcp_sendmsg+0x704/0xa99 [] local_bh_enable_ip+0x79/0xb5 [] tcp_sendmsg+0x704/0xa99 [] copy_to_user+0x2f/0x4d [] sock_aio_write+0xcb/0xe8 [] do_sync_write+0xd2/0x10e [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] read_tsc+0x6/0x28 [] getnstimeofday+0x44/0x100 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= smtp R running 0 4136 3375 f1570430 00200082 0000060c c3731aa0 00000002 f1513bf0 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f1513bf8 f1570584 c3734d00 f6417a80 c04b9e00 ffffe1e3 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] ip_finish_output+0x112/0x26b [] ip_local_out+0x15/0x17 [] ip_queue_xmit+0x195/0x327 [] dev_queue_xmit+0x36/0x509 [] ip_finish_output+0x112/0x26b [] ip_local_out+0x15/0x17 [] tcp_transmit_skb+0x363/0x659 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] tcp_sendmsg+0x704/0xa99 [] local_bh_enable_ip+0x79/0xb5 [] tcp_sendmsg+0x704/0xa99 [] nommu_map_single+0x2c/0x63 [] nommu_map_single+0x0/0x63 [] e1000_clean_rx_irq+0x29b/0x32f [e1000e] [] sock_aio_write+0xcb/0xe8 [] do_sync_write+0xd2/0x10e [] net_rx_action+0xe0/0x1db [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] read_tsc+0x6/0x28 [] getnstimeofday+0x44/0x100 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= smtp R running 0 4137 3375 f11d1590 00200082 00000002 f11d3bf4 f11d3be8 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f11d3bf8 f11d16e4 c3734d00 f7c37800 c04b9e00 ffffe1fd 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] dev_queue_xmit+0xa2/0x509 [] dev_queue_xmit+0xee/0x509 [] local_bh_enable+0x7f/0xc8 [] dev_queue_xmit+0xee/0x509 [] dev_queue_xmit+0x36/0x509 [] ip_finish_output+0x112/0x26b [] ip_local_out+0x15/0x17 [] ip_queue_xmit+0x195/0x327 [] __ip_route_output_key+0x195/0xa6e [] _spin_lock_irqsave+0x41/0x49 [] __mod_timer+0x94/0xa3 [] sk_reset_timer+0xc/0x16 [] tcp_connect+0x2fc/0x31e [] tcp_v4_connect+0x330/0x45c [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] sock_getsockopt+0xcc/0x434 [] copy_to_user+0x2f/0x4d [] tcp_getsockopt+0xa8/0x295 [] copy_from_user+0x2d/0x59 [] sys_connect+0x72/0x9f [] _spin_lock_bh+0x36/0x41 [] sock_setsockopt+0x126/0x546 [] local_bh_enable_ip+0x79/0xb5 [] sock_setsockopt+0x126/0x546 [] sock_common_getsockopt+0x1d/0x23 [] sys_getsockopt+0x4b/0x91 [] read_tsc+0x6/0x28 [] getnstimeofday+0x44/0x100 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= bounce S 00000000 0 4138 3375 f15e5610 00200086 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f15e5764 c3734d00 f6417580 c04b9e00 00200282 c04b9e00 00200282 00200282 f15e7f40 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= trivial-rewri R running 0 4146 3375 f1dd0330 00200082 c01a4b6f f19a7974 00000000 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f19a796c f1dd0488 c373ed00 f0866d00 c0171540 00000000 00000000 f1475480 fffffff7 00000e10 f04c2000 c0171995 f04c3fa0 Call Trace: [] dnotify_parent+0x1f/0x5f [] vfs_write+0xe5/0x118 [] sys_write+0x41/0x6a [] work_resched+0x5/0x2a ======================= smtp R running 0 4148 3375 f0901690 00200082 00000400 c3731aa0 00000002 f0455bf0 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f0455bf8 f09017e4 c3734d00 f7c37a80 c04b9e00 ffffe1e3 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] ip_finish_output+0x112/0x26b [] ip_local_out+0x15/0x17 [] ip_queue_xmit+0x195/0x327 [] _spin_lock_irqsave+0x41/0x49 [] __mod_timer+0x94/0xa3 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] tcp_sendmsg+0x704/0xa99 [] local_bh_enable_ip+0x79/0xb5 [] tcp_sendmsg+0x704/0xa99 [] find_get_page+0xa2/0xbf [] sock_aio_write+0xcb/0xe8 [] do_sync_write+0xd2/0x10e [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] read_tsc+0x6/0x28 [] getnstimeofday+0x44/0x100 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= smtp R running 0 4149 3375 c0319867 c04adab8 f64d0cc0 c03eb0a0 00200086 0003ffff 0003430e c0319cc1 00000000 00000002 c03eb070 00200082 0003ffff c03eb070 00200082 0003436a 0003437e c0120b4e 00034391 00200082 fffd3b1d 000343a2 c04b9800 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_lock_irqsave+0x41/0x49 [] printk+0x1b/0x1f [] printk+0x1b/0x1f [] print_trace_address+0x39/0x43 [] __kernel_text_address+0x18/0x23 [] dump_trace+0x5e/0xe9 [] show_trace_log_lvl+0x21/0x37 [] show_stack_log_lvl+0xaf/0xd7 [] show_stack+0x2f/0x35 [] show_state_filter+0x4f/0x84 [] __handle_sysrq+0x7a/0x115 [] kbd_event+0x420/0x6db [] kbd_event+0x0/0x6db [] input_pass_event+0xb7/0xb9 [] input_pass_event+0x15/0xb9 [] input_event+0x54/0x68 [] atkbd_interrupt+0x469/0x5b3 [] _spin_lock_irqsave+0x41/0x49 [] serio_interrupt+0x32/0x6b [] i8042_interrupt+0x108/0x23a [] handle_IRQ_event+0x1e/0x48 [] handle_edge_irq+0xb4/0x10a [] do_IRQ+0x3b/0x72 [] _local_bh_enable+0x44/0xa0 [] common_interrupt+0x28/0x30 [] native_read_tsc+0x6/0xf [] delay_tsc+0x12/0x47 [] __delay+0x6/0x7 [] _raw_spin_lock+0xe6/0x170 [] new_inode+0x24/0x8d [] new_inode+0x24/0x8d [] sock_alloc+0xd/0x4d [] __sock_create+0x3a/0x1e0 [] net_rx_action+0x85/0x1db [] sock_create+0x34/0x3a [] sys_socket+0x35/0x67 [] sys_socketcall+0x2a2/0x2a9 [] _local_bh_enable+0x44/0xa0 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= SysRq : Show Blocked State task PC stack pid father SysRq : Show Regs Pid: 4149, comm: smtp Not tainted (2.6.27.21 #6) EIP: 0060:[] EFLAGS: 00200246 CPU: 0 EIP is at native_read_tsc+0x8/0xf EAX: ceb5f747 EBX: 00000000 ECX: 00000000 EDX: 000000e6 ESI: 0df07e08 EDI: 00000001 EBP: ca27a004 ESP: f08ddec8 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 CR0: 8005003b CR2: b7c6f000 CR3: 3152c000 CR4: 000006f0 DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 DR6: ffff0ff0 DR7: 00000400 [] ? delay_tsc+0x12/0x47 [] ? __delay+0x6/0x7 [] ? _raw_spin_lock+0xe6/0x170 [] ? new_inode+0x24/0x8d [] ? new_inode+0x24/0x8d [] ? sock_alloc+0xd/0x4d [] ? __sock_create+0x3a/0x1e0 [] ? net_rx_action+0x85/0x1db [] ? sock_create+0x34/0x3a [] ? sys_socket+0x35/0x67 [] ? sys_socketcall+0x2a2/0x2a9 [] ? _local_bh_enable+0x44/0xa0 [] ? trace_hardirqs_on_thunk+0xc/0x10 [] ? sysenter_do_call+0x12/0x35 ======================= SysRq : Keyboard mode set to system default SysRq : Show Memory Mem-Info: DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 Normal per-cpu: CPU 0: hi: 186, btch: 31 usd: 120 CPU 1: hi: 186, btch: 31 usd: 125 HighMem per-cpu: CPU 0: hi: 186, btch: 31 usd: 62 CPU 1: hi: 186, btch: 31 usd: 44 Active:55728 inactive:25975 dirty:1120 writeback:0 unstable:0 free:674297 slab:13284 mapped:11185 pagetables:1027 bounce:0 DMA free:8804kB min:1140kB low:1424kB high:1708kB active:0kB inactive:0kB present:15788kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 868 3015 3015 Normal free:773292kB min:64392kB low:80488kB high:96588kB active:6924kB inactive:16168kB present:889680kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 17174 17174 HighMem free:1915092kB min:512kB low:40288kB high:80064kB active:215988kB inactive:87732kB present:2198284kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 DMA: 3*4kB 3*8kB 2*16kB 3*32kB 3*64kB 4*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8804kB Normal: 689*4kB 443*8kB 173*16kB 68*32kB 42*64kB 20*128kB 20*256kB 12*512kB 4*1024kB 6*2048kB 178*4096kB = 773228kB HighMem: 2*4kB 1*8kB 2*16kB 2*32kB 1*64kB 2*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 467*4096kB = 1915056kB 35431 total pagecache pages 0 pages in swap cache Swap cache stats: add 0, delete 0, find 0/0 Free swap = 979832kB Total swap = 979832kB 786176 pages RAM 556800 pages HighMem 12334 pages reserved 63865 pages shared 73801 pages non-shared SysRq : Emergency Sync SysRq : Emergency Sync SysRq : Emergency Sync SysRq : Resetting ============================================= [ INFO: possible recursive locking detected ] 2.6.27.21 #6 --------------------------------------------- dlm_controld/3513 is trying to acquire lock: (&sb->s_type->i_mutex_key#8/2){--..}, at: [] configfs_attach_group+0x43/0x18f [configfs] but task is already holding lock: (&sb->s_type->i_mutex_key#8/2){--..}, at: [] configfs_attach_group+0x43/0x18f [configfs] other info that might help us debug this: 2 locks held by dlm_controld/3513: #0: (&sb->s_type->i_mutex_key#7/1){--..}, at: [] lookup_create+0x1b/0x85 #1: (&sb->s_type->i_mutex_key#8/2){--..}, at: [] configfs_attach_group+0x43/0x18f [configfs] stack backtrace: Pid: 3513, comm: dlm_controld Not tainted 2.6.27.21 #6 [] validate_chain+0x9bd/0xf35 [] __lock_acquire+0x27b/0x932 [] lock_acquire+0x5f/0x77 [] configfs_attach_group+0x43/0x18f [configfs] [] mutex_lock_nested+0xa1/0x25c [] configfs_attach_group+0x43/0x18f [configfs] [] configfs_attach_group+0x43/0x18f [configfs] [] configfs_attach_group+0x43/0x18f [configfs] [] configfs_attach_group+0xe5/0x18f [configfs] [] configfs_mkdir+0x1fb/0x36f [configfs] [] vfs_mkdir+0x9c/0xdf [] _spin_unlock+0x14/0x1c [] sys_mkdirat+0xc5/0xd2 [] trace_hardirqs_on_thunk+0xc/0x10 [] sys_mkdir+0x1f/0x23 [] sysenter_do_call+0x12/0x35 ======================= aoe: e1.0: setting 8704 byte data frames on eth1:0030486543a9 aoe: e2.0: setting 8704 byte data frames on eth1:003048654382 aoe: e2.0: setting 8704 byte data frames on eth1:003048654383 aoe: 0030486543a9 e1.0 vace0 has 2930382333 sectors aoe: 003048654382 e2.0 vace0 has 2930382333 sectors etherd/e1.0: unknown partition table etherd/e2.0: unknown partition table dlm: Using TCP for communications dlm: got connection from 1 dlm: got connection from 5 dlm: connecting to 2 dlm: got connection from 2 dlm: connecting to 4 dlm: got connection from 4 Trying to join cluster "lock_dlm", "kfki:home" Joined cluster. Now mounting FS... GFS: fsid=kfki:home.3: jid=3: Trying to acquire journal lock... GFS: fsid=kfki:home.3: jid=3: Looking at journal... GFS: fsid=kfki:home.3: jid=3: Done GFS: fsid=kfki:home.3: Scanning for log elements... GFS: fsid=kfki:home.3: Found 0 unlinked inodes GFS: fsid=kfki:home.3: Found quota changes for 0 IDs GFS: fsid=kfki:home.3: Done Trying to join cluster "lock_dlm", "kfki:services" Joined cluster. Now mounting FS... GFS: fsid=kfki:services.3: jid=3: Trying to acquire journal lock... GFS: fsid=kfki:services.3: jid=3: Looking at journal... GFS: fsid=kfki:services.3: jid=3: Done GFS: fsid=kfki:services.3: Scanning for log elements... GFS: fsid=kfki:services.3: Found 83 unlinked inodes GFS: fsid=kfki:services.3: Found quota changes for 4 IDs GFS: fsid=kfki:services.3: Done GFS: fsid=kfki:home.3: fast statfs start time = 1238197437 GFS: fsid=kfki:services.3: fast statfs start time = 1238197439 SysRq : Show backtrace of all active CPUs CPU0: Pid: 4236, comm: smtp Not tainted (2.6.27.21 #6) EIP: 0060:[] EFLAGS: 00200246 CPU: 0 EIP is at native_read_tsc+0xe/0xf EAX: 60f2a362 EBX: 00000000 ECX: 00000000 EDX: 00000260 ESI: 60f2a296 EDI: 00000001 EBP: 00000000 ESP: efc89eec DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 CR0: 8005003b CR2: b7f5e000 CR3: 30486000 CR4: 000006f0 DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 DR6: ffff0ff0 DR7: 00000400 [] delay_tsc+0x2a/0x47 [] __delay+0x6/0x7 [] _raw_spin_lock+0xe6/0x170 [] _atomic_dec_and_lock+0x31/0x58 [] _atomic_dec_and_lock+0x31/0x58 [] iput+0x1d/0x4a [] d_kill+0x2b/0x44 [] dput+0x78/0x145 [] __fput+0x127/0x17f [] filp_close+0x3e/0x62 [] sys_close+0x5b/0x97 [] sysenter_do_call+0x12/0x35 ======================= SysRq : Show State task PC stack pid father init S f705f110 0 1 0 f705f110 00000086 f705f110 f705f110 00000002 00000002 00000002 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f705f264 c373ed00 f6b08d00 00000000 f75499a8 f75499a8 00000246 c0318668 00000000 00000002 00000000 c01997e8 Call Trace: [] mutex_lock_nested+0x18c/0x25c [] inotify_poll+0x25/0x4b [] schedule_timeout+0x69/0xa1 [] inotify_poll+0x45/0x4b [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] __lock_acquire+0x27b/0x932 [] _spin_unlock+0x14/0x1c [] number+0x25b/0x264 [] kmem_cache_free+0x4e/0x62 [] posix_cpu_timer_set+0x34d/0x405 [] unix_dgram_sendmsg+0x226/0x4e3 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] vsnprintf+0x2dc/0x581 [] validate_chain+0x395/0xf35 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] _spin_lock_irqsave+0x41/0x49 [] do_wait+0x31e/0x342 [] sys_select+0x3f/0x19e [] sysenter_do_call+0x12/0x35 ======================= kthreadd S f7063fac 0 2 0 f705e8a0 00000086 00000002 f7063fac f7063fa0 00000000 f705e8a0 c045f0c0 c0461d00 c0461d00 c0461d00 f7063fb0 f705e9f4 c373ed00 f7871080 00000246 ffff9d9f 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] kthreadd+0x14a/0x14f [] kthreadd+0x0/0x14f [] kernel_thread_helper+0x7/0x1c ======================= migration/0 R running 0 3 2 f705e030 00000082 00000002 f7067f84 f7067f78 00000000 f705e030 c045f0c0 c0461d00 c0461d00 c0461d00 f7067f88 f705e184 c3734d00 f6b08a80 00000046 0000a2f0 00000000 00000001 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] migration_thread+0x147/0x2cc [] _spin_unlock_irqrestore+0x34/0x39 [] migration_thread+0x0/0x2cc [] migration_thread+0x0/0x2cc [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ksoftirqd/0 S f706bfa8 0 4 2 f7069190 00000092 00000002 f706bfa8 f706bf9c 00000000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 f706bfac f70692e4 c3734d00 f6b08800 c040caa0 0000a715 00000000 c045f100 c0124797 ffffffff c0461b80 00000000 00000000 Call Trace: [] __do_softirq+0xa2/0xf9 [] ksoftirqd+0xee/0xf3 [] ksoftirqd+0x0/0xf3 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= migration/1 S f706df84 0 5 2 f7068920 00000082 00000002 f706df84 f706df78 00000000 f7068920 c045f0c0 c0461d00 c0461d00 c0461d00 f706df88 f7068a74 c373ed00 f64d3080 00000046 0000a569 00000000 00000001 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] migration_thread+0x147/0x2cc [] migration_thread+0x0/0x2cc [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ksoftirqd/1 S f7071fa8 0 6 2 f70680b0 00000092 00000002 f7071fa8 f7071f9c 00000000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 f7071fac f7068204 c373ed00 f7cc2300 c040caa0 0000a709 00000000 c045f100 c0124797 ffffffff c0461b80 00000000 00000000 Call Trace: [] __do_softirq+0xa2/0xf9 [] ksoftirqd+0xee/0xf3 [] ksoftirqd+0x0/0xf3 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= events/0 R running 0 7 2 f70729a0 00000082 00000002 f7079f94 f7079f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f7079f98 f7072af4 c3734d00 f78d4080 00000286 0000a715 00000000 c0319cc1 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= events/1 R running 0 8 2 f7072130 00000082 00000002 f707bf94 f707bf88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f707bf98 f7072284 c373ed00 f788c300 00000286 0000a6ea 00000000 c0319cc1 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= khelper S 00000001 0 9 2 f707d290 00000082 f707d610 00000001 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f707d3e4 c373ed00 f75d5580 00000286 00000001 f702b39c c0319cc1 00000000 f707d290 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kblockd/0 S 00000000 0 47 2 f707ca20 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f707cb74 c3734d00 f788c580 00000286 00000001 f70a651c c0319cc1 00000000 00000046 00000046 f70a6524 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kblockd/1 S 00000000 0 48 2 f707c1b0 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f707c304 c373ed00 f7cc2080 00000286 00000001 f70a649c c0319cc1 00000000 00000046 00000046 f70a64a4 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kacpid S c373baa0 0 49 2 f70cf390 00000082 00000818 c373baa0 00000002 f70d1f90 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f70d1f98 f70cf4e4 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kacpi_notify S f70d3f94 0 50 2 f70ceb20 00000082 00000002 f70d3f94 f70d3f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f70d3f98 f70cec74 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ata/0 S 00000001 0 130 2 f70ce2b0 00000082 f70ce630 00000001 00000002 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f70ce404 c3734d00 c03e81a0 00000286 00000001 f710bd1c c0319cc1 00000000 f70ce2b0 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ata/1 S f711bf94 0 131 2 f7119410 00000082 00000002 f711bf94 f711bf88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f711bf98 f7119564 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ata_aux S f711df94 0 132 2 f7118ba0 00000082 00000002 f711df94 f711df88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f711df98 f7118cf4 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ksuspend_usbd S f7117f94 0 133 2 f7118330 00000082 00000002 f7117f94 f7117f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f7117f98 f7118484 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= khubd S 00000001 0 139 2 f7143490 00000082 f7143810 00000001 00000000 c03fad50 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f71435e4 c3734d00 f78ac080 00000286 00000001 00000000 c0319cc1 00000000 f7143490 c0319d8f f7851600 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] hub_thread+0x87f/0xcef [] trace_hardirqs_on_caller+0x9d/0x113 [] _spin_unlock_irq+0x20/0x23 [] finish_task_switch+0x58/0xab [] schedule+0x228/0x708 [] autoremove_wake_function+0x0/0x35 [] hub_thread+0x0/0xcef [] trace_hardirqs_on_caller+0x9d/0x113 [] hub_thread+0x0/0xcef [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kseriod S f7145f84 0 142 2 f7142c20 00000082 00000002 f7145f84 f7145f78 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f7145f88 f7142d74 c3734d00 f78d4a80 00000286 ffff8c9a 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] serio_thread+0xa4/0x2c6 [] autoremove_wake_function+0x0/0x35 [] serio_thread+0x0/0x2c6 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= pdflush S 00000002 0 181 2 f714cd20 00000082 00000092 00000002 00000000 c03edd8c f714cd20 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f714ce74 c3734d00 c03e81a0 00000046 00000000 00000002 00000001 00000000 c0154020 f714cd20 c0319c5a c0153fc2 Call Trace: [] pdflush+0x5e/0x19c [] _spin_unlock_irq+0x20/0x23 [] pdflush+0x0/0x19c [] trace_hardirqs_on_caller+0x9d/0x113 [] pdflush+0x0/0x19c [] pdflush+0xb2/0x19c [] trace_hardirqs_on_caller+0x9d/0x113 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= pdflush R running 0 182 2 f714d590 00000082 00000002 f71b5f94 f71b5f88 00000000 f714d590 c045f0c0 c0461d00 c0461d00 c0461d00 f71b5f98 f714d6e4 c3734d00 f64d3800 00000046 0000a608 00000000 00000001 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] pdflush+0x0/0x19c [] pdflush+0xb2/0x19c [] wb_kupdate+0x0/0xe3 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kswapd0 S f71d9f28 0 183 2 f7120430 00000096 00000002 f71d9f28 f71d9f1c 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f71d9f2c f7120584 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] kswapd+0x3fd/0x420 [] schedule+0x228/0x708 [] autoremove_wake_function+0x0/0x35 [] _spin_unlock_irqrestore+0x34/0x39 [] kswapd+0x0/0x420 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= aio/0 S 00000001 0 184 2 f7120ca0 00000082 f7121020 00000001 00000002 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7120df4 c3734d00 c03e81a0 00000286 00000001 f71f8c1c c0319cc1 00000000 f7120ca0 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= aio/1 S f7303f94 0 185 2 f7121510 00000082 00000002 f7303f94 f7303f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f7303f98 f7121664 c373ed00 c03e81a0 00000286 ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kpsmoused S f6b3df94 0 335 2 f7080230 00000082 00000002 f6b3df94 f6b3df88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f6b3df98 f7080384 c373ed00 c03e81a0 00000286 ffff8bde 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kstriped S f6b83f94 0 339 2 f71c2630 00000082 00000002 f6b83f94 f6b83f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f6b83f98 f71c2784 c373ed00 c03e81a0 00000286 ffff8be4 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= edac-poller S f6b8bf94 0 343 2 f71ca8a0 00000082 00000002 f6b8bf94 f6b8bf88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f6b8bf98 f71ca9f4 c3734d00 c03e81a0 00000286 ffff8be3 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= scsi_eh_0 S f6a1a000 0 1226 2 f7a1b310 00000096 f6a1a050 f6a1a000 00000000 00000046 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f7a1b310 f7a1b464 c3734d00 f6badd00 f6a1a040 00000286 00000286 f6a1a000 00000000 c025645c 00000286 00000282 f6a1a000 Call Trace: [] __scsi_iterate_devices+0x49/0x62 [] scsi_error_handler+0x0/0x2de [] scsi_error_handler+0x0/0x2de [] scsi_error_handler+0x4b/0x2de [] __wake_up_common+0x46/0x68 [] _spin_unlock_irqrestore+0x34/0x39 [] scsi_error_handler+0x0/0x2de [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= scsi_eh_1 S f7c64000 0 1227 2 f7aa5310 00000096 f7c64050 f7c64000 00000000 00000046 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f7aa5310 f7aa5464 c3734d00 f7cc2080 f7c64040 00000286 00000286 f7c64000 00000000 c025645c 00000286 00000282 f7c64000 Call Trace: [] __scsi_iterate_devices+0x49/0x62 [] scsi_error_handler+0x0/0x2de [] scsi_error_handler+0x0/0x2de [] scsi_error_handler+0x4b/0x2de [] __wake_up_common+0x46/0x68 [] _spin_unlock_irqrestore+0x34/0x39 [] scsi_error_handler+0x0/0x2de [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= scsi_eh_2 S f79dbf68 0 1228 2 f7a58e20 00000096 00000002 f79dbf68 f79dbf5c 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f79dbf6c f7a58f74 c373ed00 f7cc2a80 f7c5a840 ffff8e16 00000000 f7c5a800 00000000 ffffffff 00000286 00000000 00000000 Call Trace: [] scsi_error_handler+0x0/0x2de [] scsi_error_handler+0x4b/0x2de [] __wake_up_common+0x46/0x68 [] _spin_unlock_irqrestore+0x34/0x39 [] scsi_error_handler+0x0/0x2de [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= scsi_eh_3 S f7c23f68 0 1229 2 f79a23b0 00000096 00000002 f7c23f68 f7c23f5c 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f7c23f6c f79a2504 c373ed00 f7cc2a80 f7c5a040 ffff8e4a 00000000 f7c5a000 00000000 ffffffff 00000286 00000000 00000000 Call Trace: [] scsi_error_handler+0x0/0x2de [] scsi_error_handler+0x4b/0x2de [] __wake_up_common+0x46/0x68 [] _spin_unlock_irqrestore+0x34/0x39 [] scsi_error_handler+0x0/0x2de [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= md2_raid1 R running 0 1389 2 f798aaa0 00000096 00000002 f78e7f68 f78e7f5c 00000000 f7a41510 c045f0c0 c0461d00 c0461d00 c0461d00 f7c03398 f798abf4 c3734d00 f6b08800 c010659e 0000a715 00000046 f7c03398 00000001 f7c03388 00000046 00000000 00000002 Call Trace: [] do_IRQ+0x40/0x72 [] md_thread+0x0/0xda [] schedule_timeout+0x69/0xa1 [] md_thread+0xb0/0xda [] autoremove_wake_function+0x0/0x35 [] md_thread+0x0/0xda [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= md2_resync R running 0 1392 2 f7ab02b0 00000086 00000002 f7cf3e04 f7cf3df8 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f7cf3e08 f7ab0404 c3734d00 f64d3800 c04b9e00 0000a710 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] msleep_interruptible+0x22/0x2f [] sync_request+0x5c7/0x795 [] is_mddev_idle+0x103/0x10f [] is_mddev_idle+0x9/0x10f [] sync_request+0x0/0x795 [] md_do_sync+0x320/0xdb7 [] validate_chain+0x395/0xf35 [] _spin_unlock_irq+0x20/0x23 [] md_thread+0x0/0xda [] trace_hardirqs_on_caller+0x9d/0x113 [] md_thread+0x0/0xda [] md_thread+0x22/0xda [] _spin_unlock_irqrestore+0x34/0x39 [] md_thread+0x0/0xda [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= md1_raid1 S f7cf1f68 0 1399 2 f7aa4230 00000096 00000002 f7cf1f68 f7cf1f5c 00000000 f798aaa0 c045f0c0 c0461d00 c0461d00 c0461d00 f7cf1f6c f7aa4384 c3734d00 f788c080 c02a19c8 00009aa8 00000000 f7c3a158 00000001 ffffffff 00000046 00000000 00000000 Call Trace: [] md_thread+0x0/0xda [] md_thread+0x0/0xda [] schedule_timeout+0x69/0xa1 [] md_thread+0xb0/0xda [] autoremove_wake_function+0x0/0x35 [] md_thread+0x0/0xda [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= md0_raid1 S f790bf68 0 1408 2 f79a2c20 00000096 00000002 f790bf68 f790bf5c 00000000 f7aa4230 c045f0c0 c0461d00 c0461d00 c0461d00 f7c0d818 f79a2d74 c3734d00 f788c080 c01f4716 ffff9d00 00000046 f7c0d818 00000001 f7c0d808 00000046 00000000 00000002 Call Trace: [] __delay+0x6/0x7 [] md_thread+0x0/0xda [] schedule_timeout+0x69/0xa1 [] md_thread+0xb0/0xda [] autoremove_wake_function+0x0/0x35 [] md_thread+0x0/0xda [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kdmflush S 00000001 0 1414 2 f79bada0 00000082 f79bb120 00000001 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79baef4 c373ed00 f7c91300 00000286 00000001 f7cd051c c0319cc1 00000000 f79bada0 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kdmflush S 00000001 0 1420 2 f79ba530 00000082 f79ba8b0 00000001 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79ba684 c373ed00 f7847800 00000286 00000001 f7c4779c c0319cc1 00000000 f79ba530 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kdmflush S 00000001 0 1428 2 f7a40ca0 00000082 f7a41020 00000001 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a40df4 c373ed00 f7847d00 00000286 00000001 f79c6a1c c0319cc1 00000000 f7a40ca0 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kjournald S f75fccd8 0 1507 2 f7944ba0 00000082 00000046 f75fccd8 00000001 f75fcd10 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7944cf4 c3734d00 f788c580 00000286 00000001 f75fcd10 c0319cc1 00000000 00000046 00000246 00000246 f75fcc14 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] kjournald+0x1bc/0x1c5 [] autoremove_wake_function+0x0/0x35 [] kjournald+0x0/0x1c5 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= udevd S f745fb14 0 1668 1 f7904c20 00000086 00000002 f745fb14 f745fb08 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f745fb18 f7904d74 c373ed00 f7871080 f7819a00 ffff9d83 00000000 00000000 00000002 ffffffff f7819a00 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] pipe_poll+0x24/0x86 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] vsnprintf+0x2dc/0x581 [] number+0x25b/0x264 [] add_to_page_cache_locked+0x46/0x8f [] vsnprintf+0x2dc/0x581 [] _spin_lock+0x31/0x3c [] handle_mm_fault+0x4ea/0x8a4 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] _spin_unlock+0x14/0x1c [] do_wp_page+0x2a2/0x537 [] _spin_lock_irqsave+0x41/0x49 [] do_wait+0x1e7/0x342 [] do_page_fault+0xa5/0x848 [] sys_select+0x3f/0x19e [] sysenter_do_call+0x12/0x35 ======================= gfs2_scand R running 0 2405 2 f7348130 00000092 00000002 f64ebf88 f64ebf7c 00000000 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f64ebf8c f7348284 c373ed00 f788c300 f706e000 0000a680 00000000 00000282 00000282 ffffffff f706e000 00000000 00000000 Call Trace: [] gfs2_scand+0x0/0x63 [gfs2] [] schedule_timeout+0x44/0xa1 [] _read_unlock+0x14/0x1c [] process_timeout+0x0/0x5 [] gfs2_scand+0x56/0x63 [gfs2] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= glock_workque S c3731aa0 0 2407 2 f7972130 00000082 00000afb c3731aa0 00000002 f6505f90 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f6505f98 f7972284 c3734d00 f6507d00 00000286 ffff9118 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= glock_workque S 00000001 0 2408 2 f7a40430 00000082 f7a407b0 00000001 00000002 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a40584 c373ed00 f6bad800 00000286 00000001 f750af9c c0319cc1 00000000 f7a40430 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ksnapd S 00000001 0 2521 2 f71d20b0 00000082 f71d2430 00000001 00000002 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f71d2204 c373ed00 f64d3080 00000286 00000001 f750a99c c0319cc1 00000000 f71d20b0 c0319d8f c012ee03 c013d34e Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] worker_thread+0x0/0x89 [] trace_hardirqs_on_caller+0x9d/0x113 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= ntpd S f41edbf4 0 2693 1 f79ae430 00200082 00000002 f41edbf4 f41edbe8 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f41edbf8 f79ae584 c3734d00 f7856800 c04b9e00 0000a28c 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] udpv6_recvmsg+0x1ac/0x335 [ipv6] [] local_bh_enable_ip+0x79/0xb5 [] udpv6_recvmsg+0x1ac/0x335 [ipv6] [] sock_common_recvmsg+0x3e/0x54 [] sock_recvmsg+0xcf/0xf3 [] sock_sendmsg+0xbc/0xde [] autoremove_wake_function+0x0/0x35 [] sys_recvfrom+0xb8/0x121 [] dnotify_parent+0x1f/0x5f [] vfs_write+0xe5/0x118 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= ntpd R running 0 2694 1 f71cb110 00200082 00000002 f41efbf4 f41efbe8 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f41efbf8 f71cb264 c3734d00 f78d4a80 c04b9e00 0000a3d1 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] do_wp_page+0x2a2/0x537 [] handle_mm_fault+0x1b3/0x8a4 [] _spin_lock+0x31/0x3c [] handle_mm_fault+0x4ea/0x8a4 [] do_page_fault+0xa5/0x848 [] up_read+0x14/0x2a [] do_page_fault+0x156/0x848 [] sock_def_readable+0x12/0x61 [] _read_lock+0x31/0x3c [] _read_unlock+0x14/0x1c [] unix_stream_sendmsg+0x1c3/0x32b [] sock_aio_write+0xcb/0xe8 [] find_lock_page+0x25/0x55 [] sock_aio_write+0x0/0xe8 [] do_sync_readv_writev+0xcb/0x107 [] __do_fault+0x192/0x3f0 [] dnotify_parent+0x1f/0x5f [] do_readv_writev+0x118/0x179 [] sock_aio_write+0x0/0xe8 [] up_read+0x14/0x2a [] do_page_fault+0x156/0x848 [] vfs_writev+0x3c/0x50 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= kjournald S 00000000 0 2727 2 f79e20b0 00000082 f79e2448 00000000 00000001 f642e510 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79e2204 c3734d00 f64d3300 f79e20b0 00000000 00000000 f642e510 c013d220 f79e20b0 00000246 00000246 f642e414 Call Trace: [] mark_held_locks+0x62/0x73 [] kjournald+0x1bc/0x1c5 [] autoremove_wake_function+0x0/0x35 [] kjournald+0x0/0x1c5 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kjournald S f75f3cd8 0 2728 2 f781c330 00000082 00000046 f75f3cd8 00000001 f75f3d10 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f781c484 c3734d00 f6bad580 00000286 00000001 f75f3d10 c0319cc1 00000000 00000046 00000246 00000246 f75f3c14 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] kjournald+0x1bc/0x1c5 [] autoremove_wake_function+0x0/0x35 [] kjournald+0x0/0x1c5 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kjournald R running 0 2729 2 f7a41510 00000082 00000046 f643a8d8 00000001 f643a910 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a41664 c3734d00 f6b08800 00000286 00000001 f643a910 c0319cc1 00000000 00000046 00000246 00000246 f643a814 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] kjournald+0x1bc/0x1c5 [] autoremove_wake_function+0x0/0x35 [] kjournald+0x0/0x1c5 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= getty S f65bbeb4 0 3051 1 f7081310 00000082 00000002 f65bbeb4 f65bbea8 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f65bbeb8 f7081464 c373ed00 f78d4580 00000000 ffff9796 00000000 00000046 f798897c ffffffff f408c000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] mark_held_locks+0x62/0x73 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= getty S 0038c000 0 3052 1 f78b6b20 00000082 00000001 0038c000 00000000 c013eb96 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f78b6c74 c373ed00 f6bada80 00000000 f78b6b20 c03182b9 00000046 f441d17c f441d000 f41c8000 00000046 00000000 Call Trace: [] __lock_acquire+0x27b/0x932 [] mutex_lock_interruptible_nested+0x192/0x2b7 [] schedule_timeout+0x69/0xa1 [] mark_held_locks+0x62/0x73 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= getty S 0038c000 0 3054 1 f78b62b0 00000082 00000001 0038c000 00000000 c013eb96 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f78b6404 c3734d00 f6bad080 00000000 f78b62b0 c03182b9 00000046 f455b17c f455b000 f41d5000 00000046 00000000 Call Trace: [] __lock_acquire+0x27b/0x932 [] mutex_lock_interruptible_nested+0x192/0x2b7 [] schedule_timeout+0x69/0xa1 [] mark_held_locks+0x62/0x73 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= getty S 0038c000 0 3055 1 f7909510 00000082 00000001 0038c000 00000000 c013eb96 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7909664 c373ed00 f75d5d00 00000000 f7909510 c03182b9 00000046 f441d97c f441d800 f4de6000 00000046 00000000 Call Trace: [] __lock_acquire+0x27b/0x932 [] mutex_lock_interruptible_nested+0x192/0x2b7 [] schedule_timeout+0x69/0xa1 [] mark_held_locks+0x62/0x73 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= login S 00000002 0 3056 1 f79d1710 00000082 f79d1710 00000002 00000000 c040c990 f79d1710 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f79d1864 c3734d00 f6b08080 00000000 00000002 00000002 00000000 00000000 c01228e1 00000246 00000246 c040c980 Call Trace: [] do_wait+0x96/0x342 [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= getty S 0038c000 0 3057 1 f7905490 00000082 00000001 0038c000 00000000 c013eb96 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79055e4 c373ed00 f7871580 00000000 f7905490 c03182b9 00000046 f455b97c f455b800 f48f0000 00000046 00000000 Call Trace: [] __lock_acquire+0x27b/0x932 [] mutex_lock_interruptible_nested+0x192/0x2b7 [] schedule_timeout+0x69/0xa1 [] mark_held_locks+0x62/0x73 [] _spin_unlock_irqrestore+0x34/0x39 [] trace_hardirqs_on_caller+0x9d/0x113 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= syslogd S f7a6e0b0 0 3102 1 f7a6e0b0 00000086 f741731c f7a6e0b0 00000002 00000000 f594dd2c c045f0c0 c0461d00 c0461d00 c0461d00 00000020 f7a6e204 c3734d00 f64d3800 00000000 00000000 c030b41c 00000246 00000246 f741730c f741730c 00000000 00000020 Call Trace: [] unix_peer_get+0x11/0x2b [] _spin_unlock+0x14/0x1c [] schedule_timeout+0x69/0xa1 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] __ext3_get_inode_loc+0xba/0x307 [] __ext3_journal_dirty_metadata+0x16/0x3b [] _spin_lock_irqsave+0x41/0x49 [] journal_stop+0x134/0x1c3 [] ext3_ordered_write_end+0xf0/0x14a [] generic_file_buffered_write+0x1a6/0x677 [] mnt_drop_write+0x1d/0xd8 [] _spin_lock+0x31/0x3c [] pipe_write+0x5c/0x450 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] pipe_write+0x0/0x450 [] do_sync_readv_writev+0xcb/0x107 [] mntput_no_expire+0x18/0xef [] autoremove_wake_function+0x0/0x35 [] copy_from_user+0x2d/0x59 [] rw_copy_check_uvector+0x64/0xf4 [] sys_select+0x3f/0x19e [] sigprocmask+0x68/0xc3 [] sys_rt_sigprocmask+0xbf/0xd4 [] sysenter_do_call+0x12/0x35 ======================= dd S f74bff04 0 3122 1 f798b310 00000086 00000002 f74bff04 f74bfef8 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f74bff08 f798b464 c373ed00 f6530300 00000282 ffff9753 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] do_syslog+0x278/0x399 [] autoremove_wake_function+0x0/0x35 [] kmsg_read+0x0/0x36 [] proc_reg_read+0x58/0x79 [] proc_reg_read+0x0/0x79 [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= klogd S 00000000 0 3123 1 f7908430 00000092 f79087c8 00000000 f79087b0 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7908584 c3734d00 f7832300 f7908430 00000000 00000000 f7908430 c0318467 f7928254 c013d34e f7928250 f7928270 Call Trace: [] __mutex_unlock_slowpath+0x89/0xf9 [] trace_hardirqs_on_caller+0x9d/0x113 [] pipe_wait+0x50/0x72 [] autoremove_wake_function+0x0/0x35 [] pipe_read+0xe3/0x3fe [] do_sync_read+0xd2/0x10e [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] do_sync_read+0x0/0x10e [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= sshd S 00000000 0 3146 1 f79043b0 00200086 00000042 00000000 00000042 00000002 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 00000032 f7904504 c3734d00 f75d5a80 f78e1c80 00000020 00200046 00000000 00000002 00000001 f6cf6d1c 00200282 f78e1c80 Call Trace: [] schedule_timeout+0x69/0xa1 [] tcp_poll+0x17/0x11e [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] mark_held_locks+0x62/0x73 [] get_page_from_freelist+0x304/0x48d [] trace_hardirqs_on_caller+0x9d/0x113 [] __lock_acquire+0x27b/0x932 [] mark_held_locks+0x62/0x73 [] _spin_unlock_irq+0x20/0x23 [] trace_hardirqs_on_caller+0x9d/0x113 [] _spin_unlock_irq+0x20/0x23 [] add_to_page_cache_locked+0x70/0x8f [] kmap_atomic+0x1c/0x21 [] shmem_getpage+0x4e9/0x661 [] kmap_atomic+0x1c/0x21 [] __lock_acquire+0x27b/0x932 [] __lock_acquire+0x27b/0x932 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] __do_fault+0x2bc/0x3f0 [] __do_fault+0x192/0x3f0 [] trace_hardirqs_on_caller+0x9d/0x113 [] handle_mm_fault+0x176/0x8a4 [] do_page_fault+0xa5/0x848 [] sys_select+0x3f/0x19e [] trace_hardirqs_on_thunk+0xc/0x10 [] do_page_fault+0x0/0x848 [] trace_hardirqs_on_caller+0x9d/0x113 [] sysenter_do_call+0x12/0x35 ======================= master S 00000000 0 3365 1 f7a31410 00000086 00000046 00000000 00000002 00000001 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a31564 c373ed00 f7847d00 f706e000 00000282 f706e000 00000282 00000282 f4903f40 f706e000 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= pickup S 00000000 0 3382 3365 f79c8e20 00200086 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79c8f74 c3734d00 f6badd00 c04b9e00 00200282 c04b9e00 00200282 00200282 f4dd3f40 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= qmgr R running 0 3384 3365 f7aa4aa0 00200086 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7aa4bf4 c3734d00 f74b5080 c04b9e00 00200282 c04b9e00 00200282 00200282 f5055f40 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= mdadm S 00000000 0 3411 1 f7a5d710 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a5d864 c3734d00 f788c080 c04b9e00 00000286 c04b9e00 00000286 00000286 f79d9b38 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] _spin_lock_irqsave+0x41/0x49 [] scsi_next_command+0x25/0x2f [] scsi_end_request+0x52/0x75 [] scsi_io_completion+0x87/0x3de [] _spin_lock+0x31/0x3c [] _spin_lock_irqsave+0x41/0x49 [] md_wakeup_thread+0x25/0x29 [] md_ioctl+0x10c/0x1082 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] do_open+0xc5/0x2b7 [] blkdev_open+0x0/0x53 [] blkdev_open+0x25/0x53 [] __dentry_open+0x176/0x20a [] nameidata_to_filp+0x35/0x3f [] mutex_lock_nested+0x18c/0x25c [] file_kill+0x13/0x2d [] sys_select+0xd6/0x19e [] filp_close+0x3e/0x62 [] sysenter_do_call+0x12/0x35 ======================= ccsd S f7477b14 0 3472 1 f7a5cea0 00000086 00000002 f7477b14 f7477b08 00000000 f55d3d2c c045f0c0 c0461d00 c0461d00 c0461d00 f7477b18 f7a5cff4 c373ed00 f74fe300 00000000 ffff9d44 00000000 00000002 00000001 ffffffff 00000282 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] unix_poll+0x1a/0xab [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] sched_clock_cpu+0x10f/0x158 [] update_curr+0x40/0x52 [] try_to_wake_up+0xa8/0x137 [] sched_clock_cpu+0x10f/0x158 [] update_curr+0x40/0x52 [] try_to_wake_up+0xa8/0x137 [] autoremove_wake_function+0x14/0x35 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] skb_dequeue+0x50/0x56 [] sys_select+0x3f/0x19e [] filp_close+0x3e/0x62 [] sysenter_do_call+0x12/0x35 ======================= ccsd S f40a5b14 0 3473 1 f79c9690 00000086 00000002 f40a5b14 f40a5b08 00000000 f549932c c045f0c0 c0461d00 c0461d00 c0461d00 f40a5b18 f79c97e4 c373ed00 f74fe300 00000000 ffff994d 00000000 00000002 00000001 ffffffff 00000282 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] unix_poll+0x1a/0xab [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] validate_chain+0x395/0xf35 [] check_noncircular+0xd/0xe7 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] sched_clock_cpu+0x10f/0x158 [] update_curr+0x40/0x52 [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] __lock_acquire+0x27b/0x932 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] validate_chain+0x395/0xf35 [] sock_aio_read+0xdb/0xef [] validate_chain+0x395/0xf35 [] sockfd_lookup_light+0x24/0x41 [] __lock_acquire+0x27b/0x932 [] _spin_unlock_irq+0x20/0x23 [] trace_hardirqs_on_caller+0x9d/0x113 [] _spin_unlock_irq+0x20/0x23 [] finish_task_switch+0x58/0xab [] finish_task_switch+0x26/0xab [] schedule+0x228/0x708 [] dnotify_parent+0x1f/0x5f [] sys_select+0x3f/0x19e [] trace_hardirqs_on_thunk+0xc/0x10 [] trace_hardirqs_on_caller+0x9d/0x113 [] sysenter_do_call+0x12/0x35 ======================= aisexec R running 0 3487 1 f7a343b0 00000082 00000002 f6455bf4 f6455be8 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f6455bf8 f7a34504 c3734d00 f6bad580 c04b9e00 0000a714 00000000 00000282 00000282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] read_tsc+0x6/0x28 [] getnstimeofday+0x44/0x100 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f7525bf4 0 3488 1 f71d2920 00000082 00000002 f7525bf4 f7525be8 00000000 c3734d00 c045f0c0 c0461d00 c0461d00 c0461d00 f7525bf8 f71d2a74 c373ed00 f6bad580 00000001 ffff990e 00000000 00000042 00000001 ffffffff 00000000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] __lock_acquire+0x27b/0x932 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] validate_chain+0x395/0xf35 [] validate_chain+0x395/0xf35 [] find_usage_backwards+0xc/0xce [] find_usage_backwards+0xc/0xce [] find_usage_backwards+0xc/0xce [] __lock_acquire+0x27b/0x932 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] __do_fault+0x2bc/0x3f0 [] __do_fault+0x192/0x3f0 [] handle_mm_fault+0x176/0x8a4 [] do_page_fault+0xa5/0x848 [] up_read+0x14/0x2a [] do_page_fault+0x156/0x848 [] trace_hardirqs_on_thunk+0xc/0x10 [] do_page_fault+0x0/0x848 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f7473df8 0 3489 1 f7a45590 00000096 00000002 f7473df8 f7473dec 00000000 f7a45590 c045f0c0 c0461d00 c0461d00 c0461d00 f7473dfc f7a456e4 c373ed00 f6bad580 f7473e3c ffff993d 00000000 09cb64c0 c0319cc1 ffffffff c0319d8f 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] _spin_unlock_irqrestore+0x34/0x39 [] futex_wait+0x318/0x396 [] __generic_file_aio_write_nolock+0x285/0x4df [] generic_file_aio_write+0x73/0xd8 [] futex_wake+0x52/0xe1 [] validate_chain+0x395/0xf35 [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] __lock_acquire+0x27b/0x932 [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] sys_futex+0x86/0x111 [] trace_hardirqs_on_thunk+0xc/0x10 [] trace_hardirqs_on_caller+0x9d/0x113 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f4c49d7c 0 3506 1 f7a44d20 00000086 00000002 f4c49d7c f4c49d70 00000000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 f4c49d80 f7a44e74 c373ed00 f6bad580 00000000 ffff9d44 00000000 00000000 f4c48000 ffffffff 00000246 00000000 00000000 Call Trace: [] sys_semtimedop+0x5a8/0x7a4 [] copy_from_user+0x2d/0x59 [] copy_from_user+0x2d/0x59 [] verify_iovec+0x2a/0x8a [] sys_sendmsg+0x232/0x237 [] validate_chain+0x395/0xf35 [] __lock_acquire+0x27b/0x932 [] sys_socketcall+0xb7/0x2a9 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f643d498 0 3523 1 f7944330 00000086 00000246 f643d498 00000000 f64a0000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f7944484 c373ed00 f6bad580 00000000 f643d488 00000001 00000000 f64a0000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] copy_from_user+0x2d/0x59 [] copy_from_user+0x2d/0x59 [] verify_iovec+0x2a/0x8a [] sys_sendmsg+0x232/0x237 [] trace_hardirqs_on_thunk+0xc/0x10 [] finish_task_switch+0x58/0xab [] sys_socketcall+0xb7/0x2a9 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f4c0fd7c 0 3527 1 f714c4b0 00000086 00000002 f4c0fd7c f4c0fd70 00000000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 f4c0fd80 f714c604 c373ed00 f6bad580 00000000 ffff9d44 00000000 00000000 f4c0e000 ffffffff 00000246 00000000 00000000 Call Trace: [] sys_semtimedop+0x5a8/0x7a4 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f643d398 0 3533 1 f7a7ea20 00000086 00000246 f643d398 00000000 f75e8000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f7a7eb74 c3734d00 f6bad580 00000000 f643d388 00000003 00000000 f75e8000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] copy_from_user+0x2d/0x59 [] copy_from_user+0x2d/0x59 [] verify_iovec+0x2a/0x8a [] sys_sendmsg+0x232/0x237 [] _spin_lock_irqsave+0x41/0x49 [] __dequeue_entity+0x45/0xaa [] update_curr+0x40/0x52 [] set_next_entity+0x14/0x38 [] pick_next_task_fair+0x8f/0xa8 [] sys_socketcall+0xb7/0x2a9 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f7cd7298 0 3634 1 f7a30330 00000086 00000246 f7cd7298 00000000 f4c60000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f7a30484 c373ed00 f6bad580 00000000 f7cd7288 00000004 00000000 f4c60000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] copy_from_user+0x2d/0x59 [] copy_from_user+0x2d/0x59 [] verify_iovec+0x2a/0x8a [] sys_sendmsg+0x232/0x237 [] sys_socketcall+0xb7/0x2a9 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f65f7b98 0 3669 1 f4cbee20 00000086 00000246 f65f7b98 00000000 f6444000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f4cbef74 c3734d00 f6bad580 00000000 f65f7b88 00000005 00000000 f6444000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] copy_from_user+0x2d/0x59 [] copy_from_user+0x2d/0x59 [] verify_iovec+0x2a/0x8a [] sys_sendmsg+0x232/0x237 [] _spin_lock_irqsave+0x41/0x49 [] __dequeue_entity+0x45/0xaa [] update_curr+0x40/0x52 [] set_next_entity+0x14/0x38 [] pick_next_task_fair+0x8f/0xa8 [] sys_socketcall+0xb7/0x2a9 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f741da98 0 3673 1 f7a0a9a0 00000086 00000246 f741da98 00000000 f4dc8000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f7a0aaf4 c3734d00 f6bad580 00000000 f741da88 00000006 00000000 f4dc8000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] copy_from_user+0x2d/0x59 [] copy_from_user+0x2d/0x59 [] verify_iovec+0x2a/0x8a [] sys_sendmsg+0x232/0x237 [] __dequeue_entity+0x45/0xaa [] update_curr+0x40/0x52 [] set_next_entity+0x14/0x38 [] pick_next_task_fair+0x8f/0xa8 [] sys_socketcall+0xb7/0x2a9 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f741d698 0 3693 1 f79a3490 00000086 00000246 f741d698 00000000 f41ea000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f79a35e4 c3734d00 f6bad580 00000000 f741d688 00000007 00000000 f41ea000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] futex_wait+0xb6/0x396 [] _spin_unlock+0x14/0x1c [] futex_wait+0xeb/0x396 [] try_to_wake_up+0xa8/0x137 [] delay_tsc+0x2a/0x47 [] __delay+0x6/0x7 [] _raw_spin_lock+0xe6/0x170 [] futex_wake+0x52/0xe1 [] _spin_unlock+0x14/0x1c [] futex_wake+0xc0/0xe1 [] do_futex+0x7a/0x951 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= aisexec S f7c1b798 0 3698 1 f65a28a0 00000086 00000246 f7c1b798 00000000 f441a000 00000246 c045f0c0 c0461d00 c0461d00 c0461d00 c01d720f f65a29f4 c3734d00 f6bad580 00000000 f7c1b788 00000008 00000000 f441a000 c01d720f 00000246 00000246 00000246 Call Trace: [] ipc_lock+0x60/0xb8 [] ipc_lock+0x60/0xb8 [] sys_semtimedop+0x5a8/0x7a4 [] sys_ipc+0x6a/0x293 [] trace_hardirqs_on_thunk+0xc/0x10 [] sysenter_do_call+0x12/0x35 ======================= groupd S f6049bf4 0 3504 1 f7a5c630 00000082 00000002 f6049bf4 f6049be8 00000000 f7073210 c045f0c0 c0461d00 c0461d00 c0461d00 f6049bf8 f7a5c784 c373ed00 f7cc2800 f6049efc ffff9d44 00000000 00000002 00000001 ffffffff 00000286 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] unix_poll+0x1a/0xab [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] _read_unlock+0x14/0x1c [] unix_stream_sendmsg+0x1c3/0x32b [] sock_aio_write+0xcb/0xe8 [] sockfd_lookup_light+0x24/0x41 [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] vfs_write+0xe5/0x118 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= fenced S f514fbf4 0 3509 1 f78b7390 00000082 00000002 f514fbf4 f514fbe8 00000000 f3c55d2c c045f0c0 c0461d00 c0461d00 c0461d00 f514fbf8 f78b74e4 c3734d00 f6507800 f514febc ffff9aaa 00000000 00000002 00000001 ffffffff 00000286 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] unix_poll+0x1a/0xab [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] sched_clock_cpu+0x10f/0x158 [] update_curr+0x40/0x52 [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] unix_stream_recvmsg+0x3a0/0x4d8 [] sock_aio_read+0xdb/0xef [] sockfd_lookup_light+0x24/0x41 [] skb_dequeue+0x50/0x56 [] dput+0x97/0x145 [] mntput_no_expire+0x18/0xef [] filp_close+0x3e/0x62 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= dlm_controld S f4c19bf4 0 3513 1 f79bb610 00000082 00000002 f4c19bf4 00000002 00000000 f58aed2c c045f0c0 c0461d00 c0461d00 c0461d00 f4c19bf8 f79bb764 c373ed00 f74b5800 f4c19ebc 00000046 00000000 00000002 00000001 f58aed1c 00000286 f7a8bc80 f4c19ebc Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] schedule_timeout+0x69/0xa1 [] unix_poll+0x1a/0xab [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] _spin_lock_irqsave+0x41/0x49 [] netlink_recvmsg+0x1d1/0x294 [] __wake_up_common+0x46/0x68 [] sock_recvmsg+0xcf/0xf3 [] autoremove_wake_function+0x0/0x35 [] sock_aio_read+0xdb/0xef [] sockfd_lookup_light+0x24/0x41 [] do_sync_read+0xd2/0x10e [] autoremove_wake_function+0x0/0x35 [] sys_recv+0x37/0x3b [] sys_socketcall+0x1a9/0x2a9 [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= gfs_controld S f604bbf4 0 3517 1 f79d0630 00000082 00000002 f604bbf4 f604bbe8 00000000 00000000 c045f0c0 c0461d00 c0461d00 c0461d00 f604bbf8 f79d0784 c373ed00 f74fe800 f79d0630 0000a61d 00000000 f78c6180 f604bed4 ffffffff f893b698 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] _spin_unlock+0x14/0x1c [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] ipc_lock+0x60/0xb8 [] _spin_lock+0x31/0x3c [] sys_semtimedop+0x2bf/0x7a4 [] autoremove_wake_function+0x0/0x35 [] find_lock_page+0x25/0x55 [] shmem_getpage+0x128/0x661 [] futex_wake+0x52/0xe1 [] do_futex+0x7a/0x951 [] do_page_fault+0xa5/0x848 [] sys_futex+0x86/0x111 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= clvmd S f403fb14 0 3627 1 f781a230 00000086 00000002 f403fb14 f403fb08 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f403fb18 f781a384 c3734d00 f7847580 c04b9e00 00009ed0 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] default_wake_function+0x0/0x8 [] _spin_lock_irqsave+0x41/0x49 [] _spin_lock_irqsave+0x41/0x49 [] sched_clock_cpu+0x10f/0x158 [] update_curr+0x40/0x52 [] try_to_wake_up+0xa8/0x137 [] __wake_up_common+0x46/0x68 [] unix_stream_recvmsg+0x3a0/0x4d8 [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] sock_aio_read+0xdb/0xef [] sockfd_lookup_light+0x24/0x41 [] sys_recvfrom+0x11f/0x121 [] _spin_unlock_irq+0x20/0x23 [] finish_task_switch+0x58/0xab [] finish_task_switch+0x26/0xab [] schedule+0x228/0x708 [] dnotify_parent+0x1f/0x5f [] sys_select+0xd6/0x19e [] sysenter_do_call+0x12/0x35 ======================= clvmd S 00000002 0 3638 1 f4d77610 00000086 f4d77610 00000002 00000000 f7582da0 f4d77610 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f4d77764 c3734d00 f7847580 f7582dd8 00000282 f7582dd8 b7d55368 c0319cc1 00000046 00000246 00000246 f7582d90 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] device_read+0x12d/0x347 [dlm] [] default_wake_function+0x0/0x8 [] device_read+0x0/0x347 [dlm] [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= clvmd S 00000002 0 3639 1 f79b24b0 00000096 f79b24b0 00000002 00000000 c082c81c f79b24b0 c045f0c0 c0461d00 c0461d00 c0461d00 00000046 f79b2604 c3734d00 f7847580 f4d75e3c 00000282 00000000 080b1fa4 c0319cc1 00000046 00000046 f4d75e3c 00000282 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] futex_wait+0x318/0x396 [] futex_wake+0x52/0xe1 [] _spin_unlock+0x14/0x1c [] default_wake_function+0x0/0x8 [] do_futex+0xbc/0x951 [] net_rx_action+0x85/0x1db [] sys_futex+0x86/0x111 [] _local_bh_enable+0x44/0xa0 [] do_IRQ+0x40/0x72 [] sysenter_do_call+0x12/0x35 ======================= dlm_astd S f4c05fa8 0 3628 2 f7a7e1b0 00000092 00000002 f4c05fa8 f4c05f9c 00000000 f7a7e1b0 c045f0c0 c0461d00 c0461d00 c0461d00 f4c05fac f7a7e304 c373ed00 f7cc2300 00000246 0000a709 00000000 00000046 00000046 ffffffff 00000000 00000000 00000000 Call Trace: [] gdlm_bast+0x0/0xa4 [gfs] [] dlm_astd+0x62/0x155 [dlm] [] dlm_astd+0x0/0x155 [dlm] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_scand R running 0 3629 2 f79b3590 00000092 00000002 f49c7f88 f49c7f7c 00000000 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f49c7f8c f79b36e4 c373ed00 f788c300 f706e000 0000a61c 00000000 00000282 00000282 ffffffff f706e000 00000000 00000000 Call Trace: [] dlm_scand+0x0/0x77 [dlm] [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] dlm_scand+0x69/0x77 [dlm] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_recv/0 R running 0 3630 2 f7ab0b20 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7ab0c74 c3734d00 f6b08800 00000286 00000001 f504251c c0319cc1 00000000 00000046 00000046 f5042524 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_recv/1 S f49cbf94 0 3631 2 f7a1a230 00000082 00000002 f49cbf94 f49cbf88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f49cbf98 f7a1a384 c373ed00 f74fe800 00000286 ffff9c7c 00000000 c0319cc1 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_send R running 0 3632 2 f79aeca0 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79aedf4 c3734d00 f6b08800 00000286 00000001 f504269c c0319cc1 00000000 00000046 00000046 f50426a4 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_recoverd S f49cff80 0 3633 2 f71c2ea0 00000086 00000002 f49cff80 f49cff74 00000000 00002bd4 c045f0c0 c0461d00 c0461d00 c0461d00 f49cff84 f71c2ff4 c373ed00 f7871080 00000002 ffff9caf 00000000 f71c2ea0 00000046 ffffffff f7001280 00000000 00000000 Call Trace: [] dlm_recoverd+0x163/0x55a [dlm] [] __wake_up_common+0x46/0x68 [] dlm_recoverd+0x0/0x55a [dlm] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kdmflush S 00000000 0 3646 2 f71ca030 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f71ca184 c373ed00 f7871080 00000286 00000001 f5d6c91c c0319cc1 00000000 00000046 00000046 f5d6c924 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= kdmflush S 00000000 0 3652 2 f79b2d20 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79b2e74 c373ed00 f7871080 00000286 00000001 f5d6bb9c c0319cc1 00000000 00000046 00000046 f5d6bba4 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] worker_thread+0x0/0x89 [] worker_thread+0x7a/0x89 [] autoremove_wake_function+0x0/0x35 [] worker_thread+0x0/0x89 [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= lock_dlm1 S f74d5f50 0 3670 2 f7908ca0 00000086 00000002 f74d5f50 f74d5f44 00000000 f7908ca0 c045f0c0 c0461d00 c0461d00 c0461d00 f74d5f54 f7908df4 c3734d00 f7cc2d00 00000246 0000a6d9 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] gdlm_thread+0x112/0x74a [gfs] [] autoremove_wake_function+0x0/0x35 [] gdlm_thread1+0x0/0xa [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= lock_dlm2 S 00000002 0 3671 2 f7839590 00000086 00000000 00000002 00000000 f4070964 f7839590 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f78396e4 c373ed00 f788c300 00000246 00000000 00000002 00000000 00000000 f8a45d59 00000246 00000246 f4070954 Call Trace: [] gdlm_thread+0x12d/0x74a [gfs] [] gdlm_thread+0x112/0x74a [gfs] [] autoremove_wake_function+0x0/0x35 [] gdlm_thread2+0x0/0x7 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_recoverd S f893b428 0 3672 2 f7a30ba0 00000086 00000000 f893b428 f7a30ba0 f40b2bf4 00002bd4 c045f0c0 c0461d00 c0461d00 c0461d00 f65cf800 f7a30cf4 c373ed00 f6bad580 00000002 00000000 00000000 f7a30ba0 00000046 00000000 f7001280 f704b000 f7001280 Call Trace: [] dlm_recoverd+0x163/0x55a [dlm] [] __wake_up_common+0x46/0x68 [] dlm_recoverd+0x0/0x55a [dlm] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_scand R running 0 3684 2 f4d76da0 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f4d76ef4 c3734d00 f6bad580 c04b9e00 00000286 c04b9e00 00000286 00000286 f6413fa4 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] gfs_scand+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_scand+0x39/0x48 [gfs] [] gfs_scand+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_glockd S f4c63f94 0 3685 2 f7838d20 00000082 00000002 f4c63f94 f4c63f88 00000000 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 f4c63f98 f7838e74 c373ed00 f7cc2300 00000286 0000a663 00000000 c0319cc1 00000000 ffffffff 00000046 00000000 00000000 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] gfs_glockd+0x80/0xb4 [gfs] [] autoremove_wake_function+0x0/0x35 [] gfs_glockd+0x0/0xb4 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_recoverd S f490df80 0 3687 2 f7a0a130 00000086 00000002 f490df80 f490df74 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f490df84 f7a0a284 c3734d00 f6bad580 c04b9e00 00009f00 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_recoverd+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_recoverd+0x39/0x48 [gfs] [] gfs_recoverd+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_logd R running 0 3688 2 f7965190 00000082 00000002 f490ff30 f490ff24 00000000 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f490ff34 f79652e4 c373ed00 f788c300 f706e000 0000a6e6 00000000 00000286 00000286 ffffffff f706e000 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_logd+0x23/0xb7 [gfs] [] process_timeout+0x0/0x5 [] gfs_logd+0x41/0xb7 [gfs] [] gfs_logd+0x0/0xb7 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_quotad R running 0 3689 2 f73489a0 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7348af4 c3734d00 f7cc2d00 c04b9e00 00000282 c04b9e00 00000282 00000282 f4d41f88 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] gfs_quotad+0x88/0x16d [gfs] [] process_timeout+0x0/0x5 [] gfs_quotad+0xa6/0x16d [gfs] [] gfs_quotad+0x0/0x16d [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_inoded R running 0 3690 2 f781d410 00000086 00000002 f4d43f80 f4d43f74 00000000 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f4d43f84 f781d564 c373ed00 f788c300 f706e000 0000a4f1 00000000 00000286 00000286 ffffffff f706e000 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_inoded+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_inoded+0x39/0x48 [gfs] [] gfs_inoded+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= lock_dlm1 S 00000002 0 3694 2 f4c4f710 00000086 c011abae 00000002 00000000 f78a9764 f4c4f710 c045f0c0 c0461d00 c0461d00 c0461d00 00000246 f4c4f864 c3734d00 f6b08800 00000246 00000000 00000002 00000000 00000000 f8a45d59 00000246 00000246 f78a9754 Call Trace: [] try_to_wake_up+0xa8/0x137 [] gdlm_thread+0x12d/0x74a [gfs] [] gdlm_thread+0x112/0x74a [gfs] [] autoremove_wake_function+0x0/0x35 [] gdlm_thread1+0x0/0xa [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= lock_dlm2 S f49f7f50 0 3695 2 f4c4eea0 00000086 00000002 f49f7f50 f49f7f44 00000000 f4c4eea0 c045f0c0 c0461d00 c0461d00 c0461d00 f49f7f54 f4c4eff4 c3734d00 f6b08800 00000246 0000a715 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 Call Trace: [] gdlm_thread+0x112/0x74a [gfs] [] autoremove_wake_function+0x0/0x35 [] gdlm_thread2+0x0/0x7 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= dlm_recoverd S f893b428 0 3697 2 f65a3110 00000086 00000000 f893b428 f65a3110 f5046bf4 00002bd4 c045f0c0 c0461d00 c0461d00 c0461d00 f6586800 f65a3264 c3734d00 f64d3300 00000002 00000000 00000000 f65a3110 00000046 00000000 f7001280 f700d800 f7001280 Call Trace: [] dlm_recoverd+0x163/0x55a [dlm] [] __wake_up_common+0x46/0x68 [] dlm_recoverd+0x0/0x55a [dlm] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_scand R running 0 3702 2 f4dfe920 00000086 00000046 00000000 00000002 00000001 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f4dfea74 c373ed00 f7847d00 f706e000 00000286 f706e000 00000286 00000286 f4dfdfa4 f706e000 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] gfs_scand+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_scand+0x39/0x48 [gfs] [] gfs_scand+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_glockd S f0499aec 0 3703 2 f4dff190 00000082 f0499b18 f0499aec 00000246 f8b81e4c 00000046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f4dff2e4 c373ed00 f7cc2300 00000286 00000001 f65a1fbc c0319cc1 00000000 00000046 00000046 f8b83504 00000286 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] gfs_glockd+0x80/0xb4 [gfs] [] autoremove_wake_function+0x0/0x35 [] gfs_glockd+0x0/0xb4 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_recoverd S f26cff80 0 3705 2 f65a2030 00000086 00000002 f26cff80 f26cff74 00000000 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f26cff84 f65a2184 c373ed00 f788c300 f706e000 00009f5d 00000000 00000286 00000286 ffffffff f706e000 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_recoverd+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_recoverd+0x39/0x48 [gfs] [] gfs_recoverd+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_logd R running 0 3706 2 f4c4e630 00000082 00000002 f26d1f30 f26d1f24 00000000 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f26d1f34 f4c4e784 c373ed00 f788c300 f706e000 0000a6db 00000000 00000286 00000286 ffffffff f706e000 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_logd+0x23/0xb7 [gfs] [] process_timeout+0x0/0x5 [] gfs_logd+0x41/0xb7 [gfs] [] gfs_logd+0x0/0xb7 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_quotad R running 0 3707 2 f4dfe0b0 00000086 00000002 f26d3f64 f26d3f58 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f26d3f68 f4dfe204 c3734d00 f6bad580 c04b9e00 0000a557 00000000 00000282 00000282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_quotad+0x88/0x16d [gfs] [] process_timeout+0x0/0x5 [] gfs_quotad+0xa6/0x16d [gfs] [] gfs_quotad+0x0/0x16d [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= gfs_inoded R running 0 3708 2 f26d5210 00000086 00000002 f26d7f80 f26d7f74 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f26d7f84 f26d5364 c3734d00 f64d3580 c04b9e00 0000a668 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] gfs_inoded+0x1b/0x48 [gfs] [] process_timeout+0x0/0x5 [] gfs_inoded+0x39/0x48 [gfs] [] gfs_inoded+0x0/0x48 [gfs] [] kthread+0x34/0x55 [] kthread+0x0/0x55 [] kernel_thread_helper+0x7/0x1c ======================= atd S f050ff2c 0 3845 1 f7973210 00000082 00000002 f050ff2c f050ff20 00000000 f793f15c c045f0c0 c0461d00 c0461d00 c0461d00 f050ff30 f7973364 c373ed00 f6530d00 f050ff60 ffff9fc5 00000000 0000000c c045ef20 ffffffff 00000352 00000000 00000000 Call Trace: [] do_nanosleep+0x6d/0x96 [] hrtimer_nanosleep+0x50/0xad [] hrtimer_wakeup+0x0/0x18 [] sys_nanosleep+0x58/0x5c [] sysenter_do_call+0x12/0x35 ======================= cron S f04fff2c 0 3860 1 f7946630 00000082 00000002 f04fff2c f04fff20 00000000 c04782ec c045f0c0 c0461d00 c0461d00 c0461d00 f04fff30 f7946784 c3734d00 f6530a80 f04fff60 0000a1b1 00000000 000000a6 c045ef20 ffffffff 000000b4 00000000 00000000 Call Trace: [] do_nanosleep+0x6d/0x96 [] hrtimer_nanosleep+0x50/0xad [] hrtimer_wakeup+0x0/0x18 [] sys_nanosleep+0x58/0x5c [] sysenter_do_call+0x12/0x35 ======================= bash S f08ffeb4 0 4049 3056 f79640b0 00000082 00000002 f08ffeb4 f08ffea8 00000000 c03eb0b0 c045f0c0 c0461d00 c0461d00 c0461d00 f08ffeb8 f7964204 c3734d00 f7940d00 00000000 0000a6fd 00000000 00000046 f718397c ffffffff f7184000 00000000 00000000 Call Trace: [] schedule_timeout+0x69/0xa1 [] _spin_lock_irqsave+0x41/0x49 [] read_chan+0x1e1/0x640 [] default_wake_function+0x0/0x8 [] tty_read+0x70/0x9d [] read_chan+0x0/0x640 [] tty_read+0x0/0x9d [] vfs_read+0x85/0x11b [] sys_read+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= mailmanctl S f48f3f24 0 4221 1 f71c3710 00200082 00000002 f48f3f24 f48f3f18 00000000 f71c3710 c045f0c0 c0461d00 c0461d00 c0461d00 f48f3f28 f71c3864 c373ed00 f6530080 00000000 00007f95 00000000 00000000 00000000 ffffffff 00200246 00000000 00000000 Call Trace: [] do_wait+0x28c/0x342 [] default_wake_function+0x0/0x8 [] sys_wait4+0x69/0xa4 [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4222 4221 f4cbe5b0 00000086 00000002 f05c3b14 f05c3b08 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 f05c3b18 f4cbe704 c3734d00 f64d3300 c04b9e00 0000a6fd 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] wait_for_common+0x26/0x11b [] wait_for_common+0x26/0x11b [] _spin_unlock_irq+0x20/0x23 [] wait_for_common+0xcd/0x11b [] find_get_page+0xa2/0xbf [] gfs_sort+0x80/0x103 [gfs] [] do_filldir_main+0x28/0x1d1 [gfs] [] compare_dents+0x0/0x65 [gfs] [] getbuf+0xfe/0x196 [gfs] [] wait_for_common+0x26/0x11b [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] filldir64+0x0/0xc5 [] _spin_unlock+0x14/0x1c [] gfs_readdir+0x329/0x3c2 [gfs] [] filldir_reg_func+0x0/0x1c9 [gfs] [] copy_to_user+0x2f/0x4d [] filldir64+0x0/0xc5 [] file_kill+0x13/0x2d [] sys_select+0xd6/0x19e [] filp_close+0x3e/0x62 [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4223 4221 f4cbf690 00000086 00000002 efd09b14 efd09b08 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 efd09b18 f4cbf7e4 c3734d00 f7cc2d00 c04b9e00 0000a6d9 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] wait_for_common+0x26/0x11b [] find_get_page+0xa2/0xbf [] find_get_page+0xa2/0xbf [] find_get_page+0xd/0xbf [] do_filldir_main+0x28/0x1d1 [gfs] [] compare_dents+0x0/0x65 [gfs] [] wait_for_common+0x26/0x11b [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] filldir64+0x0/0xc5 [] _spin_unlock+0x14/0x1c [] gfs_readdir+0x329/0x3c2 [gfs] [] filldir_reg_func+0x0/0x1c9 [gfs] [] copy_to_user+0x2f/0x4d [] filldir64+0x0/0xc5 [] file_kill+0x13/0x2d [] sys_select+0xd6/0x19e [] copy_to_user+0x2f/0x4d [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4224 4221 f79729a0 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7972af4 c3734d00 f64d3580 c04b9e00 00000286 c04b9e00 00000286 00000286 f0c3bb38 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] e1000_clean_rx_irq+0x0/0x41a [e1000] [] __pollwait+0x0/0xba [] wait_for_common+0x26/0x11b [] find_get_page+0xa2/0xbf [] find_get_page+0xa2/0xbf [] gfs_sort+0x80/0x103 [gfs] [] do_filldir_main+0x28/0x1d1 [gfs] [] compare_dents+0x0/0x65 [gfs] [] wait_for_common+0x26/0x11b [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] filldir64+0x0/0xc5 [] _spin_unlock+0x14/0x1c [] gfs_readdir+0x329/0x3c2 [gfs] [] filldir_reg_func+0x0/0x1c9 [gfs] [] copy_to_user+0x2f/0x4d [] filldir64+0x0/0xc5 [] file_kill+0x13/0x2d [] sys_select+0xd6/0x19e [] filp_close+0x3e/0x62 [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4225 4221 f7945410 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7945564 c3734d00 f6b08300 c04b9e00 00000286 c04b9e00 00000286 00000286 f04ddb38 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] task_rq_lock+0x2e/0x55 [] __pollwait+0x0/0xba [] wait_for_common+0x26/0x11b [] wait_for_common+0x26/0x11b [] _spin_unlock_irq+0x20/0x23 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] dput+0x97/0x145 [] gfs_drevalidate+0x1a6/0x1f0 [gfs] [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] copy_to_user+0x2f/0x4d [] cp_new_stat64+0xf9/0x10b [] sys_select+0xd6/0x19e [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4226 4221 f79e3190 00000086 00000002 f1171b14 f1171b08 00000000 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 f1171b18 f79e32e4 c373ed00 f788c300 f706e000 0000a6da 00000000 00000286 00000286 ffffffff f706e000 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] wait_for_common+0x26/0x11b [] wait_for_common+0x26/0x11b [] _spin_unlock_irq+0x20/0x23 [] wait_for_common+0xcd/0x11b [] find_get_page+0xa2/0xbf [] gfs_sort+0x80/0x103 [gfs] [] do_filldir_main+0x28/0x1d1 [gfs] [] compare_dents+0x0/0x65 [gfs] [] getbuf+0xfe/0x196 [gfs] [] wait_for_common+0x26/0x11b [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] filldir64+0x0/0xc5 [] _spin_unlock+0x14/0x1c [] gfs_readdir+0x329/0x3c2 [gfs] [] filldir_reg_func+0x0/0x1c9 [gfs] [] copy_to_user+0x2f/0x4d [] filldir64+0x0/0xc5 [] dput+0xa9/0x145 [] sys_select+0xd6/0x19e [] filp_close+0x3e/0x62 [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4227 4221 0000008a 08fb5358 f8a4ddb4 f4c65fa0 00000000 f8a4e41f ef825530 f4c65fa0 08fb5358 f8a4de07 0000008a f8a4de1f f4c65fa0 00000000 f658cd80 c01714e2 f4c65fa0 00000967 f658cd80 fffffff7 08fb5358 f4c64000 c0171995 f4c65fa0 Call Trace: [] __gfs_write+0x9a/0xb1 [gfs] [] do_write_buf+0x0/0x73d [gfs] [] gfs_write+0x0/0x1d [gfs] [] gfs_write+0x18/0x1d [gfs] [] vfs_write+0x87/0x118 [] sys_write+0x41/0x6a [] sysenter_do_call+0x12/0x35 ======================= python R running 0 4228 4221 f79af510 00000082 f48e2ccc efc7dedc 00000246 00000000 00000002 c045f0c0 c0461d00 c0461d00 c0461d00 f48e2b64 f79af668 c3734d00 f6b08800 00000246 f48e2ccc f48e2b64 c35dfba4 efc7dedc c0319867 00000000 f8a4c788 00000286 Call Trace: [] _spin_unlock+0x14/0x1c [] gfs_writepage+0x72/0xae [gfs] [] __cond_resched+0x22/0x37 [] _cond_resched+0x24/0x2a [] write_cache_pages+0x2c8/0x389 [] __writepage+0x0/0x28 [] generic_writepages+0x20/0x29 [] do_writepages+0x33/0x35 [] __filemap_fdatawrite_range+0x65/0x71 [] filemap_fdatawrite+0x23/0x27 [] gfs_sync_page_i+0x56/0x5c [gfs] [] gfs_drop_inode+0x55/0x5d [gfs] [] iput+0x44/0x4a [] do_unlinkat+0xc6/0x140 [] do_munmap+0x183/0x1e0 [] sysenter_do_call+0x12/0x35 ======================= python S 00000000 0 4229 4221 f7ab1390 00000086 00000046 00000000 00000002 00000001 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7ab14e4 c373ed00 f6b08580 f706e000 00000286 f706e000 00000286 00000286 f0599b38 f706e000 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_select+0x3ef/0x4cb [] do_select+0x14/0x4cb [] __pollwait+0x0/0xba [] wait_for_common+0x26/0x11b [] wait_for_common+0x26/0x11b [] _spin_unlock_irq+0x20/0x23 [] wait_for_common+0xcd/0x11b [] find_get_page+0xa2/0xbf [] gfs_sort+0x80/0x103 [gfs] [] do_filldir_main+0x28/0x1d1 [gfs] [] compare_dents+0x0/0x65 [gfs] [] getbuf+0xfe/0x196 [gfs] [] wait_for_common+0x26/0x11b [] core_sys_select+0x1f3/0x335 [] core_sys_select+0x22/0x335 [] gfs_glock_dq+0xf0/0x157 [gfs] [] _spin_lock+0x31/0x3c [] gfs_glock_dq+0x9a/0x157 [gfs] [] filldir64+0x0/0xc5 [] _spin_unlock+0x14/0x1c [] gfs_readdir+0x329/0x3c2 [gfs] [] filldir_reg_func+0x0/0x1c9 [gfs] [] filldir64+0x0/0xc5 [] dput+0xa9/0x145 [] sys_select+0xd6/0x19e [] filp_close+0x3e/0x62 [] sysenter_do_call+0x12/0x35 ======================= smtpd S 00000000 0 4230 3365 f71d3190 00200086 00200046 00000000 00000002 00000001 00200046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f71d32e4 c373ed00 f788c800 00200282 00000001 f78d1a6c c0319cc1 00000000 00200046 00200046 f78d1a6c 00200282 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] flock_lock_file_wait+0x177/0x275 [] autoremove_wake_function+0x0/0x35 [] sys_flock+0x11d/0x122 [] sysenter_do_call+0x12/0x35 ======================= cleanup R running 0 4233 3365 f7a585b0 00200086 00200046 00000000 00000002 00000001 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a58704 c373ed00 f788c580 f706e000 00200282 f706e000 00200282 00200282 efcf5f40 f706e000 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= smtp R running 0 4234 3365 f7a444b0 00200082 00000002 efd8fbf4 efd8fbe8 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 efd8fbf8 f7a44604 c3734d00 f78d4080 c04b9e00 0000a715 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] ip_finish_output+0x112/0x26b [] ip_local_out+0x15/0x17 [] ip_queue_xmit+0x195/0x327 [] dev_queue_xmit+0x36/0x509 [] tcp_transmit_skb+0x363/0x659 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] tcp_sendmsg+0x704/0xa99 [] local_bh_enable_ip+0x79/0xb5 [] tcp_sendmsg+0x704/0xa99 [] tcp_recvmsg+0x2cf/0x7ff [] local_bh_enable_ip+0x79/0xb5 [] tcp_recvmsg+0x2cf/0x7ff [] sock_common_recvmsg+0x3e/0x54 [] sock_aio_write+0xcb/0xe8 [] do_sync_write+0xd2/0x10e [] sock_setsockopt+0x126/0x546 [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] read_tsc+0x6/0x28 [] getnstimeofday+0x44/0x100 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= smtp R running 0 4235 3365 f7a7f290 00200082 00000002 efc97bf4 efc97be8 00000000 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 efc97bf8 f7a7f3e4 c3734d00 f788cd00 c04b9e00 0000a715 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 Call Trace: [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] do_sys_poll+0x238/0x311 [] __pollwait+0x0/0xba [] default_wake_function+0x0/0x8 [] ip_finish_output+0x112/0x26b [] ip_local_out+0x15/0x17 [] ip_queue_xmit+0x195/0x327 [] dev_queue_xmit+0x36/0x509 [] tcp_transmit_skb+0x363/0x659 [] release_sock+0x24/0xac [] _spin_lock_bh+0x36/0x41 [] tcp_sendmsg+0x704/0xa99 [] local_bh_enable_ip+0x79/0xb5 [] tcp_sendmsg+0x704/0xa99 [] tcp_recvmsg+0x2cf/0x7ff [] local_bh_enable_ip+0x79/0xb5 [] tcp_recvmsg+0x2cf/0x7ff [] sock_common_recvmsg+0x3e/0x54 [] sock_aio_write+0xcb/0xe8 [] do_sync_write+0xd2/0x10e [] sock_setsockopt+0x126/0x546 [] autoremove_wake_function+0x0/0x35 [] dnotify_parent+0x1f/0x5f [] read_tsc+0x6/0x28 [] getnstimeofday+0x44/0x100 [] copy_to_user+0x2f/0x4d [] sys_poll+0x2d/0x71 [] sysenter_do_call+0x12/0x35 ======================= smtp R running 0 4236 3365 00000000 c0319867 f6417780 c02cea4b f0478480 f7852c8c 00000000 f7549880 00200082 00000009 00000009 c049d362 f7852c80 c03eb0a0 00200082 0003ffff 00023bc3 c0319cc1 00000000 00000002 c03eb070 00200082 0003ffff c03eb070 Call Trace: [] _spin_unlock+0x14/0x1c [] printk+0x1b/0x1f [] __print_symbol+0x21/0x2a [] __print_symbol+0x21/0x2a [] printk+0x1b/0x1f [] printk+0x1b/0x1f [] print_trace_address+0x39/0x43 [] __kernel_text_address+0x18/0x23 [] dump_trace+0x5e/0xe9 [] show_trace_log_lvl+0x21/0x37 [] show_stack_log_lvl+0xaf/0xd7 [] show_stack+0x2f/0x35 [] show_state_filter+0x4f/0x84 [] __handle_sysrq+0x7a/0x115 [] kbd_event+0x420/0x6db [] kbd_event+0x0/0x6db [] input_pass_event+0xb7/0xb9 [] input_pass_event+0x15/0xb9 [] input_event+0x54/0x68 [] atkbd_interrupt+0x469/0x5b3 [] _spin_lock_irqsave+0x41/0x49 [] serio_interrupt+0x32/0x6b [] i8042_interrupt+0x108/0x23a [] handle_IRQ_event+0x1e/0x48 [] handle_edge_irq+0xb4/0x10a [] do_IRQ+0x3b/0x72 [] trace_hardirqs_on_thunk+0xc/0x10 [] common_interrupt+0x28/0x30 [] delay_tsc+0x12/0x47 [] __delay+0x6/0x7 [] _raw_spin_lock+0xe6/0x170 [] _atomic_dec_and_lock+0x31/0x58 [] _atomic_dec_and_lock+0x31/0x58 [] iput+0x1d/0x4a [] d_kill+0x2b/0x44 [] dput+0x78/0x145 [] __fput+0x127/0x17f [] filp_close+0x3e/0x62 [] sys_close+0x5b/0x97 [] sysenter_do_call+0x12/0x35 ======================= smtp S 00000000 0 4237 3365 f7a34c20 00200086 00200046 00000000 00000002 00000001 00200046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a34d74 c373ed00 f78d4300 00200282 00000001 ef4c2230 c0319cc1 00000000 00200046 00200046 ef4c2230 00200282 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] flock_lock_file_wait+0x177/0x275 [] autoremove_wake_function+0x0/0x35 [] sys_flock+0x11d/0x122 [] sysenter_do_call+0x12/0x35 ======================= scache S 00000000 0 4259 3365 f798a230 00200086 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f798a384 c3734d00 f74b5300 c04b9e00 00200282 c04b9e00 00200282 00200282 efd93f40 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= smtp S 00000000 0 4261 3365 f7947710 00200086 00200046 00000000 00000002 00000001 00200046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7947864 c3734d00 f7856080 00200282 00000001 f78d19f0 c0319cc1 00000000 00200046 00200046 f78d19f0 00200282 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] flock_lock_file_wait+0x177/0x275 [] autoremove_wake_function+0x0/0x35 [] sys_flock+0x11d/0x122 [] sysenter_do_call+0x12/0x35 ======================= smtp S 00000000 0 4262 3365 f79d0ea0 00200086 00200046 00000000 00000002 00000001 00200046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79d0ff4 c3734d00 f74b5580 00200282 00000001 ef4c22ac c0319cc1 00000000 00200046 00200046 ef4c22ac 00200282 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] flock_lock_file_wait+0x177/0x275 [] autoremove_wake_function+0x0/0x35 [] sys_flock+0x11d/0x122 [] sysenter_do_call+0x12/0x35 ======================= smtp S 00000000 0 4263 3365 f7946ea0 00200086 00200046 00000000 00000002 00000001 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7946ff4 c373ed00 f74b5d00 f706e000 00200282 f706e000 00200282 00200282 ef455f40 f706e000 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= smtp S f5d81f50 0 4264 3365 f7a1aaa0 00200086 00000002 f5d81f50 f5d81f44 00000000 00200046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a1abf4 c3734d00 f7847800 00200282 00000001 f6441be0 c0319cc1 00000000 00200046 00200046 f6441be0 00200282 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] flock_lock_file_wait+0x177/0x275 [] autoremove_wake_function+0x0/0x35 [] sys_flock+0x11d/0x122 [] sysenter_do_call+0x12/0x35 ======================= smtp S 00000000 0 4265 3365 f781aaa0 00200086 00200046 00000000 00000002 00000001 00200046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f781abf4 c3734d00 f6507300 00200282 00000001 f6441c5c c0319cc1 00000000 00200046 00200046 f6441c5c 00200282 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] flock_lock_file_wait+0x177/0x275 [] autoremove_wake_function+0x0/0x35 [] sys_flock+0x11d/0x122 [] sysenter_do_call+0x12/0x35 ======================= smtp S f096ff50 0 4266 3365 f7a0b210 00200086 00000002 f096ff50 f096ff44 00000000 00200046 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a0b364 c3734d00 f64d3d00 00200282 00000001 f6441dd0 c0319cc1 00000000 00200046 00200046 f6441dd0 00200282 Call Trace: [] _spin_lock_irqsave+0x41/0x49 [] flock_lock_file_wait+0x177/0x275 [] autoremove_wake_function+0x0/0x35 [] sys_flock+0x11d/0x122 [] sysenter_do_call+0x12/0x35 ======================= trivial-rewri R running 0 4274 3365 f79c85b0 00200086 00200046 00000000 00000002 00000001 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f79c8704 c373ed00 f7cc2580 f706e000 00200282 f706e000 00200282 00200282 ef493f40 f706e000 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= smtpd R running 0 4275 3365 f781cba0 00200086 00200046 00000000 00000002 00000001 f706e000 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f781ccf4 c373ed00 f7cc2080 f706e000 00200282 f706e000 00200282 00200282 efc59f40 f706e000 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= bounce S 00000000 0 4304 3365 f7a6e920 00200086 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 c0461d00 c0461d00 c0461d00 00000000 f7a6ea74 c3734d00 f7856d00 c04b9e00 00200282 c04b9e00 00200282 00200282 efd63f40 c04b9e00 c0128a5a 00000000 Call Trace: [] __mod_timer+0x94/0xa3 [] schedule_timeout+0x44/0xa1 [] process_timeout+0x0/0x5 [] sys_epoll_wait+0x15b/0x500 [] default_wake_function+0x0/0x8 [] sysenter_do_call+0x12/0x35 ======================= SysRq : Show Blocked State task PC stack pid father SysRq : Show Memory Mem-Info: DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 Normal per-cpu: CPU 0: hi: 186, btch: 31 usd: 99 CPU 1: hi: 186, btch: 31 usd: 69 HighMem per-cpu: CPU 0: hi: 186, btch: 31 usd: 80 CPU 1: hi: 186, btch: 31 usd: 40 Active:36919 inactive:29448 dirty:405 writeback:0 unstable:0 free:691569 slab:12398 mapped:8688 pagetables:375 bounce:2 DMA free:8804kB min:1140kB low:1424kB high:1708kB active:0kB inactive:0kB present:15788kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 868 3015 3015 Normal free:756732kB min:64392kB low:80488kB high:96588kB active:7780kB inactive:36908kB present:889680kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 17174 17174 HighMem free:2000740kB min:512kB low:40288kB high:80064kB active:139896kB inactive:80884kB present:2198284kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 DMA: 3*4kB 3*8kB 2*16kB 3*32kB 3*64kB 4*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8804kB Normal: 148*4kB 439*8kB 214*16kB 129*32kB 45*64kB 18*128kB 12*256kB 7*512kB 8*1024kB 8*2048kB 173*4096kB = 756680kB HighMem: 1*4kB 0*8kB 2*16kB 2*32kB 2*64kB 5*128kB 2*256kB 1*512kB 0*1024kB 2*2048kB 487*4096kB = 2000740kB 39327 total pagecache pages 0 pages in swap cache Swap cache stats: add 0, delete 0, find 0/0 Free swap = 979832kB Total swap = 979832kB 786176 pages RAM 556800 pages HighMem 12334 pages reserved 35711 pages shared 58628 pages non-shared SysRq : Emergency Sync SysRq : Emergency Sync SysRq : Emergency Sync SysRq : Resetting ============================================= [ INFO: possible recursive locking detected ] 2.6.27.21 #6 --------------------------------------------- dlm_controld/3528 is trying to acquire lock: (&sb->s_type->i_mutex_key#8/2){--..}, at: [] configfs_attach_group+0x43/0x18f [configfs] but task is already holding lock: (&sb->s_type->i_mutex_key#8/2){--..}, at: [] configfs_attach_group+0x43/0x18f [configfs] other info that might help us debug this: 2 locks held by dlm_controld/3528: #0: (&sb->s_type->i_mutex_key#7/1){--..}, at: [] lookup_create+0x1b/0x85 #1: (&sb->s_type->i_mutex_key#8/2){--..}, at: [] configfs_attach_group+0x43/0x18f [configfs] stack backtrace: Pid: 3528, comm: dlm_controld Not tainted 2.6.27.21 #6 [] validate_chain+0x9bd/0xf35 [] __lock_acquire+0x27b/0x932 [] lock_acquire+0x5f/0x77 [] configfs_attach_group+0x43/0x18f [configfs] [] mutex_lock_nested+0xa1/0x25c [] configfs_attach_group+0x43/0x18f [configfs] [] configfs_attach_group+0x43/0x18f [configfs] [] configfs_attach_group+0x43/0x18f [configfs] [] configfs_attach_group+0xe5/0x18f [configfs] [] configfs_mkdir+0x1fb/0x36f [configfs] [] vfs_mkdir+0x9c/0xdf [] _spin_unlock+0x14/0x1c [] sys_mkdirat+0xc5/0xd2 [] trace_hardirqs_on_thunk+0xc/0x10 [] sys_mkdir+0x1f/0x23 [] sysenter_do_call+0x12/0x35 ======================= aoe: e1.0: setting 8704 byte data frames on eth1:0030486543a9 aoe: e2.0: setting 8704 byte data frames on eth1:003048654382 aoe: e2.0: setting 8704 byte data frames on eth1:003048654383 aoe: 0030486543a9 e1.0 vace0 has 2930382333 sectors aoe: 003048654382 e2.0 vace0 has 2930382333 sectors etherd/e1.0: unknown partition table etherd/e2.0: unknown partition table dlm: Using TCP for communications dlm: got connection from 1 dlm: got connection from 5 dlm: connecting to 2 dlm: got connection from 4 dlm: got connection from 2 Trying to join cluster "lock_dlm", "kfki:home" Joined cluster. Now mounting FS... GFS: fsid=kfki:home.3: jid=3: Trying to acquire journal lock... GFS: fsid=kfki:home.3: jid=3: Looking at journal... GFS: fsid=kfki:home.3: jid=3: Done GFS: fsid=kfki:home.3: Scanning for log elements... GFS: fsid=kfki:home.3: Found 0 unlinked inodes GFS: fsid=kfki:home.3: Found quota changes for 0 IDs GFS: fsid=kfki:home.3: Done Trying to join cluster "lock_dlm", "kfki:services" Joined cluster. Now mounting FS... GFS: fsid=kfki:services.3: jid=3: Trying to acquire journal lock... GFS: fsid=kfki:services.3: jid=3: Looking at journal... GFS: fsid=kfki:services.3: jid=3: Done GFS: fsid=kfki:services.3: Scanning for log elements... GFS: fsid=kfki:services.3: Found 46 unlinked inodes GFS: fsid=kfki:services.3: Found quota changes for 3 IDs GFS: fsid=kfki:services.3: Done GFS: fsid=kfki:home.3: fast statfs start time = 1238198245 GFS: fsid=kfki:services.3: fast statfs start time = 1238198246