Intel driver (bug in the i40e)

Tags

Intel Corporation is an American multinational corporation and technology company headquartered in Santa Clara, California, in Silicon Valley.

A few months ago, after finally having received the new disks for our new Ceph cluster, we decided to benchmark them, so we could tune our cluster to get the best performance out of it, but that’s a story for another time. After setting everything up for our first benchmark, and starting it, we knew it would take about an hour and a half before completion, so in the meantime, we moved on to other things. Coming back to the test after about two hours, it still hadn’t finished, and it was stuck somewhere at the beginning.

During our preparation for it, we had run into this issue, and we thought it was just the tool that was mis-configured. However, after running a simple ceph status, we saw that some Ceph services (in this particular case, a monitor, a manager and two OSDs) were down. There was no joy in SSHing to the node (hereafter called node-1) that was running those services as it wasn’t answering anymore. We had to use the machine’s iDRAC (a remote control interface embedded into the server) to find a kernel panic on its console and force-reboot the machine.

To better understand why the kernel panicked, we started to panic with it, thinking there might be a huge problem with our brand new servers. Looking at the call trace from the kernel didn’t give us a lot of information, except that it could be memory-related. After a successful reboot, we went to the kernel logs from the previous boot to see what had happened. The first stack trace we were greeted with was this one (excluding some lines to keep it short):

kernel: Call Trace:

kernel:  __check_heap_object+0xe7/0x110

kernel:  __check_object_size.part.0+0x128/0x150

kernel:  __check_object_size+0x1c/0x1e

kernel:  simple_copy_to_iter+0x2b/0x50

kernel:  __skb_datagram_iter+0x19f/0x2e0

kernel:  ? zerocopy_sg_from_iter+0x50/0x50

kernel:  skb_copy_datagram_iter+0x40/0x90

kernel:  tcp_recvmsg+0x6a4/0xa10

kernel:  ? _cond_resched+0x1a/0x50

kernel:  ? aa_sk_perm+0x43/0x1b0

kernel:  inet_recvmsg+0x5e/0x100

kernel:  sock_recvmsg+0x69/0x80

kernel:  __sys_recvfrom+0xe5/0x1d0

kernel:  ? _cond_resched+0x1a/0x50

kernel:  ? ktime_get_ts64+0x4c/0xf0

kernel:  ? __prepare_exit_to_usermode+0x62/0xe0

kernel:  __x64_sys_recvfrom+0x29/0x30

kernel:  do_syscall_64+0x49/0xc0

kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xa9

kernel: RIP: 0033:0x7fb86243cddc

Full Version


Intel supplies microprocessors for computer system manufacturers such as Lenovo, HP, and Dell.