23-05-2022 21:43:05:576 [pid=16891 tid=16895] ERROR 0:1475 IOCTL failed io-type 1 ret -5 expected 32768 gpu_page_offset 0 23-05-2022 21:43:05:576 [pid=16891 tid=16895] ERROR 0:877 write failed at file_offset 102400 cur_size 32768 retval -5 23-05-2022 21:43:05:576 [pid=16891 tid=16895] ERROR cufio:1951 could not perform unaligned writes for fd: 51 23-05-2022 21:43:05:607 [pid=16891 tid=16893] ERROR 0:1475 IOCTL failed io-type 1 ret -5 expected 36864 gpu_page_offset 0 23-05-2022 21:43:05:608 [pid=16891 tid=16893] ERROR 0:877 write failed at file_offset 253952 cur_size 36864 retval -5 23-05-2022 21:43:05:608 [pid=16891 tid=16893] ERROR cufio:1951 could not perform unaligned writes for fd: 49 23-05-2022 21:43:05:616 [pid=16891 tid=16896] ERROR 0:1475 IOCTL failed io-type 1 ret -5 expected 36864 gpu_page_offset 0 23-05-2022 21:43:05:616 [pid=16891 tid=16896] ERROR 0:877 write failed at file_offset 294912 cur_size 36864 retval -5 23-05-2022 21:43:05:616 [pid=16891 tid=16896] ERROR cufio:1951 could not perform unaligned writes for fd: 52 23-05-2022 21:43:05:636 [pid=16891 tid=16900] ERROR 0:1475 IOCTL failed io-type 1 ret -5 expected 40960 gpu_page_offset 0 23-05-2022 21:43:05:636 [pid=16891 tid=16900] ERROR 0:877 write failed at file_offset 462848 cur_size 40960 retval -5 23-05-2022 21:43:05:636 [pid=16891 tid=16900] ERROR cufio:1951 could not perform unaligned writes for fd: 56 23-05-2022 21:43:05:640 [pid=16891 tid=16898] ERROR 0:1475 IOCTL failed io-type 1 ret -5 expected 40960 gpu_page_offset 0 23-05-2022 21:43:05:640 [pid=16891 tid=16898] ERROR 0:877 write failed at file_offset 417792 cur_size 40960 retval -5 23-05-2022 21:43:05:640 [pid=16891 tid=16898] ERROR cufio:1951 could not perform unaligned writes for fd: 54 23-05-2022 21:43:05:723 [pid=16891 tid=16899] ERROR 0:1475 IOCTL failed io-type 1 ret -5 expected 53248 gpu_page_offset 0 23-05-2022 21:43:05:724 [pid=16891 tid=16899] ERROR 0:877 write failed at file_offset 1073152 cur_size 53248 retval -5 23-05-2022 21:43:05:724 [pid=16891 tid=16899] ERROR cufio:1951 could not perform unaligned writes for fd: 55 23-05-2022 21:57:07:353 [pid=17324 tid=17324] ERROR 0:258 Batch Ctx state transition failed in cuFileBatchGetContextFromID 23-05-2022 21:57:07:353 [pid=17324 tid=17324] ERROR cufio_batch:358 Error in submit GetContext for the batch_id 0xfedf00000001 23-05-2022 21:57:07:353 [pid=17324 tid=17324] ERROR 0:263 Batch Ctx version mismatch in cuFileBatchGetContextFromID 23-05-2022 21:57:07:353 [pid=17324 tid=17324] ERROR cufio_batch:418 Error in cuFileBatchGetContextFromID batch handle: 0xfedf00000001 23-05-2022 21:57:07:353 [pid=17324 tid=17324] ERROR 0:258 Batch Ctx state transition failed in cuFileBatchGetContextFromID 23-05-2022 21:57:17:968 [pid=17329 tid=17329] ERROR cuf-aio:463 Cancelling IOs using get status: 0 23-05-2022 21:57:19:904 [pid=17329 tid=17329] ERROR cuf-aio:463 Cancelling IOs using get status: 0 23-05-2022 21:57:19:905 [pid=17329 tid=17329] ERROR 0:83 ERROR Cancelling Batch IOs, err: 64 23-05-2022 21:57:19:905 [pid=17329 tid=17329] ERROR 0:258 Batch Ctx state transition failed in cuFileBatchGetContextFromID 23-05-2022 21:57:19:905 [pid=17329 tid=17329] ERROR cufio_batch:597 Batch state is not in IDLE state, try destroy later 23-05-2022 21:57:23:924 [pid=17329 tid=17329] ERROR cufio_batch:405 Minimum number of IOs requested: 65 greater than maximum reqsted: 64 batch_handle 0x1fedf00000004 23-05-2022 21:57:23:924 [pid=17329 tid=17329] ERROR cuf-aio:463 Cancelling IOs using get status: 0 23-05-2022 21:57:26:16 [pid=17329 tid=17329] ERROR cuf-aio:463 Cancelling IOs using get status: 0 23-05-2022 21:57:28:131 [pid=17329 tid=17329] ERROR 0:258 Batch Ctx state transition failed in cuFileBatchGetContextFromID 23-05-2022 21:57:30:61 [pid=17329 tid=17329] ERROR cufio_batch:411 Max nr 0, batch handle: 0x1fedf00000007 23-05-2022 21:57:34:149 [pid=17329 tid=17329] ERROR cufio_batch:405 Minimum number of IOs requested: 64 greater than maximum reqsted: 63 batch_handle 0x1fedf00000009 23-05-2022 21:57:34:149 [pid=17329 tid=17329] ERROR cuf-aio:463 Cancelling IOs using get status: 0 23-05-2022 21:57:34:150 [pid=17329 tid=17329] ERROR 0:83 ERROR Cancelling Batch IOs, err: 64 23-05-2022 21:57:34:150 [pid=17329 tid=17329] ERROR 0:258 Batch Ctx state transition failed in cuFileBatchGetContextFromID 23-05-2022 21:57:34:150 [pid=17329 tid=17329] ERROR cufio_batch:597 Batch state is not in IDLE state, try destroy later 23-05-2022 21:57:38:169 [pid=17329 tid=17329] ERROR 0:258 Batch Ctx state transition failed in cuFileBatchGetContextFromID 23-05-2022 21:57:38:169 [pid=17329 tid=17329] ERROR cufio_batch:597 Batch state is not in IDLE state, try destroy later 23-05-2022 21:57:44:541 [pid=17333 tid=17333] ERROR cuf-aio:463 Cancelling IOs using get status: 0 23-05-2022 21:57:50:576 [pid=17336 tid=17336] ERROR cufio_batch:597 Batch state is not in IDLE state, try destroy later 23-05-2022 21:57:50:576 [pid=17336 tid=17336] ERROR cufio_batch:597 Batch state is not in IDLE state, try destroy later 23-05-2022 21:57:50:763 [pid=17336 tid=17336] ERROR 0:258 Batch Ctx state transition failed in cuFileBatchGetContextFromID 23-05-2022 21:57:50:763 [pid=17336 tid=17336] ERROR cufio_batch:597 Batch state is not in IDLE state, try destroy later 23-05-2022 21:57:59:700 [pid=17341 tid=17341] ERROR cufio:757 cuFile buffer check range error, invalid dev ptr size psize= 1048576 length= 1048577 23-05-2022 21:57:59:700 [pid=17341 tid=17341] ERROR cufio:918 cuFileBufRegister checks failed: -5014 23-05-2022 21:57:59:700 [pid=17341 tid=17341] ERROR cufio:1173 cuFile buffer object device memory check failed 23-05-2022 21:57:59:700 [pid=17341 tid=17341] ERROR cufio:1236 cuFileBufRegister error pointer range exceeds allocated address range 23-05-2022 21:58:08:538 [pid=17347 tid=17347] ERROR cufio:825 cuFileBufRegister error, detected invalid device pointer CUDA_ERROR_INVALID_VALUE 23-05-2022 21:58:08:538 [pid=17347 tid=17347] ERROR cufio:918 cuFileBufRegister checks failed: -5012 23-05-2022 21:58:08:538 [pid=17347 tid=17347] ERROR cufio:1173 cuFile buffer object device memory check failed 23-05-2022 21:58:08:538 [pid=17347 tid=17347] ERROR cufio:1236 cuFileBufRegister error invalid device pointer 23-05-2022 21:58:12:742 [pid=17350 tid=17350] ERROR cufio:1161 cuFileBufRegister error, invalid api parameters 23-05-2022 21:58:12:742 [pid=17350 tid=17350] ERROR cufio:1236 cuFileBufRegister error invalid arguments 23-05-2022 21:58:17:158 [pid=17352 tid=17352] ERROR cufio:851 cuFileBufRegister error, cudaMallocManaged memory. not supported by nvidia_p2p_pages 23-05-2022 21:58:17:158 [pid=17352 tid=17352] ERROR cufio:918 cuFileBufRegister checks failed: -5013 23-05-2022 21:58:17:158 [pid=17352 tid=17352] ERROR cufio:1173 cuFile buffer object device memory check failed 23-05-2022 21:58:17:158 [pid=17352 tid=17352] ERROR cufio:1236 cuFileBufRegister error invalid pointer memory type 23-05-2022 21:58:21:540 [pid=17355 tid=17355] ERROR cufio:834 cuFileBufRegister error, invalid memory type 1 23-05-2022 21:58:21:541 [pid=17355 tid=17355] ERROR cufio:918 cuFileBufRegister checks failed: -5013 23-05-2022 21:58:21:541 [pid=17355 tid=17355] ERROR cufio:1173 cuFile buffer object device memory check failed 23-05-2022 21:58:21:541 [pid=17355 tid=17355] ERROR cufio:1236 cuFileBufRegister error invalid pointer memory type 23-05-2022 21:58:25:905 [pid=17358 tid=17358] ERROR cufio:825 cuFileBufRegister error, detected invalid device pointer CUDA_ERROR_INVALID_VALUE 23-05-2022 21:58:25:905 [pid=17358 tid=17358] ERROR cufio:918 cuFileBufRegister checks failed: -5012 23-05-2022 21:58:25:905 [pid=17358 tid=17358] ERROR cufio:1173 cuFile buffer object device memory check failed 23-05-2022 21:58:25:905 [pid=17358 tid=17358] ERROR cufio:1236 cuFileBufRegister error invalid device pointer 23-05-2022 21:58:30:284 [pid=17361 tid=17361] ERROR cufio:834 cuFileBufRegister error, invalid memory type 1 23-05-2022 21:58:30:284 [pid=17361 tid=17361] ERROR cufio:918 cuFileBufRegister checks failed: -5013 23-05-2022 21:58:30:284 [pid=17361 tid=17361] ERROR cufio:1173 cuFile buffer object device memory check failed 23-05-2022 21:58:30:284 [pid=17361 tid=17361] ERROR cufio:1236 cuFileBufRegister error invalid pointer memory type 23-05-2022 21:58:34:762 [pid=17364 tid=17364] ERROR cufio:834 cuFileBufRegister error, invalid memory type 1 23-05-2022 21:58:34:763 [pid=17364 tid=17364] ERROR cufio:918 cuFileBufRegister checks failed: -5013 23-05-2022 21:58:34:763 [pid=17364 tid=17364] ERROR cufio:1173 cuFile buffer object device memory check failed 23-05-2022 21:58:34:763 [pid=17364 tid=17364] ERROR cufio:1236 cuFileBufRegister error invalid pointer memory type 23-05-2022 21:58:39:221 [pid=17367 tid=17367] ERROR cufio:851 cuFileBufRegister error, cudaMallocManaged memory. not supported by nvidia_p2p_pages 23-05-2022 21:58:39:221 [pid=17367 tid=17367] ERROR cufio:918 cuFileBufRegister checks failed: -5013 23-05-2022 21:58:39:221 [pid=17367 tid=17367] ERROR cufio:1173 cuFile buffer object device memory check failed 23-05-2022 21:58:39:221 [pid=17367 tid=17367] ERROR cufio:1236 cuFileBufRegister error invalid pointer memory type 23-05-2022 21:58:48:159 [pid=17373 tid=17373] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 21:58:48:160 [pid=17373 tid=17373] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 21:59:01:414 [pid=17382 tid=17382] ERROR cufio:1167 cuFileBufRegister error, device memory already registered 23-05-2022 21:59:01:414 [pid=17382 tid=17382] ERROR cufio:1236 cuFileBufRegister error device pointer already registered 23-05-2022 21:59:05:635 [pid=17385 tid=17385] ERROR cufio:1258 cuFileBufDeregister error, invalid device pointer to deregister 23-05-2022 21:59:05:635 [pid=17385 tid=17385] ERROR cufio:1323 cuFileBufDeregister error: invalid arguments 23-05-2022 21:59:18:547 [pid=17391 tid=17391] ERROR cufio-obj:54 cuFile error fetching fd, invalid CUfileHandle 23-05-2022 21:59:18:547 [pid=17391 tid=17391] ERROR cufio:1109 cuFile invalid file handle to destroy 23-05-2022 21:59:39:900 [pid=17402 tid=17402] ERROR cufio:967 cuFileHandleRegister error, win32 file handle type not supported 23-05-2022 21:59:39:900 [pid=17402 tid=17402] ERROR cufio:1082 cuFileHandleRegister error: GPUDirect Storage not supported on current file 23-05-2022 21:59:44:153 [pid=17404 tid=17404] ERROR cufio:1003 cuFileHandleRegister error, unsupported file type 23-05-2022 21:59:44:153 [pid=17404 tid=17404] ERROR cufio:1082 cuFileHandleRegister error: unsupported file type 23-05-2022 21:59:48:393 [pid=17406 tid=17406] ERROR cufio:1018 cuFileHandleRegister error, fd opened with unsupported flags: 33793 23-05-2022 21:59:48:393 [pid=17406 tid=17406] ERROR cufio:1082 cuFileHandleRegister error: unsupported file open flags 23-05-2022 21:59:52:642 [pid=17408 tid=17408] ERROR cufio:973 cuFileHandleRegister error, invalid file descriptor: -1 23-05-2022 21:59:52:642 [pid=17408 tid=17408] ERROR cufio:1082 cuFileHandleRegister error: invalid arguments 23-05-2022 21:59:56:898 [pid=17410 tid=17410] ERROR cufio:84 fstat failed for fd 65536 Bad file descriptor 23-05-2022 21:59:56:898 [pid=17410 tid=17410] ERROR cufio:991 cuFileHandleRegister error, invalid file descriptor passed -9 23-05-2022 21:59:56:898 [pid=17410 tid=17410] ERROR cufio:1082 cuFileHandleRegister error: invalid arguments 23-05-2022 22:00:32:676 [pid=17432 tid=17432] ERROR cufio:1018 cuFileHandleRegister error, fd opened with unsupported flags: 33792 23-05-2022 22:00:32:676 [pid=17432 tid=17432] ERROR cufio:1082 cuFileHandleRegister error: unsupported file open flags 23-05-2022 22:00:36:949 [pid=17434 tid=17434] ERROR cufio:1018 cuFileHandleRegister error, fd opened with unsupported flags: 34816 23-05-2022 22:00:36:949 [pid=17434 tid=17434] ERROR cufio:1082 cuFileHandleRegister error: unsupported file open flags 23-05-2022 22:00:45:469 [pid=17438 tid=17438] ERROR cufio:1025 cuFileHandleRegister file, O_DIRECT flag not set on fd 23-05-2022 22:00:45:470 [pid=17438 tid=17438] ERROR cufio:1082 cuFileHandleRegister error: fd direct IO not set 23-05-2022 22:00:49:759 [pid=17440 tid=17440] ERROR cufio:1018 cuFileHandleRegister error, fd opened with unsupported flags: 163840 23-05-2022 22:00:49:759 [pid=17440 tid=17440] ERROR cufio:1082 cuFileHandleRegister error: unsupported file open flags 23-05-2022 22:00:54:4 [pid=17442 tid=17442] ERROR cufio:1025 cuFileHandleRegister file, O_DIRECT flag not set on fd 23-05-2022 22:00:54:4 [pid=17442 tid=17442] ERROR cufio:1082 cuFileHandleRegister error: fd direct IO not set 23-05-2022 22:01:11:110 [pid=17450 tid=17450] ERROR 0:256 invalid bar size usage specified, (KB) 8 min: 64 max: 18014398509481980 23-05-2022 22:01:11:110 [pid=17450 tid=17450] ERROR cufio:2546 cuFileDriverSetMaxPinnedMemSize error: property range error 23-05-2022 22:01:15:412 [pid=17452 tid=17452] ERROR 0:177 invalid directIO size (KB) specified: 8 min: 64 max: 16384 23-05-2022 22:01:15:412 [pid=17452 tid=17452] ERROR cufio:2497 cuFileDriverSetMaxDirectIOSize error: property range error 23-05-2022 22:01:34:311 [pid=17460 tid=17474] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:336 [pid=17460 tid=17496] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:336 [pid=17460 tid=17496] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:360 [pid=17460 tid=17470] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:360 [pid=17460 tid=17470] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:396 [pid=17460 tid=17504] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:396 [pid=17460 tid=17504] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:420 [pid=17460 tid=17464] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:420 [pid=17460 tid=17464] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:444 [pid=17460 tid=17518] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:444 [pid=17460 tid=17518] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:467 [pid=17460 tid=17500] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:468 [pid=17460 tid=17500] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:503 [pid=17460 tid=17488] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:504 [pid=17460 tid=17488] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:528 [pid=17460 tid=17506] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:528 [pid=17460 tid=17506] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:563 [pid=17460 tid=17468] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:564 [pid=17460 tid=17468] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:596 [pid=17460 tid=17462] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:596 [pid=17460 tid=17462] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:631 [pid=17460 tid=17510] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:632 [pid=17460 tid=17510] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:655 [pid=17460 tid=17486] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:656 [pid=17460 tid=17486] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:679 [pid=17460 tid=17516] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:680 [pid=17460 tid=17516] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:716 [pid=17460 tid=17502] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:716 [pid=17460 tid=17502] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:752 [pid=17460 tid=17520] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:752 [pid=17460 tid=17520] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:943 [pid=17460 tid=17484] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:944 [pid=17460 tid=17484] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:980 [pid=17460 tid=17508] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:34:980 [pid=17460 tid=17508] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:3 [pid=17460 tid=17476] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:4 [pid=17460 tid=17476] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:27 [pid=17460 tid=17478] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:28 [pid=17460 tid=17478] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:52 [pid=17460 tid=17492] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:52 [pid=17460 tid=17492] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:76 [pid=17460 tid=17490] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:76 [pid=17460 tid=17490] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:112 [pid=17460 tid=17524] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:112 [pid=17460 tid=17524] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:136 [pid=17460 tid=17522] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:136 [pid=17460 tid=17522] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:160 [pid=17460 tid=17480] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:160 [pid=17460 tid=17480] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:195 [pid=17460 tid=17512] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:196 [pid=17460 tid=17512] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:220 [pid=17460 tid=17466] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:220 [pid=17460 tid=17466] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:255 [pid=17460 tid=17498] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:280 [pid=17460 tid=17494] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:280 [pid=17460 tid=17494] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:35:280 [pid=17460 tid=17494] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:39:168 [pid=17460 tid=17472] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:39:169 [pid=17460 tid=17472] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:39:169 [pid=17460 tid=17472] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:39:170 [pid=17460 tid=17472] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:39:170 [pid=17460 tid=17472] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:39:170 [pid=17460 tid=17472] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:39:170 [pid=17460 tid=17472] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:39:170 [pid=17460 tid=17472] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:39:170 [pid=17460 tid=17472] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:43:885 [pid=17531 tid=17535] ERROR cufio:1210 cuFileBufRegister error Memory already registered 23-05-2022 22:01:43:885 [pid=17531 tid=17579] ERROR cufio:1210 cuFileBufRegister error Memory already registered 23-05-2022 22:01:43:885 [pid=17531 tid=17549] ERROR cufio:1210 cuFileBufRegister error Memory already registered 23-05-2022 22:01:43:886 [pid=17531 tid=17577] ERROR cufio:1210 cuFileBufRegister error Memory already registered 23-05-2022 22:01:43:886 [pid=17531 tid=17587] ERROR cufio:1210 cuFileBufRegister error Memory already registered 23-05-2022 22:01:43:886 [pid=17531 tid=17537] ERROR cufio:1210 cuFileBufRegister error Memory already registered 23-05-2022 22:01:43:886 [pid=17531 tid=17559] ERROR cufio:1210 cuFileBufRegister error Memory already registered 23-05-2022 22:01:43:886 [pid=17531 tid=17575] ERROR cufio:1210 cuFileBufRegister error Memory already registered 23-05-2022 22:01:43:886 [pid=17531 tid=17594] ERROR 0:909 Failed to obtain bounce buffer from domain: 1 GPU: 3 23-05-2022 22:01:44:55 [pid=17531 tid=17594] ERROR 0:1206 failed to get bounce buffer for PCI group 1 GPU 3 23-05-2022 22:01:44:55 [pid=17531 tid=17564] ERROR 0:909 Failed to obtain bounce buffer from domain: 1 GPU: 3 23-05-2022 22:01:44:56 [pid=17531 tid=17564] ERROR 0:1206 failed to get bounce buffer for PCI group 1 GPU 3 23-05-2022 22:01:44:79 [pid=17531 tid=17549] ERROR cufio:1236 cuFileBufRegister error device pointer already registered 23-05-2022 22:01:44:82 [pid=17531 tid=17549] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:82 [pid=17531 tid=17549] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:82 [pid=17531 tid=17549] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:115 [pid=17531 tid=17587] ERROR cufio:1236 cuFileBufRegister error device pointer already registered 23-05-2022 22:01:44:118 [pid=17531 tid=17587] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:118 [pid=17531 tid=17587] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:118 [pid=17531 tid=17587] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:151 [pid=17531 tid=17577] ERROR cufio:1236 cuFileBufRegister error device pointer already registered 23-05-2022 22:01:44:154 [pid=17531 tid=17577] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:154 [pid=17531 tid=17577] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:154 [pid=17531 tid=17577] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:187 [pid=17531 tid=17579] ERROR cufio:1236 cuFileBufRegister error device pointer already registered 23-05-2022 22:01:44:190 [pid=17531 tid=17579] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:235 [pid=17531 tid=17537] ERROR cufio:1236 cuFileBufRegister error device pointer already registered 23-05-2022 22:01:44:238 [pid=17531 tid=17537] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:238 [pid=17531 tid=17537] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:259 [pid=17531 tid=17559] ERROR cufio:1236 cuFileBufRegister error device pointer already registered 23-05-2022 22:01:44:262 [pid=17531 tid=17559] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:262 [pid=17531 tid=17559] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:262 [pid=17531 tid=17559] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:295 [pid=17531 tid=17535] ERROR cufio:1236 cuFileBufRegister error device pointer already registered 23-05-2022 22:01:44:298 [pid=17531 tid=17535] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:298 [pid=17531 tid=17535] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:298 [pid=17531 tid=17535] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:323 [pid=17531 tid=17575] ERROR cufio:1236 cuFileBufRegister error device pointer already registered 23-05-2022 22:01:44:325 [pid=17531 tid=17557] ERROR cufio:1210 cuFileBufRegister error Memory already registered 23-05-2022 22:01:44:325 [pid=17531 tid=17551] ERROR cufio:1210 cuFileBufRegister error Memory already registered 23-05-2022 22:01:44:326 [pid=17531 tid=17575] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:403 [pid=17531 tid=17557] ERROR cufio:1236 cuFileBufRegister error device pointer already registered 23-05-2022 22:01:44:406 [pid=17531 tid=17557] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:406 [pid=17531 tid=17557] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:406 [pid=17531 tid=17557] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:428 [pid=17531 tid=17551] ERROR cufio:1236 cuFileBufRegister error device pointer already registered 23-05-2022 22:01:44:430 [pid=17531 tid=17551] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:44:430 [pid=17531 tid=17551] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:49:412 [pid=17531 tid=17566] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:412 [pid=17531 tid=17536] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:412 [pid=17531 tid=17550] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:412 [pid=17531 tid=17556] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:412 [pid=17531 tid=17580] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17542] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17558] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17568] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17578] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17592] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17590] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17534] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17546] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17533] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17563] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17576] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17572] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17570] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17544] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17540] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17596] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17582] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17552] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17554] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17538] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17584] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17569] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17585] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17541] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17593] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17567] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17574] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17573] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17561] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17588] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17553] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17581] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17583] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17548] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:413 [pid=17531 tid=17555] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:414 [pid=17531 tid=17589] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:414 [pid=17531 tid=17547] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:414 [pid=17531 tid=17533] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:414 [pid=17531 tid=17533] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:414 [pid=17531 tid=17545] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:414 [pid=17531 tid=17563] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:414 [pid=17531 tid=17563] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:414 [pid=17531 tid=17569] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:414 [pid=17531 tid=17569] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:414 [pid=17531 tid=17585] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:414 [pid=17531 tid=17585] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:414 [pid=17531 tid=17541] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:414 [pid=17531 tid=17541] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:414 [pid=17531 tid=17593] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:414 [pid=17531 tid=17593] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:414 [pid=17531 tid=17567] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:414 [pid=17531 tid=17567] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:414 [pid=17531 tid=17573] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:414 [pid=17531 tid=17573] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:414 [pid=17531 tid=17561] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:414 [pid=17531 tid=17561] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:414 [pid=17531 tid=17553] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:415 [pid=17531 tid=17553] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:415 [pid=17531 tid=17581] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:415 [pid=17531 tid=17581] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:415 [pid=17531 tid=17583] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:415 [pid=17531 tid=17583] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:415 [pid=17531 tid=17555] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:415 [pid=17531 tid=17555] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:415 [pid=17531 tid=17589] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:415 [pid=17531 tid=17589] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:415 [pid=17531 tid=17547] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:415 [pid=17531 tid=17547] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:415 [pid=17531 tid=17545] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:415 [pid=17531 tid=17545] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:415 [pid=17531 tid=17585] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:415 [pid=17531 tid=17585] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:415 [pid=17531 tid=17539] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:416 [pid=17531 tid=17589] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:416 [pid=17531 tid=17589] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:416 [pid=17531 tid=17539] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:416 [pid=17531 tid=17539] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:417 [pid=17531 tid=17533] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:417 [pid=17531 tid=17533] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:417 [pid=17531 tid=17589] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:417 [pid=17531 tid=17589] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:417 [pid=17531 tid=17541] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:417 [pid=17531 tid=17541] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:418 [pid=17531 tid=17539] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:418 [pid=17531 tid=17539] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:418 [pid=17531 tid=17569] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:418 [pid=17531 tid=17569] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:419 [pid=17531 tid=17569] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:419 [pid=17531 tid=17569] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:419 [pid=17531 tid=17565] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:419 [pid=17531 tid=17539] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:419 [pid=17531 tid=17539] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:419 [pid=17531 tid=17547] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:419 [pid=17531 tid=17547] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:419 [pid=17531 tid=17561] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:419 [pid=17531 tid=17561] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:420 [pid=17531 tid=17565] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:420 [pid=17531 tid=17565] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:420 [pid=17531 tid=17565] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:420 [pid=17531 tid=17565] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:421 [pid=17531 tid=17571] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:421 [pid=17531 tid=17561] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:421 [pid=17531 tid=17561] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:421 [pid=17531 tid=17563] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:421 [pid=17531 tid=17563] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:421 [pid=17531 tid=17533] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:421 [pid=17531 tid=17533] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:421 [pid=17531 tid=17589] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:421 [pid=17531 tid=17589] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:421 [pid=17531 tid=17569] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:421 [pid=17531 tid=17569] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:422 [pid=17531 tid=17571] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:422 [pid=17531 tid=17571] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:422 [pid=17531 tid=17581] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:422 [pid=17531 tid=17581] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:422 [pid=17531 tid=17593] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:422 [pid=17531 tid=17593] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:422 [pid=17531 tid=17561] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:422 [pid=17531 tid=17561] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:422 [pid=17531 tid=17543] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:423 [pid=17531 tid=17541] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:423 [pid=17531 tid=17541] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:423 [pid=17531 tid=17581] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:423 [pid=17531 tid=17581] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:423 [pid=17531 tid=17555] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:423 [pid=17531 tid=17555] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:423 [pid=17531 tid=17593] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:423 [pid=17531 tid=17593] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:423 [pid=17531 tid=17571] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:423 [pid=17531 tid=17571] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:423 [pid=17531 tid=17543] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:423 [pid=17531 tid=17543] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:423 [pid=17531 tid=17583] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:423 [pid=17531 tid=17583] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:423 [pid=17531 tid=17541] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:423 [pid=17531 tid=17541] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:423 [pid=17531 tid=17583] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:423 [pid=17531 tid=17583] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:423 [pid=17531 tid=17585] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:423 [pid=17531 tid=17585] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:423 [pid=17531 tid=17581] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:423 [pid=17531 tid=17581] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:424 [pid=17531 tid=17573] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:424 [pid=17531 tid=17573] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:424 [pid=17531 tid=17553] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:424 [pid=17531 tid=17553] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:424 [pid=17531 tid=17585] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:424 [pid=17531 tid=17585] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:424 [pid=17531 tid=17555] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:424 [pid=17531 tid=17555] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:424 [pid=17531 tid=17545] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:424 [pid=17531 tid=17545] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:424 [pid=17531 tid=17573] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:424 [pid=17531 tid=17573] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:424 [pid=17531 tid=17567] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:424 [pid=17531 tid=17567] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:425 [pid=17531 tid=17573] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:425 [pid=17531 tid=17573] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:425 [pid=17531 tid=17593] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:425 [pid=17531 tid=17593] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:426 [pid=17531 tid=17553] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:426 [pid=17531 tid=17553] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:426 [pid=17531 tid=17555] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:426 [pid=17531 tid=17555] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:427 [pid=17531 tid=17545] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:427 [pid=17531 tid=17545] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:427 [pid=17531 tid=17563] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:427 [pid=17531 tid=17563] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:427 [pid=17531 tid=17533] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:427 [pid=17531 tid=17533] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:428 [pid=17531 tid=17583] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:428 [pid=17531 tid=17583] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:428 [pid=17531 tid=17563] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:428 [pid=17531 tid=17563] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:428 [pid=17531 tid=17553] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:428 [pid=17531 tid=17553] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:428 [pid=17531 tid=17567] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:428 [pid=17531 tid=17567] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:496 [pid=17531 tid=17567] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:496 [pid=17531 tid=17567] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:496 [pid=17531 tid=17547] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:496 [pid=17531 tid=17547] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:496 [pid=17531 tid=17547] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:497 [pid=17531 tid=17547] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:49:577 [pid=17531 tid=17595] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:01:49:578 [pid=17531 tid=17595] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:01:49:578 [pid=17531 tid=17595] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:01:56:221 [pid=17602 tid=17616] ERROR cufio:2253 driver shutdown in progress. 23-05-2022 22:01:56:221 [pid=17602 tid=17616] ERROR cufio:2285 cuFileRead invalid parameters driver shutdown in progress fd 60 offset 59768832 length 1048576 bufOff 0 devPtr 0x7fafe4f00000 23-05-2022 22:01:56:221 [pid=17602 tid=17612] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:221 [pid=17602 tid=17664] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:221 [pid=17602 tid=17618] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:221 [pid=17602 tid=17666] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:221 [pid=17602 tid=17644] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:222 [pid=17602 tid=17622] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:222 [pid=17602 tid=17622] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:222 [pid=17602 tid=17664] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:222 [pid=17602 tid=17616] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:222 [pid=17602 tid=17616] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:222 [pid=17602 tid=17616] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:222 [pid=17602 tid=17616] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:222 [pid=17602 tid=17666] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:222 [pid=17602 tid=17664] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:222 [pid=17602 tid=17644] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:223 [pid=17602 tid=17612] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:223 [pid=17602 tid=17644] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:223 [pid=17602 tid=17618] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:223 [pid=17602 tid=17644] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:223 [pid=17602 tid=17618] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:223 [pid=17602 tid=17618] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:223 [pid=17602 tid=17666] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:223 [pid=17602 tid=17664] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:223 [pid=17602 tid=17666] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:223 [pid=17602 tid=17612] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:224 [pid=17602 tid=17612] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:224 [pid=17602 tid=17622] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:224 [pid=17602 tid=17622] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:284 [pid=17602 tid=17608] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:284 [pid=17602 tid=17608] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:284 [pid=17602 tid=17608] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:332 [pid=17602 tid=17652] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:332 [pid=17602 tid=17652] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:01:56:332 [pid=17602 tid=17652] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:00:965 [pid=17602 tid=17647] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:965 [pid=17602 tid=17609] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:965 [pid=17602 tid=17651] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:965 [pid=17602 tid=17663] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:965 [pid=17602 tid=17624] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:965 [pid=17602 tid=17607] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:965 [pid=17602 tid=17642] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:965 [pid=17602 tid=17656] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:965 [pid=17602 tid=17645] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:965 [pid=17602 tid=17649] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:965 [pid=17602 tid=17665] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:965 [pid=17602 tid=17633] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:965 [pid=17602 tid=17659] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:965 [pid=17602 tid=17657] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:965 [pid=17602 tid=17614] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:965 [pid=17602 tid=17606] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:965 [pid=17602 tid=17628] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:966 [pid=17602 tid=17631] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:966 [pid=17602 tid=17620] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:966 [pid=17602 tid=17638] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:966 [pid=17602 tid=17610] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:966 [pid=17602 tid=17630] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:966 [pid=17602 tid=17626] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:966 [pid=17602 tid=17648] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:966 [pid=17602 tid=17643] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:966 [pid=17602 tid=17658] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:966 [pid=17602 tid=17660] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:966 [pid=17602 tid=17640] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:966 [pid=17602 tid=17627] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:966 [pid=17602 tid=17624] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:966 [pid=17602 tid=17624] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:966 [pid=17602 tid=17642] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:966 [pid=17602 tid=17642] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:966 [pid=17602 tid=17656] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:966 [pid=17602 tid=17656] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:966 [pid=17602 tid=17654] ERROR cufio:2172 cuFile error issuing IO, file descriptor is not registered 23-05-2022 22:02:00:967 [pid=17602 tid=17614] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:967 [pid=17602 tid=17614] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:967 [pid=17602 tid=17606] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:967 [pid=17602 tid=17606] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:967 [pid=17602 tid=17628] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:967 [pid=17602 tid=17628] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:967 [pid=17602 tid=17642] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:967 [pid=17602 tid=17642] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:967 [pid=17602 tid=17620] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:967 [pid=17602 tid=17620] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:967 [pid=17602 tid=17638] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:967 [pid=17602 tid=17638] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:967 [pid=17602 tid=17610] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:967 [pid=17602 tid=17610] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:967 [pid=17602 tid=17648] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:967 [pid=17602 tid=17648] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:967 [pid=17602 tid=17626] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:967 [pid=17602 tid=17626] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:967 [pid=17602 tid=17630] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:967 [pid=17602 tid=17630] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:967 [pid=17602 tid=17658] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:967 [pid=17602 tid=17658] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:967 [pid=17602 tid=17660] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:967 [pid=17602 tid=17660] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:967 [pid=17602 tid=17640] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:967 [pid=17602 tid=17640] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:967 [pid=17602 tid=17648] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:967 [pid=17602 tid=17648] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:967 [pid=17602 tid=17638] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:967 [pid=17602 tid=17638] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:967 [pid=17602 tid=17656] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:967 [pid=17602 tid=17656] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:968 [pid=17602 tid=17624] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:968 [pid=17602 tid=17624] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:968 [pid=17602 tid=17654] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:968 [pid=17602 tid=17654] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:968 [pid=17602 tid=17642] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:968 [pid=17602 tid=17642] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:968 [pid=17602 tid=17610] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:968 [pid=17602 tid=17610] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:968 [pid=17602 tid=17656] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:968 [pid=17602 tid=17656] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:968 [pid=17602 tid=17624] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:968 [pid=17602 tid=17624] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:968 [pid=17602 tid=17610] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:968 [pid=17602 tid=17610] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:968 [pid=17602 tid=17642] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:969 [pid=17602 tid=17642] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:969 [pid=17602 tid=17648] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:969 [pid=17602 tid=17648] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:969 [pid=17602 tid=17658] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:969 [pid=17602 tid=17654] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:969 [pid=17602 tid=17654] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:969 [pid=17602 tid=17658] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:969 [pid=17602 tid=17628] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:969 [pid=17602 tid=17628] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:969 [pid=17602 tid=17620] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:969 [pid=17602 tid=17620] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:969 [pid=17602 tid=17628] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:969 [pid=17602 tid=17628] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:969 [pid=17602 tid=17656] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:969 [pid=17602 tid=17656] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:969 [pid=17602 tid=17658] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:969 [pid=17602 tid=17658] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:969 [pid=17602 tid=17638] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:969 [pid=17602 tid=17638] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:969 [pid=17602 tid=17638] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:969 [pid=17602 tid=17638] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:969 [pid=17602 tid=17630] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:969 [pid=17602 tid=17630] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:969 [pid=17602 tid=17628] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:969 [pid=17602 tid=17628] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:970 [pid=17602 tid=17648] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:970 [pid=17602 tid=17648] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:970 [pid=17602 tid=17654] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:970 [pid=17602 tid=17654] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:970 [pid=17602 tid=17658] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:970 [pid=17602 tid=17658] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:971 [pid=17602 tid=17640] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:971 [pid=17602 tid=17640] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:971 [pid=17602 tid=17626] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:971 [pid=17602 tid=17626] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:971 [pid=17602 tid=17640] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:971 [pid=17602 tid=17640] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:972 [pid=17602 tid=17626] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:972 [pid=17602 tid=17626] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:972 [pid=17602 tid=17606] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:972 [pid=17602 tid=17606] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:972 [pid=17602 tid=17626] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:972 [pid=17602 tid=17626] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:972 [pid=17602 tid=17640] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:972 [pid=17602 tid=17640] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:973 [pid=17602 tid=17630] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:973 [pid=17602 tid=17630] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:973 [pid=17602 tid=17660] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:973 [pid=17602 tid=17660] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:974 [pid=17602 tid=17660] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:974 [pid=17602 tid=17660] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:974 [pid=17602 tid=17620] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:974 [pid=17602 tid=17620] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:00:975 [pid=17602 tid=17660] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:00:975 [pid=17602 tid=17660] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:01:4 [pid=17602 tid=17630] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:01:4 [pid=17602 tid=17630] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:01:4 [pid=17602 tid=17624] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:01:4 [pid=17602 tid=17624] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:01:4 [pid=17602 tid=17610] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:01:4 [pid=17602 tid=17610] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:01:4 [pid=17602 tid=17606] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:01:4 [pid=17602 tid=17606] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:01:4 [pid=17602 tid=17606] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:01:4 [pid=17602 tid=17606] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:01:5 [pid=17602 tid=17620] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:01:5 [pid=17602 tid=17620] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:01:5 [pid=17602 tid=17614] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:01:5 [pid=17602 tid=17614] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:01:5 [pid=17602 tid=17614] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:01:5 [pid=17602 tid=17614] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:01:5 [pid=17602 tid=17614] ERROR cufio:1275 cuFileBufDeregister error, object for device pointer is not registered 23-05-2022 22:02:01:5 [pid=17602 tid=17614] ERROR cufio:1323 cuFileBufDeregister error: device pointer lookup failure 23-05-2022 22:02:13:845 [pid=17711 tid=17711] ERROR cufio:2219 cuFile error issuing IO, buffer object is stale, user must deregister prior submitting new IO 23-05-2022 22:02:13:845 [pid=17711 tid=17711] ERROR cufio:2285 cuFileWrite invalid parameters invalid device pointer fd 48 offset 0 length 4096 bufOff 0 devPtr 0x7f81e3200000 23-05-2022 22:02:18:233 [pid=17714 tid=17714] ERROR cufio:2219 cuFile error issuing IO, buffer object is stale, user must deregister prior submitting new IO 23-05-2022 22:02:18:233 [pid=17714 tid=17714] ERROR cufio:2285 cuFileWrite invalid parameters invalid device pointer fd 48 offset 0 length 4096 bufOff 0 devPtr 0x7f30bb200000 23-05-2022 22:02:23:393 [pid=17717 tid=17731] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:393 [pid=17717 tid=17721] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:395 [pid=17717 tid=17731] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:395 [pid=17717 tid=17721] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:395 [pid=17717 tid=17731] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:395 [pid=17717 tid=17721] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:395 [pid=17717 tid=17731] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:396 [pid=17717 tid=17721] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:435 [pid=17717 tid=17779] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:436 [pid=17717 tid=17779] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:436 [pid=17717 tid=17779] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:460 [pid=17717 tid=17765] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:460 [pid=17717 tid=17765] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:460 [pid=17717 tid=17765] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:496 [pid=17717 tid=17737] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:496 [pid=17717 tid=17737] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:496 [pid=17717 tid=17737] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:520 [pid=17717 tid=17775] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:520 [pid=17717 tid=17775] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:520 [pid=17717 tid=17775] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:592 [pid=17717 tid=17723] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:592 [pid=17717 tid=17723] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:592 [pid=17717 tid=17723] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:615 [pid=17717 tid=17777] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:616 [pid=17717 tid=17777] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:616 [pid=17717 tid=17777] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:640 [pid=17717 tid=17759] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:640 [pid=17717 tid=17759] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:640 [pid=17717 tid=17759] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:664 [pid=17717 tid=17733] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:664 [pid=17717 tid=17733] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:664 [pid=17717 tid=17733] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:688 [pid=17717 tid=17763] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:688 [pid=17717 tid=17763] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:689 [pid=17717 tid=17763] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:712 [pid=17717 tid=17735] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:712 [pid=17717 tid=17735] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:712 [pid=17717 tid=17735] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:735 [pid=17717 tid=17757] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:736 [pid=17717 tid=17757] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:736 [pid=17717 tid=17757] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:784 [pid=17717 tid=17753] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:784 [pid=17717 tid=17753] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:784 [pid=17717 tid=17753] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:808 [pid=17717 tid=17781] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:808 [pid=17717 tid=17781] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:808 [pid=17717 tid=17781] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:832 [pid=17717 tid=17773] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:832 [pid=17717 tid=17773] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:832 [pid=17717 tid=17773] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:856 [pid=17717 tid=17745] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:856 [pid=17717 tid=17745] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:856 [pid=17717 tid=17745] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:879 [pid=17717 tid=17725] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:880 [pid=17717 tid=17725] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:880 [pid=17717 tid=17725] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:904 [pid=17717 tid=17769] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:904 [pid=17717 tid=17769] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:904 [pid=17717 tid=17769] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:932 [pid=17717 tid=17771] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:932 [pid=17717 tid=17771] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:932 [pid=17717 tid=17771] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:951 [pid=17717 tid=17729] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:951 [pid=17717 tid=17729] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:952 [pid=17717 tid=17729] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:988 [pid=17717 tid=17727] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:988 [pid=17717 tid=17727] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:23:988 [pid=17717 tid=17727] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:12 [pid=17717 tid=17751] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:12 [pid=17717 tid=17751] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:12 [pid=17717 tid=17751] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:35 [pid=17717 tid=17739] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:36 [pid=17717 tid=17739] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:36 [pid=17717 tid=17739] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:72 [pid=17717 tid=17743] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:72 [pid=17717 tid=17743] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:72 [pid=17717 tid=17743] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:108 [pid=17717 tid=17749] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:108 [pid=17717 tid=17749] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:108 [pid=17717 tid=17749] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:156 [pid=17717 tid=17719] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:156 [pid=17717 tid=17719] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:156 [pid=17717 tid=17719] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:192 [pid=17717 tid=17755] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:192 [pid=17717 tid=17755] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:192 [pid=17717 tid=17755] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:215 [pid=17717 tid=17747] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:216 [pid=17717 tid=17747] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:216 [pid=17717 tid=17747] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:264 [pid=17717 tid=17767] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:264 [pid=17717 tid=17767] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:264 [pid=17717 tid=17767] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:288 [pid=17717 tid=17761] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:288 [pid=17717 tid=17761] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:24:288 [pid=17717 tid=17761] ERROR cufio:1323 cuFileBufDeregister error: driver shutdown in progress 23-05-2022 22:02:41:357 [pid=17826 tid=17826] ERROR 0:867 error copying user data to bounce buffer for write error num: 1 error: CUDA_ERROR_INVALID_VALUE 23-05-2022 22:02:45:744 [pid=17829 tid=17829] ERROR cufio:2226 cuFile error issuing IO, size exceeds mapped length 23-05-2022 22:02:45:744 [pid=17829 tid=17829] ERROR cufio:2285 cuFileWrite invalid parameters access beyond mapped size fd 48 offset 0 length 196608 bufOff 0 devPtr 0x7f7a13200000 23-05-2022 22:02:50:147 [pid=17832 tid=17832] ERROR cufio:2201 cuFile access exceeds limits, input size + offset > 9223372036854775807 23-05-2022 22:02:50:147 [pid=17832 tid=17832] ERROR cufio:2285 cuFileRead invalid parameters invalid arguments fd 31 offset 9223372036854775803 length 32768 bufOff 0 devPtr 0x7f4f63200000 23-05-2022 22:02:50:147 [pid=17832 tid=17832] ERROR cufio:2201 cuFile access exceeds limits, input size + offset > 9223372036854775807 23-05-2022 22:02:50:147 [pid=17832 tid=17832] ERROR cufio:2285 cuFileRead invalid parameters invalid arguments fd 31 offset 9223372036854775803 length 32768 bufOff 0 devPtr 0x7f4f63200000 23-05-2022 22:02:50:147 [pid=17832 tid=17832] ERROR cufio:2194 cuFile invalid file offset: -9223372036854775808 23-05-2022 22:02:50:147 [pid=17832 tid=17832] ERROR cufio:2285 cuFileRead invalid parameters invalid arguments fd 31 offset -9223372036854775808 length 32768 bufOff 0 devPtr 0x7f4f63200000