Failed to allocate memory on WSL

[realies]

Getting the following on Windows 11, Docker using WSL, and a 3090. Any ideas on what might be going wrong?

dalle-backend    | 2022-06-19 16:51:07.064468: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 1073741824 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:07.064509: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 1073741824
dalle-backend    | 2022-06-19 16:51:07.189267: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 966367744 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:07.189312: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 966367744
dalle-backend    | 2022-06-19 16:51:07.280465: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 869731072 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:07.280506: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 869731072
dalle-backend    | 2022-06-19 16:51:07.370207: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 782758144 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:07.370250: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 782758144
dalle-backend    | 2022-06-19 16:51:07.460675: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 704482304 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:07.460716: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 704482304
dalle-backend    | 2022-06-19 16:51:07.550655: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 634034176 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:07.550697: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 634034176
dalle-backend    | 2022-06-19 16:51:07.641023: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 570630912 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:07.641068: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 570630912
dalle-backend    | 2022-06-19 16:51:07.731488: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 513568000 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:07.731531: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 513568000
dalle-backend    | 2022-06-19 16:51:07.852876: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 462211328 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:07.852919: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 462211328
dalle-backend    | 2022-06-19 16:51:07.945450: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 415990272 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:07.945495: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 415990272
dalle-backend    | 2022-06-19 16:51:08.034007: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 374391296 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:08.034055: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 374391296
dalle-backend    | 2022-06-19 16:51:08.125173: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 336952320 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:08.125218: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 336952320
dalle-backend    | 2022-06-19 16:51:08.215963: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 303257088 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:08.216005: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 303257088
dalle-backend    | 2022-06-19 16:51:08.304813: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 272931584 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:08.304853: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 272931584
dalle-backend    | 2022-06-19 16:51:08.394675: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 245638656 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:08.394712: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 245638656
dalle-backend    | 2022-06-19 16:51:08.485141: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 221074944 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:08.485183: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 221074944
dalle-backend    | 2022-06-19 16:51:08.576900: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 198967552 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:08.576941: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 198967552
dalle-backend    | 2022-06-19 16:51:08.669168: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 179070976 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:08.669204: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 179070976
dalle-backend    | 2022-06-19 16:51:08.760085: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 161164032 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:08.760125: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 161164032
dalle-backend    | 2022-06-19 16:51:08.852423: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 145047808 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:08.852464: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 145047808
dalle-backend    | 2022-06-19 16:51:08.941447: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 130543104 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
dalle-backend    | 2022-06-19 16:51:08.941490: W external/org_tensorflow/tensorflow/core/common_runtime/device/device_host_allocator.h:46] could not allocate pinned host memory of size: 130543104