RandomGridStack/DRISEStack out of memory for > 200 masks

[aagrawal357]

Hi,

I followed example notebooks ModelComparisonWithSaliency.ipynb and DRISE.ipynb and was able to make the RandomGrid and DRISE work for 200 masks. DRISE seems to work much better than RandomGrid for 200 masks. As per your instructions in the RandomGrid notebook, I am trying more masks (1200) for RandomGrid but the system runs out of memory very quickly and the process fails. Would you know why? I have 16 GB RAM and 16 GB swap space. Is there any other way to run it without failing and without increasing the RAM size? Let me know if you need any more information around this issue. I would really appreciate your help with this!

Thank you!

[aagrawal357]

Is there no GPU support available with this implementation? Please let me know if there's any other way to generate more masks without OOM error. Thanks!

[Purg]

Hi @aagrawal357! Sorry for your troubles. I will have to give this a go again on my development machine to regain an understanding of the resource needs for that example. If I recall correctly, a major influence on RAM consumed is the pixel size of the input image. Are you using the image showcased in the notebook? If you happen to not be, maybe try out a smaller image or even a crop of an AOI (area of interest) from your original image.

As an aside, I would suggest trying to not perform anything that would enter your swap space: even with an SSD, the speed of processing when memory enters swap can be so drastically diminished it becomes infeasible to finish.

As for GPU utilization, that would only come into the picture with respect to your inferencing model, not specifically the RandomGridStack processing. The two models used within that notebook are by default constructed with the "use CUDA" parameter disabled so as to make them more universally processable (e.g. by continuous-integration CPU machines). However, to enable GPU processing, find the two use_cuda=False, constructor arguments and change what is being set to True, instead. Please give this a try and see if you find you have enough CPU RAM. I also do not remember off-hand what the GPU RAM requirements are for the models and hyperparameterization configured.

[aagrawal357]

Hi @Purg, Thank you for your response! You were right about the pixel size of the input image. I was using a different image than the notebook and it had a pixel size of 2688 × 1520. I was able to run both DRISE and RandomGrid with >200 masks when I tried with an image of pixel size 640x480.

Can I ask you a couple of follow-up questions?:

1. Is there a relation between the object size and number of masks? My object is relatively small as compared to dog/cat in your example image. I tried both DRISE and RandomGrid with 1200/2400 masks but results don't look good at all. How do you know how many masks one must use for a reliable saliency map generation? Attaching my results for your quick reference.

2. RandomGrid performs better than DRISE. Would you care to explain what's the difference between RandomGrid and DRISE? I have used the same (8, 8) occlusion grid cell size for both methods. Different p1 values were used i.e. 0.5 for DRISE and 0.7 for RandomGrid. Is that the reason for better performance of RandomGrid or is there something fundamentally different between these 2 techniques?

xaitk-test.zip

Thank you for your help! I appreciate it.

[aagrawal357]

@Purg Good morning! I would really appreciate your response on my questions above. Thank you!

[brianhhu]

Hi @aagrawal357, sorry for the delayed response! Glad you were able to get DRISE and RandomGrid to run when reducing the image size. In regards to your two questions:

1. Is there a relation between the object size and number of masks?

I do not believe there is any fundamental relation between object size and number of masks. However, the grid cell size might need to be adjusted to better account for smaller/larger objects (e.g. you might use s=16 for smaller objects). The number of masks essentially controls the final estimate of saliency maps, i.e. more masks generally leads to a less "noisy" saliency map, as more combinations of occlusion are tested with more masks. There is a fundamental trade-off in that using more masks also makes the saliency map computation more expensive (each mask requires a forward pass through the model). The number of masks can be somewhat problem-specific, but generally you should target a large number of masks you can fit for your given compute budget.

2. Would you care to explain what's the difference between RandomGrid and DRISE?

FYI, we ran into a similar issue as you when trying to apply black-box detection saliency to images/datasets with small objects. As a result, we created the RandomGridStack saliency method, which instead of partitioning the image into equally sized cells (as done by DRISE), fixes the absolute size of each grid cell. This might be useful if you know something about the average size of objects in your dataset (in pixels), and want to ensure your masks are smaller than this so that you can generate object-level saliency. You can see our example notebook here, which demos saliency map computation on aerial drone imagery from the VisDrone dataset (see the "Generate Saliency
/ Saliency Generation Object" section). The p1 parameter is also a tunable parameter of saliency map generation and may be somewhat problem-specific- you may need to test different values to get a good sense of results.

Hope this helps!