Use Jmap command:

jmap.exe -dump:format=b,file=HeapDump.hprof <pid>, pid can be found from system task manager or ps(linux)

 

Generate automatically when Java heap space out of memory

Add JVM options, -XX:-HeapDumpOnOutOfMemoryError

Where the file loaction?

After that, you can use visualVM or MAT(Eclipse), or other tools to open it.

 

VisualVM heap memory setting

on Windows, if you want to increase the memory for the Visual VM tool, you must can do it from in the file
%JDK_HOME%\lib\visualvm\etc\visualvm.conf
changing the default_options property