Styjun

I'm using Julia 1.1 with JLD and HDF5 to save a file onto the disk, where I met a couple of question about the memory usage.

Issue 1:

First, I defined a 4 GB matrix A.

A = zeros(ComplexF64,(243,243,4000));

When I type the command and look at windows task manager:

A=nothing

It took several minutes for Julia to release those memory back to me. Most of the time, (In Task manager) Julia just doesn't release the memory usage at all, even though the command returned results saying that A occupied 0 bytes instantly.

varinfo()

 name size summary
–––––––––––––––– ––––––––––– –––––––
A 0 bytes Nothing
Base Module
Core Module
InteractiveUtils 162.930 KiB Module
Main Module
ans 0 bytes Nothing

Issue 2:

Further, when I tried to use JLD and HDF5 to save file onto the disk. This time, the task manager told me that, when using the save("test.jld", "A", A) command, an extra 4GB memory was used.

using JLD,HDF5
A = zeros(ComplexF64,(243,243,4000));
save("test.jld", "A", A)

Further, after I typed

A=nothing

Julia won't release the 8 GB memory back to me.

Finding 3:

An interesting thing I found was that, if I retype the command

A = zeros(ComplexF64,(243,243,4000));

The task manager would told me the cashed memory was released, and the total memory usage was again only 4GB.

Question 1:

What's going on with memory management in Julia? Was it just a mistake by Windows, or some command in Julia? How to check the Julia memory usage instantly?

Question 2:

How to tell the Julia to instantly release the memory usage?

Question 3:

Is there a way to tell JLD package not use those extra 4GB meomory?

(Better, could someone tell me how to create A directly on the disk without even creating it in the memory? I knew there's memory mapped I/O in JLD package. I have tried it, but it seemed to require me to create matrix A in the memory and save A onto the disk first, before I could recall the memory mapped A again. )

This is a long question, so thanks ahead!

Julia uses garbage collector to de-alocate the memory. Usually a garbage collector does not run after every line of code but only when needed.

Try to force garbage collection by running the command:

GC.gc()

This releases memory space for unreferenced Julia objects. In this way you can check whether the memory actually has been released.

Side note: JLD used to be somewhat not-always-working (I do not know the current status). Hence you first consideration for non-cross-platform object persistence always should be the serialize function from the in-built Serialization package - check the documentation at https://docs.julialang.org/en/v1/stdlib/Serialization/index.html#Serialization.serialize