If you want the entry to be garbage collected, use WeakHashMap. Once the key is not strongly referenced, the entry will be automatically reclaimed.
Here is a good article to understand this:
http://www.codeinstructions.com/2008/09/weakhashmap-is-not-cache-understanding.html