Garbage Collector : Ignoring a reference

[Begah via Digitalmars-d-learn]

I am trying to create an asset manager for my textures. I had the 
idea ( it may be a wrong idea ) to create a hashmap of my 
textures with a string as the key. When the program request a 
texture, it firts check if it is in the hashmap and then returns 
if it is :

Texture[string] textures;

Texture loadTexture(string filename) {
   if(filename in textures)
     return textures[filename]
   else
     // Load image and put it in hashmap
}

Warning : I haven't tested if it actually doesn't work, but 
thinking about it, i think it should not.
My problem is that i return a reference of the texture to the 
program, but i keep one to myself and i want to free the texture 
if my program isn't using it anymore ( no more reference to it ). 
The problem i see, is that i will always have atleast one 
reference to the texture in my hashmap, but i want the garbage 
collector to ignore that reference and to free the texture if 
there are no more references anywhere in my program ( except in 
the hashmap ).

How could i tell the garbage collector to ignore the reference in 
the hashmap and to free it if there isn't any other reference 
that in my hashmap?