Access Denied when deleting the assembly created by the CodeDomProvider compiler?

If you use the CodeDomProvider class in .NET after compilation is complete, you cannot remove the output assembly. I want to remove the output assembly. File.Delete returns an access denied exception.

string asmPath = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString("N") + ".exe");
string keyPath = "some path to a *.snk file that works";
// build compiler
CodeDomProvider dom = CodeDomProvider.CreateProvider("VisualBasic");
CompilerParameters cp = new CompilerParameters();
cp.TreatWarningsAsErrors = false;
cp.GenerateInMemory = false;
cp.GenerateExecutable = true;
cp.CompilerOptions = "/target:winexe /keyfile:\"" + keyPath + "\"";
cp.OutputAssembly = asmPath;
// add all the other assembly references
string netPath = @"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.0\";
cp.ReferencedAssemblies.Add(netPath + @"System.dll");
cp.ReferencedAssemblies.Add(netPath + @"System.Core.dll");
cp.ReferencedAssemblies.Add(netPath + @"System.Data.dll");
cp.ReferencedAssemblies.Add(netPath + @"System.Xml.dll");
CompilerResults cr = dom.CompileAssemblyFromSource(cp, new string[] { code });
if (cr.Errors.Count == 0)
{
    cr.TempFiles.Delete();
    dom.Dispose();
// do stuff
...
//
    File.Delete(asmPath); // fails here Access Denied
}

Edit As a job, I use a static class constructor that looks for a temporary folder and deletes previously created assemblies.