Is it possible to always return http 304 status for requests with "If-Modified-Since" if the content is static

In our Asp.Net application, we have IHttpHandler processing requests for images. The handler is called with a special identifier that identifies the image in our image cache. When an image is placed in cash, it never changes. My question is:

Is it possible to always return an http 304 status code for requests with the heading "If-Modified-Since" without actually checking this date? The reason is that the browser should already have a copy of the image (since it provided a title with a changed name).

This will simplify life, because we do not yet track the date the image was created.

Here is the actual code (Update: I have included the if-modified header now in the server response, as recommended by Aristos):

public void ProcessRequest(HttpContext context)
{
    if (!String.IsNullOrEmpty(context.Request.Headers["If-Modified-Since"])) {
        //Is this okay?
        context.Response.StatusCode = 304;
        context.Response.StatusDescription = "Not Modified";
        return;
    }
    var thumbnailId = context.Request.QueryString["thumbnail"];
    using (var thumbnailCache = new CThumbnailCache()) {
        var imageBytes = thumbnailCache.GetImageById(thumbnailId);

        context.Response.ContentType = "image/png";
        var outputStream = context.Response.OutputStream;
        outputStream.Write(imageBytes, 0, imageBytes.Count());
        context.Response.Cache.SetCacheability(HttpCacheability.Public);
        context.Response.Cache.SetLastModified(DateTime.UtcNow);
        // added after Aristos post
        context.Response.AddHeader("If-Modified-Since", DateTime.UtcNow.ToString("r"));

        const int maxAge = 86400 * 14; // 14 Tage
        context.Response.Cache.SetExpires(DateTime.Now.AddSeconds(maxAge));
        context.Response.Cache.SetMaxAge(new TimeSpan(0, 0, maxAge));
        context.Response.CacheControl = "private";
        context.Response.Cache.SetValidUntilExpires(true);

    }
}