Lac and SSL with the Pound

I installed my Varnish server as follows:

backend web1 {.host = "XXX.XXX.XXX.XXX"; .port = "80";}
backend web2 {.host = "XXX.XXX.XXX.XXX"; .port = "80";}
backend web3 {.host = "XXX.XXX.XXX.XXX"; .port = "80";}

backend web1_ssl {.host = "XXX.XXX.XXX.XXX"; .port = "443";}
backend web2_ssl {.host = "XXX.XXX.XXX.XXX"; .port = "443";}
backend web3_ssl {.host = "XXX.XXX.XXX.XXX"; .port = "443";}

director default_director round-robin {
  { .backend = web1; }
  { .backend = web2; }
  { .backend = web3; }
}

director ssl_director round-robin {
  { .backend = web1_ssl; }
  { .backend = web2_ssl; }
  { .backend = web3_ssl; }
}

# Respond to incoming requests.
sub vcl_recv {
# Set the director to cycle between web servers.
set req.grace = 120s;

if (req.http.X-Forwarded-Proto == "https" ) {
   set req.http.X-Forwarded-Port = "443";
   set req.backend = ssl_director;
} else {
   set req.http.X-Forwarded-Port = "80";
   set req.http.X-Forwarded-Proto = "http";
   set req.backend = default_director;
 }

 ...
}

This works fine if I hit my IP address (without SSL) in the browser, but if I enable Pound (config below):

ListenHTTPS
    Address XXX.XXX.XXX.XXX #Local IP of the VarnishWebServer
    Port 443
    Cert "/etc/apache2/ssl/apache.pem"
    AddHeader "X-Forwarded-Proto: https"
    HeadRemove "X-Forwarded-Proto"
    Service
            BackEnd
                    Address 127.0.0.1
                    Port 80
            End
    End

End

I get 503 every time I try to get to the local IP address (from varnishlog -0):

   11 RxURL        c /favicon.ico
   11 RxProtocol   c HTTP/1.1
   11 RxHeader     c Host: XXX.XXX.XXX (Varnish Server IP Address)
   11 RxHeader     c Connection: keep-alive
   11 RxHeader     c Accept: */*
   11 RxHeader     c User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36
   11 RxHeader     c Accept-Encoding: gzip,deflate,sdch
   11 RxHeader     c Accept-Language: en-US,en;q=0.8
   11 RxHeader     c X-Forwarded-Proto: https
   11 RxHeader     c X-SSL-cipher: DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(128) Mac=AEAD
   11 RxHeader     c X-Forwarded-For: XXX.XXX.XXX.XXX (My Local machine IP)
   11 VCL_call     c recv lookup
   11 VCL_call     c hash
   11 Hash         c /favicon.ico
   11 Hash         c 198.61.252.81
   11 VCL_return   c hash
   11 VCL_call     c miss fetch
   11 Backend      c 14 ssl_director web2_ssl
   11 FetchError   c http read error: -1 0 (Success)
   11 VCL_call     c error deliver
   11 VCL_call     c deliver deliver
   11 TxProtocol   c HTTP/1.1
   11 TxStatus     c 503
   11 TxResponse   c Service Unavailable
   11 TxHeader     c Server: Varnish
   ...
   11 ReqEnd       c 1175742305 1391779282.930887222 1391779282.934647560 0.000097752 0.003678322 0.000082016
   11 SessionClose c error

I looked at my http listeners and I see this:

 root@machine:/etc/apache2/ssl# lsof -i -n|grep http
 pound     7947 www-data    5u  IPv4  63264      0t0  TCP XXX.XXX.XXX.XXXX:https (LISTEN)
 pound     7948 www-data    5u  IPv4  63264      0t0  TCP XXX.XXX.XXX.XXXX:https (LISTEN)
 varnishd  8333   nobody    7u  IPv4  64977      0t0  TCP *:http (LISTEN)
 varnishd  8333   nobody    8u  IPv6  64978      0t0  TCP *:http (LISTEN)
 varnishd  8333   nobody   13u  IPv4  65029      0t0  TCP XXX.XXX.XXX.XXXX:37493-   >YYYY.YYYY.YYYY.YYYY3:http (CLOSE_WAIT)
 apache2  19433     root    3u  IPv4  31020      0t0  TCP *:http-alt (LISTEN)
 apache2  19438 www-data    3u  IPv4  31020      0t0  TCP *:http-alt (LISTEN)
 apache2  19439 www-data    3u  IPv4  31020      0t0  TCP *:http-alt (LISTEN)
 pound    19669 www-data    5u  IPv4  31265      0t0  TCP 127.0.0.1:https (LISTEN)
 pound    19670 www-data    5u  IPv4  31265      0t0  TCP 127.0.0.1:https (LISTEN)

Where XXX.XXX.XXX.XXX is the internal IP address of the varnish WebServer, and YYYY.YYYY.YYYY.YYY is the IP address of one of the backend systems defined in VCL.

Any idea why I keep getting 503s?

UPDATE

As noted, Varnish does not support SSL, so using the pound can transfer traffic from 443 to 80, but when it is finished, it cannot use port 443 (ssl_diretector) to serve the traffic. Removing ssl_director and making default_director the main one works fine.