Multiple sockets raise ConnectException: connection rejected: connect

I am programming java in Eclipse on Windows XP. I have a multiprocessing simulation that uses ProcessBuilder to start a server and two clients. The server starts the thread to listen to two different sockets - one for each client. I can comment on the code for each client, and the other is excellent. When I try to start both of them, one client will always fail with ConnectException: Connection refused: connect. To which client does he think he is working more slowly, although this is hard to say. I can pause after starting the server, but before the clients, and netstat checks that both sockets are active. What could be the reason for this? I have some simplified code below.

Update. Based on the comments, I edited the code for a multi-threaded server on the same socket, but I still have the same problem. The code below shows the changes. It looks like a socket is opening and closing by one client before another can open it. I can throw suspension statements at the end of each client, which allows the other to complete, but this is a fix, not a solution. So now the real question is: how do I keep listening to ServerSocket until I close it?

Server

try{
    server = new ServerSocket(sockNum); 
} catch (IOException e) {
        System.out.printf("Could not listen on port %d\n",sockNum);
        System.exit(-1);
}
while(true){
    ClientWorker w;
    try{
        Socket connection = server.accept();
        w = new ClientWorker(connection);
        Thread t = new Thread(w);
        t.start();
    } catch (IOException e) {
        System.out.printf("Accept failed: %d\n",sockNum);
        System.exit(-1);
    }
}

class ClientWorker implements Runnable {
    private Socket client;

          ClientWorker(Socket client) {
           this.client = client;
          }

          public void run(){
            Object line;
            ObjectInputStream in = null;
            PrintWriter out = null;
            try{
              in = new ObjectInputStream(client.getInputStream());
              out = new PrintWriter(client.getOutputStream(), true);
            } catch (IOException e) {
              System.out.println("in or out failed");
              System.exit(-1);
            }

            while(true){
                try{
                    line = in.readObject();
                    //Send data back to client
                    out.println("Sent from broker");
                    if(line instanceof String)
                        System.out.println(line);
                    else
                        System.out.println(line.getClass());                        
                } catch (IOException e) {
                    System.out.println("Read failed");
                    System.exit(-1);
                } catch (ClassNotFoundException e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }
            }
        }
    }

customers

 try{
    socket = new Socket("localhost", socketNum);
    out = new ObjectOutputStream(socket.getOutputStream());
    in = new BufferedReader(new InputStreamReader(socket.getInputStream()));

    out.writeObject(message);
    String line = in.readLine();
    System.out.println(line);
    } catch (UnknownHostException e) {
        System.out.println("Unknown host: localhost.eng");
        System.exit(1);
    } catch  (IOException e) {
        System.out.println("No I/O");
        System.exit(1);
    }

controller

ProcessBuilder server = new ProcessBuilder("java.exe","-Xss64m","-cp","bin;jscheme.jar","ServerProcess");
server.redirectErrorStream(true);
Process runServer = server.start();

ProcessBuilder clientA = new ProcessBuilder("java.exe","-Xss64m","-cp","bin;jscheme.jar","ClientAProcess");
clientA.redirectErrorStream(true);
Process runClientA = clientA.start();

ProcessBuilder clientB = new ProcessBuilder("java.exe","-Xss64m","-cp","bin;jscheme.jar","ClientBProcess");
clientB.redirectErrorStream(true);
Process runClientB = clientB.start();