Assuming that you use <CR>to "select a line in the quick access list", you can simply reassign <CR>in the quick fix window to return to the previous window after switching to the corresponding line.
Make sure you have one filetype plugin indent onand add the line below to ~/.vim/ftplugin/qf.vim:
nnoremap <buffer> <CR> <CR><C-w>p
source
share