Managing large external dependencies using mercury

I am working on a project that uses several large third-party libraries such as boost, tbb and ACE. I would like to precisely control which versions of these third-party libraries are used in my project, and not rely on everything that is available on the build machine. In the past, I have done such things by adding these libraries to the project repository. This works fine, but it slows down the mercury movement, causing it to track a couple of orders of magnitude more files than what is in my project. Is there a better way to manage external dependencies?

Thanks in advance