Generally it depends on your level of clearance and where you are going. If you are DV'd then you need to be careful with travelling to certain countries and its best to let the unit know. If SC and its somewhere normal then don't worry too much.
The key thing is if you tell them before you go you will be advised on what sort of contact to be aware off and thus be able to report any contact on return, thus keeping them happy.
A mate of mine many years ago went on holiday behing the 'curtain' and though it would appear no real attempt was made to subvert him, there were a few contacts that could have led to something if he had not had the briefing before he went.