Immigration minister pledges to avoid EU labour