"It would not be safe" - Irish Loyal and Patriotic Union