The relationship between Prince Harry and the royal family has been tenuous at best over the last few years, so it’s no surprise that we haven’t seen him return to the United Kingdom often — and when he does, it’s usually without his children in tow.

But now, Harry’s opening up about the real reason Archie and Lilibet usually stay behind when he and Meghan Markle make their way across the pond, and it could mean that King Charles will never see two of his grandchildren again.

More from LittleThings: Hollywood Has Reportedly ‘Turned’ Against Prince Harry And Meghan Markle

According to what Harry said in a new interview with ITV that aired in the UK in late July, his main concern about bringing Archie, 5, and Lilibet, 3, over isn’t his feud with his family — instead, it all comes down to security concerns.

“All it takes is for one lone actor who reads this stuff to act on what they’ve read,” Harry said. “And whether it’s a knife or acid, these are things that are genuine concerns for me. It’s one of the reasons why I won’t bring my wife back to this country.”

Meghan Markle is who he mentioned, but he’s likely thinking about Archie and Lilibet, too. If they were to travel with the Duke and Duchess of Sussex, they’d also be in danger — and since it’s unlikely that Charles would travel to the US, that may keep him from spending any real time with his grandchildren … possibly forever.

Royal experts have told the Daily Mail they believe that the window of opportunity for Charles to have a relationship with Harry’s children is rapidly closing.

“It would be great to think that at some point soon the King would get to visit the grandchildren that he has seen very little of, but he is 75 and still not in the best of health,” Majesty magazine managing editor Joe Little told the outlet. “A visit to America is unlikely to be a high priority now, given the other demands on his time. It’s an incredibly sad situation that few would have predicted even five years ago.”

More from LittleThings: Piers Morgan Thinks Kate Middleton Should Tell Us What Kind Of Cancer She Has

Royal author Margaret Holder seems to agree, telling the Daily Mail that it would be “very difficult” for Charles to be able to make it to California to see Archie and Lilibet safely.

“So if the children can’t come here and the King can’t go there, you are looking at a situation where you’ve got a grandfather, who is undergoing cancer treatment, becoming more and more remote,” she said.