Brad Pitt and Angelina Jolie’s children have apparently chosen sides in their messy divorce.

Their daughters have chosen to drop Brad’s last name and use their mother’s, and the six children recently reunited to celebrate Angelina’s birthday at her house.

Amid their convoluted custody battle, Brad Pitt still gets visitation rights but has reportedly been diminishing the time he spends with his youngest children.

Regarding his adult kids, he can’t force them to spend time with him, so their contact is almost nonexistent right now.

Brad Pitt Is Limiting His Visits With His Younger Kids

Brad Pitt’s relationship with his children is strained at best, especially with his older kids, as a lot of them are legally adults and he can’t enforce any kind of visitation rights. But even with the children who are still underage, he still has very little contact with them.

“He has virtually no contact with the adult kids,” a source told PEOPLE, before adding that his busy schedule is also interfering with his visiting time.

“His engagement with the younger kids is more limited in recent months because of his filming schedule. [Angelina] has the kids most of the time, but per their agreement, he has visitation with the younger kids.”

This comes amid the news that Shiloh, Vivienne, and Zahara dropped his last name, opting to go by just Jolie. Apparently, Brad Pitt is worried about what that will mean for him in the future, as he thinks they might expose details about their childhood that might make him look bad.

“This is a dark cloud hanging over Brad’s head. He can muzzle Angelina with his lawyers, but he can’t muzzle his own kids,” a source told In Touch.

“They’ve stayed totally quiet so far, but at some point down the line it’s very plausible that they’re going to want to tell their side of the story. There’s nothing to stop them from doing it,” the source added. “And when they decide to go ahead and talk, Brad will just have to sit back and take it.”

Brad Pitt’s Reaction To His Daughter Changing Her Name

While Zahara and Vivienne have dropped Brad Pitt’s name socially, Shiloh has actually gone through the trouble of getting it legally changed to just Jolie. Vivienne isn’t 18 yet, so only time will tell if she will eventually decide to formally change her last name, but Shiloh’s decision reportedly broke the actor’s heart.

“Brad was devastated by this choice,” a source told Us Weekly. “To him, it was more than a change of name — it was a symbol of a deeper estrangement that has been brewing for years.”

Another source who spoke to PEOPLE revealed that he knew what Shiloh wanted to do.

“He’s aware and upset that Shiloh dropped his last name. He’s never felt more joy than when she was born. He always wanted a daughter. The reminders that he’s lost his children, is of course not easy for Brad. He loves his children and misses them. It’s very sad.”

This report caused outrage among fans, as this implies that Shiloh was his first daughter, and by the time she was born, Brad and Angelina Jolie had already adopted Zahara.

Only Brad Pitt and his children know what’s truly going on between them, but it doesn’t look as though a reconciliation will happen any time soon.