Ben Affleck‘s divorce from Jennifer Lopez is hard for everyone in a blended family, but the one who might be feeling some strange after-effects is reportedly Jennifer Garner‘s boyfriend, John Miller. The business entrepreneur finds himself oddly in the middle of a Hollywood A-list breakup simply because of his relationship with Garner.

Miller is having to share his girlfriend with Affleck because the family is navigating the split with their three children, Violet, 18, Fin, 15, and Samuel, 12. They’ve grown quite close to Lopez’s 16-year-old twins, Max and Emme, and it’s been difficult for everyone involved. Understandably, Miller doesn’t love Garner spending so much time with her ex-husband. “John doesn’t like sharing Jen and there is a divide when it comes to her and Ben,” a source told the Daily Mail.

Miller does not have “any fears that they are getting back together,” but he might feel a bit jealous. The insider noted that “it has been tough coping with her support for Ben during his split from JLo.” In addition to Affleck’s split from J.Lo, the former couple also prepped their oldest daughter for her freshman year at Yale University — it’s been a busy summer. “John is looking forward to Ben and J.Lo’s split being done so he can start focusing on Jen,” they noted. “Bumpy roads recently are looking to be smoothed out.”

A source close to Garner confirmed to People that she and Miller are “happy” after breakup reports surfaced this week amid Lopez’s Aug. 20 divorce filing. The Yes Day‘s star’s relationship is usually under the radar, but Affleck’s situation unwillingly thrust everyone into the spotlight. After this hurdle, fans can expect Garner and Miller to go back to what an Us Weekly source described as a “very private” dating strategy.

Garner reportedly feels like she made the right choice in partners with Miller too. “The relationship that Ben and Jen have is different, and where that bothers John, Jen actually likes that it is different because that is what attracted her to John in the first place,” the Daily Mail source added. It sounds like Miller is going to have to have patience and ride through the Hollywood headlines because the media will soon move onto another A-list hot topic.