When Prince Harry and Meghan Markle announced that they were stepping away from their duties as senior members of the royal family in favor of living more private lives, they surely believed that media attention surrounding them would die down. Unfortunately, that isn’t the case and the couple is now filing suit in their new home of Los Angeles, California, against unspecified paparazzi who are accused of violating the Sussex family’s privacy.

Filed on Thursday in L.A. County Superior Court, the suit alleges that within hours of moving into their new home, photographers camped outside. Wanting more privacy for themselves and their 14-month-old child Archie, Harry and Meghan went so far as installing a huge mesh fence.