I don’t think you’re alone with this. When I started work back in the late 60s, we had a manager who was like this. Yes, we had office banter but never crossed the line to be offensive. Our office was multi cultural, Asian, West and East African, Caribbean, Eastern European and South American. We got on well together and did Josh each other but never racially or religiously, although one guy insisted on calling Roman Catholics “rock cakes” for some reason.
The crude manager didn’t like it when all “the girlies” as he called us, reported him. He got transferred to another site in a far flung part of the Commonwealth. This was in the days before any discrimination laws.