Being young, single, and female

Today I had a three hour lunch with the pastor and two older gentlemen from the church. We drove out to Waynesboro to a little place that serves true Southern cuisine buffet style. The food was excellent and the conversation was very engaging.

Throughout our time, one of the gentlemen kept calling me young lady. I didn't take any particular offense to it. However, when he and the other gentlemen were in the restroom, the pastor told me that I should "sock him one" the next time he called me young lady.

While I appreciate his defensive stance on my behalf, it's a difficult thing to address. For instance, on my first Sunday assisting in worship I was referred to as "the 19 year old" by one congregation member (and at this point I was even collared and robed, which helps to give one an air of authority). I kindly told the member that I was 24 and moved on to line up for the processional.

I think I seem younger to these congregation members because I am female and single. But how do I address it? Should I even respond? I would rather focus on the ministry to which God has called me. But what about times when these three factors are a hindrance?