Permit me to explain.
God gives each of us a specific nature to be used with our specific gifts. The proper use of our gifts will bring great joy to us, be of spiritual benefit to others and, above all, will glorify the Lord. If someone asks you to do something, and you hate every second of it, that's a huge clue that it may not be your gift. Using your gift as the Lord intends brings incredible satisfaction.
For example, we often expect our pastors to be brilliant preachers, gifted teachers, wise counselors, have the administrative skills of corporate CEOs, have perfect marriages, children to die for, be witty writers...and that's just for starters. We expect them to be on call 24/7, take a cut in pay when times are tough, never complain or show irritation, never disagree with anyone and generally keep everyone happy with themselves and with him. And the thanks he gets? If he makes one wrong move the members go home after church on Sunday and have 'roast pastor.'
How do I know this? My parents were in the ministry.
Now this is clearly impossible. We cannot make everyone happy. The Lord Himself certainly didn't. And yet the man is clearly called to the ministry, one of the toughest jobs on the face of the earth. Who does he go to for support and succor? The stress of the ministry is horrendous. Do you pray for your pastor and his family? Make sure you do it. And often.
(Now how did I end up going down that rabbit trail?)
This is not a case of a man going into the wrong field or misusing his gifts. He was called of the Lord. It's a case of people expecting too much of one individual. This is especially true in the smaller churches, and it's a shame because pastoral burnout is a rampant problem. Where is the class that teaches this truth to church members? There are all kinds of retreats, seminars, conferences, support groups, etc. for church members to attend, but I have never seen one that addressed support and consideration for the pastor's family.
Another example is my own experience. My gifts are essentially speaking, teaching, writing, counseling and music. The Lord gave me a nature to accommodate those things. I have done all of these for many years and have seldom run into any problem. You may notice that these are not 'team' gifts as a general rule. There is a ton of preparation to do any one of them, and I love working alone...thinking, reading, etc. I've been doing all of these for so many years that they are second nature to me. Give me a subject, give me five minutes, and I can talk about it, write about it, counsel others about it, maybe even write a song about it! That comes from experience.
I love to do them.
On the other hand, if you ask me to be on a church board, there's no way it's going to happen. To me, that would be like going to an insurance seminar or something. I would be bored, restless, and disgusted that there was so much talk about something like what color to paint the nursery. I couldn't care less. And all that discussion seems to me a colossal waste of time. It will irritate me no end. (But I won't say anything...I just won't make that mistake again.)
What else don't I do? I can't sew. I hate crafts. I feel that I don't relate well to children ... I don't know what to say to them. I dislike small talk. Preparing and taking a dish to a potluck makes me wonder if this is what having a nervous breakdown is like (although I will do it). I wouldn't want to ever plan an event. I hate practicing anything. Sitting and just being an observer is absolute torture, with one exception: the sermon is my favorite part of any service. In other words I must be learning, thinking or accomplishing something in order to sit still at all.
So, when one gets to a certain age in life, saying 'no' is not a problem. You've had your gifts for a long, long time. You've sifted out the things that do not spiritually benefit yourself or others, nor do they glorify the Lord. If anyone tells you they wish they could do what you do, tell them that they can do things you cannot do. No one does everything. I always wished that I (round peg) could be a soloist (square hole), but the Lord in His infinite wisdom did not give me that gift. Perhaps when I get to heaven.... :-)
Know thyself. There is no rule that says you must bake the best pie at the county fair.
And don't forget to pray for your pastor and his family.