I would go with 4# for the increased penetration. At the range you'l lbe using it, a few more smallers holes shouldnt matter.
You'd do best to check it though. It may pattern more consistent with 6's than 4's. If its pretty obvious, I'd go with whatever patterns the best.