I don't think there's any requirement to go as a full party so I'd assume with care this would be possible. You'd just have to be careful about aggro. Until we know whether these random things we have to engage with impact things like gathering it's hard to be sure but I can't see why they'd restrict gatherers.