If I encounter pot holes
Will it make much of difference what the wheel size is? 26” v 650b v 700c ?
Not much. Being the largest diameter 700C is theoretically better, which explains the current 29er trend for ATBs, but I reckon the most important thing is tyre width. A reasonably fat tyre aids comfort and protects the rim to some extent from rocks and potholes.
26”/650b/700c are all significantly better than 16"/20", which I have also used on mild off-road riding.
For a 26" wheel touring bike that is used occasionally off-road (my case) I use lightweight 50 mm width tyres (Marathon Supreme), which are a reasonable compromise of good rolling resistance on good roads and still acceptable on mild off-road (tracks, footpaths).
If planning more off-road use I would go to Marathon Duremes in the same width, still lightweight but a bit slower and probably better able to resist rocks, with slightly more grip. For expedition type rides I would probably go for something like the Marathon Mondial, either in the same 50 mm width or slightly bigger if compatible with the mudguards.