I ride a Rohloff Thorn tourer loaded with four panniers and have tried touring a couple of times with the addition of a bar bag.
Both times I did not like feel of the front end with the bar bag loaded and soon took it off and lashed it to the rear rack!
So it might be worth looking for a frame fitting bag instead of using a bar bag.
Revelate Designs make a good looking 'Tangle Bag'. This could go some way to making the bike feel stiffer.
How much heel clearance do the rear panniers have? Moving them as far forwards as possible will minimise any wag that they can give to the back end and make the bike feel stiffer.
Reading the Sterling/Nomad brochure gives the impression that the Sterling is a lighter weight bike and will need a gentler riding style then the Nomad Mk 2 - maybe you are riding around the limit of what the bike is designed for.
If all else fails, sell the frame and refit the gear onto a Nomad Mk 2 frame which definitely will be stiffer.