It has been awhile since we've seen you here, Eddie; welcome back!

You asked...
Would I notice the weight difference of not having the racks on and zoom along faster?
As it happens, I've done just that from the same motivations and can share what I found.
Removing weight does make the bike lighter by the amount removed (of course it does), but besides making the bike lighter to lift and carry and to pedal uphill, I didn't really
notice much difference in my day-to-day riding once the bike was up to speed. Any difference in
acceleration (say, across intersections and between traffic signals) was pretty much nil compared to, say, a lighter wheelset I swapped-in. The bike frame (and weight like racks attached to it) was static weight whereas the wheels were dynamic weight I had to accelerate every time I started up from a stop.
Apart from having a tube of medium thread-locking compound ready at hand to prevent the rack bolts from loosening in the future, it is a pretty straightforward operation to remove the rear rack to give it a try. However,
If your rear mudguard stays are attached to the rear rack (either to bosses or to P-clips), then you'd need to replace the 'guard stays with unbent ones attaching directly to the dropout eyelets or conduct your trials on dry days with no mudguards as those would need to come off as well. That would be the biggest hassle to conducting an A-B trial...swapping stays or removing the rear 'guards along with the rear rack.
For me, the value of having a rear rack proves itself more often than a front rack when riding without a touring load, as I can strap things atop the rear rack with a bungee or straps, use it as a perch for my Carradice saddle bag or a rack-top pack, or to carry front panniers on the rear for small loads. In contrast, my front lowriders are good only for hauling panniers. A front platform rack would be more versatile than the lowrider but would also weigh more (George/mckeg here uses a very small platform rack in addition to his lowriders and finds it handy). The largest gain I've noticed from removing the
front lowrider is less wheel flop and steering that feels a bit quicker than with the rack. That made a noticeable difference to feel but was no faster than before in my tests.
So, take these as my data points offered in good heart and see what others have to say as well.
Oh! As to weather, it was gorgeous here in the US PNW (Pacific Northwest) yesterday with blue skies, sunshine, and 23.3°C for the high. Today is grey, rainy, and cold in contrast at "only" 12.7C but the promise of a return to nicer is in the cards for Thursday.
Best, Dan.