I'd fertilize...
On the soils, that combo is certainly nutrient dense, but also heavy and may not drain as well as tomatoes would like, thus holding moisture for long enough to cause issues. There's gas exchange at the root level, and heavy, dense soils when watered heavily don't allow for the gas exchange, as the space between the particles is filled with water, and also the soil settles tighter in the pot. Container growing typically needs a light, fast draining soil , much faster than in ground soils drainage wise.
This extended wet period is not to a tomato plants liking, and can lead to other issues such as Fusarium.
Lift a pot and feel around in the drain holes at the bottom, do a potted plant prostate exam so to speak, stick a finger in the drain hole as far as you can...use two fingers if you want a second opinion...
If your finger comes out wet, and muddy, there's drainage issues. You can drill holes in the bottom of the pot to help alleviate this, and even use a bamboo stake to poke through the soils aligning with your new drilled holes, but you may damage some roots, so be gentle.
You'll then need to dry things out a bit, put some sort of riser under the pot like a piece of wood or anything that gets the pot away from direct contact to the ground, then only water when the plant 'asks' for it, when the leaves droop a bit in the morning...as it's not uncommon for leaves on tomatoes in pots to droop a bit in hot direct sun, as the black pots can raise soil temps above their comfort zone.
Not seeing Alterneria symptoms, so next look for dark brown lesions on the stems at ground level...an indicator for Fusarium.
Based on what you have said, I'm inclined to think poor drainage or associated soil borne pathogens, but its hard to diagnose things without hands on of course.
Tomatoes are reasonably heavy feeders, but Nitrogen isn't something they need a ton of, so not to concerning...use the fertilize you have, and it's formulated fine...Nitrogen is water soluble, so feed regularly.
