Leaving fish to find fish is a tough call, especially offshore. Ive done exactly that more than a few times but that was based on being out there 7 days a week and knowing that if I went a bit further I'd find WFO fishing, or getting a reliable report of truly great fishing.
I also personally prefer to never fuck with green water fish if I can avoid it. Caught a lot of green water fish, and often some of the biggest of the year come out of that water, but they can be fickle bastards in the green stuff. I'm happy to run miles past those fish to find more cooperative fish in the blue water if available.
IMO having a solid plan, and sticking to that plan more often than not, is a key for consistent success in the ocean. Not that there's not room for deviation, but I see far too many people chasing their tails out there because they have no game plan. I can certainly understand your change of plans, a chance to hop on a bad ass boat is hard to pass up! I also know you had a solid plan that would have worked out very well. But, it's tough when you hop on with someone else and aren't the one making the decisions anymore. It's probably the biggest downside of those scenarios.
Still, you found some toads and made the best of it, and saw first hand that your plan would have worked out perfectly if you hadn't been given the chance to go on the other boat, so at least you don't have to wonder about that.