I think if minority voters were more evenly split between the parties, then your argument would make more sense. But minorities (for the most part) are heavily aligned towards the Democrats.
Trump isn't really helping his reelection chances if he marginally increases his share of the minority vote, while losing massive ground with seniors and women voters.