Grant actually tried to do quite a bit for civil rights, but failed because he didn't do enough. Grant gets a really bad rep, but was a better president and person than he is portrayed.
As always, the truth is complicated.
https://www.nytimes.com/2019/03/04/opinion/ulysses-grant-president.html
This article helps a bit.