If only groups have rights, then no one has rights because there would be no individuals. Beyond that, providing rights based on group identity is the formula for tribalism, tribal warfare, and injustice.
Conflict and wars between tribal collectives are part of world history, and we do not have to look far to be reminded how tribalism has played out over time. Native American tribes often fought each other; some enslaved members of other tribes. The Republic of Sudan, which has 500 ethnic groups who speak over 400 languages, has been in some form of civil war for decades. The same disaster has befallen other African countries, such as Ethiopia, not to mention non-African countries such as Syria and Myanmar.
Many modern intellectuals claim to be baffled about what American exceptionalism is, but it is simply the principle of individual rights. It is not pure democracy as such—not just voting—but a republican form of government. The Constitution protects individuals from mob rule.
With some exceptions at certain periods, almost everyone was welcome. America was a country of immigrants. The ideal was for America to be a melting pot—a place where people with radically different backgrounds and traditions would have equal rights and could live peacefully together, intermarry if they chose to do so, and make their way economically through free trade. They were free to have their own traditions and neighborhoods within the limits of the law.
To a great extent, the melting pot has been a success. Between 1820 and 1996, over 61 million immigrants from over 40 countries and territories came to the United States. In the U.S., all world religions including more than 200 Christian denominations exist. People are free to be agnostics or atheists or pragmatists. We have freedom of philosophy and no state religion. We are officially a secular country. Religious traditions are allowed, but they cannot contradict the Constitution. Everyone has the same rights and, in action, is bound by the same laws (though, in some cases, unfortunately, there are religious exemptions from the law).
This is not to say that the path has always been smooth—and many of the holes in the road have been caused by collectivism and group identity. The first issue to be addressed was slavery—this is discussed in Some Comments on the Issue of American Slavery.
It should be noted that another moral element is here. If certain collectives are to be favored, which ones will get priority? Black, White, Hispanic, and Asian are not the only possible collectives. And what of mixed races? What percentage of each racial DNA puts a person in a favored or disfavored group? Is the government to be the arbiter of the correct racial mixtures? What about non-racial collectives, such as the overweight, the weak and the uncoordinated in sports, the tone-deaf in music, the dyslexic in reading, and so on. What about the seven definitions of gender that are formally recognized by some organizations?
The fact is, when rights are given based on a collective or group identity, rights are also taken away based on a collective or group identity. How would this work? Obviously, it would have to be based on which collective has the most political pull at a given time.
As I noted, all this pushes us in the direction of tribalism, tribal warfare, racism, and injustice. If it comes, this will be the end of America.