I think it depends on how much he contributed to your life. We tend to miss people who have enhanced our life and we tend to not miss those who did not contribute.
If you are on a floor at work, and somebody obnoxious leaves, you don’t miss him. If somebody leaves who would clog up the frig with smelly food, you don’t miss him. If somebody leaves who helps out other people and usually maintains a cheerful demeanor, you miss that person. The more they contribute, the more you miss them.
I think this same thinking should apply to groups. Those groups that contribute the most to America would be the most missed if they left. Those groups whose contributions are outweighed by their harms are not missed. In fact, they are in danger of the majority eventually acting in their self-interest and forcefully expelling or killing them.