I just want to make sure that you understand this: The column set builder simply is another view of the column definitions that you create directly in the column definition screen and vice versa. If you delete a column in the column set builder then that column will be deleted from the column definition screen as well. So I really didn't understand your question properly.
Try creating a report without visiting column set builder. You will have to go through quite a bit of trial and errors to get the report right but that way you will avoid all this column set builder confusion.
See if you can upload the image of your report. That way I can understand better what is going on.