The fundamental problem here is that Bootstrap grids are based on rows. If, for example, you squeeze 3 grid items into a row that has 2 columns then the third item will flow onto a new line (although still part of the same "row").
Then the next row starts and you left a gap at the end of the second visual line.
Which means it can work if you go from 6 to 3 to 1 columns, or from 4 to 2 to 1 columns, but not from 3 to 2 to 1 columns or anything similar.
There are two choices in such scenarios.
You can modify your View so that you jettison the normal markup generated for a Bootstrap grid so that it is more simple, with the div.row moved outside the wpv-loop tags so that there is only a single row.