The easiest way would be to archive any teammates that you would like to remove from your 1-on-1 list.
Go to the Streams section, and click on the 1-on-1 with that person. In the upper right hand corner there is a 'More actions' button, click the down arrow and 'Archive this stream'.
They will now be moved to the 'Archived' section and will hidden from view.
(However, if you ever need to access those notes, you can navigate to them)
Here is a gif showing the whole process: