If there are more than one character in the video, let me pick which one to track

I have some videos with two people in them and I want to track just one person's motion.  I would like to be able to tell the AI, frame by frame if i must, which of the two actors to track.  I tested an animation and it looks like the problem is it can't tell which actor to follow. Instead of moving each individual point one at a time, it would be good if the AI was able to retrack based on the one i choose from frame to frame.