My children 7 and 11 share their yoto speaker we have no problems with this unless they are listening to the same card and manage to catch up and be on the same chapter as eachother.
There is no way to fast forward or rewind even if listening to through the app. Could this be made a thing through an update? fast forward 10secs at a time by twisting the orange button clockwise (like turning up volume) and anti clockwise to rewind 10secs at at a time? Or something that can be turned on or off in the settings?
My son had 14mins left of the last chapter of his second diary of a wimpy kid card and had to re-listen to the whole 42min chapter.
I know why this isn't a thing it makes the player simpler for the children to control but when older children are using it it kind of make sense to have the option.