Receiving from UDP is or can be blocking and can be non blocking, you really don't need to dedicate a thread per identified connection, just recv/read and pass to a thread then write/send from anywhere.
simplify the the groups need and keep only one, also if you like loop like that then wait on call and block on recv then
handle then send, loop again, this will even simplify the things further, removing the need to pass data between threads altogether.