The 'brush' referred someone with a learning difficulty employed to sweep up in a factory, hence the 'daft'. It's horrifically un-PC now!
That's awful. I really liked that expression and use it myself. Guess I won't anymore. Thanks for clearing up the mystery - I've wondered so often what it meant.