If you start eith debian and then use minstall and its clean-packages, you would be down to 4-8mb usage.
Then you can start lookin on whats left, like changing to dropbear like alteady said.
LEAs posts are a good startig point, but it feels somewhat incomplete now since time has passed.