As long as the drawer is sturdy, it's not a problem with a very young baby.
I had a friend who babysat her cousin's baby and tried that. The kid was a chunky thing, and strong. He wasn't old enough to sit up, but he might have been old enough to roll over, and there was no room in the drawer to do that. When he woke up he kicked the crap out of that drawer.
I helped her try to put it back together again, but the tiny little nails wouldn't hold it together. We had to glue it, but the glue didn't work well, and I think we didn't glue it up square. Anyway, the drawer was useless af