How does ROBOTS.TXT say and is it being honored?

/Pete