The DNS  change that  solved the  MSN.COM mail  delivery problem  has now
propagated and expired from caches, and just about everyone has confirmed
that mail  is now coming in.  We diagnosed the problem  yesterday evening
with  the MSN  Escalation Team  after noticing  that there  was something
wrong with the  DNS data for MSN.COM. As you  know, MSN technical support
had been telling people  that the lack of incoming mail  was due to known
problems  with the  MSN.COM servers  and that  these problems  were being
worked on  with high priority, so  everyone assumed that this  was indeed
the case and that  the best thing to do was to give  them a break and let
them spend  their time working  in peace rather than  answering questions
and  not getting  any  work done.  But  the problem  that  they had  been
referring to was unrelated  (at least we think so). Due  to a shortage of
accurate, unemotional, first-hand information :-), it took a few hours to
pin-point the exact cause and location  of the problem, but once this was
done MSN promptly fixed  the problem. As it was a DNS  change, it may not
have expired from caches until about now.

It  is of  course very  unfortunate that  problem resolution  was delayed
because MSN technical support was seeing the same symptoms as for another
problem and providing the same stock  answer, but in all fairness to them
this DNS  problem was  far beyond  the knowledge of  any level  1 support
person, or  even a  junior level  2 person.  Remember that  MSN technical
support deals mainly with questions  from people who need help installing
their  modem  or  configuring  their  mail client,  etc.  There  must  be
thousands of people calling every day to explain that they aren't getting
their  mail, most  of the  time because  their client  is not  configured
properly. Much as I think this should  not have happened, I am not really
sure how one could train a level 1 person to tell these problems from the
one that was reported or the one for which they had the boilerplate about
ongoing changes. Maybe  some kind of pattern analysis, a  sudden surge of
complaints  from  avid mailing  list  users  triggering a  more  in-depth
investigation, something like that. Either way,  once it got to the right
people it was  processed promptly. They were not aware  of the problem at
all because  the complaints had  been identified as being  something else
that involves entirely different people. Likewise there was no reason for
anyone to doubt the original diagnostic until enough time had passed that
it was starting to sound more and more suspicious.

Anyway,  if your  users are  still experiencing  the problem  by tomorrow
morning US time, please let me know so that we can investigate.

  Eric