Click below to get to a section quickly

INDEX

How do I?

 

 

 

Operator Quick Start

Connecting to the Messenger clouds:

  1. You must have a SecureID key fob as well as a unix account setup on the Messenger Machines. (Contact: mshinn or wlai)

2.      Run SecureCRT on any machine setup to connect to the public Internet.  This can be a corpnet machine that has Remote Winsock proxy installed. (Contact wlai for a license, but a 30-day trial copy is available for download from http://www.vandyke.com/download/SecureCRT/index.html )

3.      Click New on the Session List tab to create a new session.  Enter the following information:

·         Name = whatever friendly name you like

·         Protocol = SSH

·         Hostname or IP = law-l2.hotmail.com

·         Port = 22

·         Username = <leave blank>

·         Cipher = 3DES

·         Authentication = Password

·         Password = <leave blank>

·         <Click OK>

4.      Back on the Session List tab, select this newly created session, and click OK

5.      If this is your first time connecting to the law-l2.hotmail.com, then you will be prompted to save the identification and key for this sever.  Click Accept & Save.

6.      You will be prompted to enter your UserID.  Enter the UserID that Lorrie Wood gave you.

7.      You will be prompted the password.  First type in the four-digit personal prefix followed by the 6 digits currently showing on your SecureID key fob.

8.      You are now logged into the law-l2 machine, which is located on the Hotmail facility.  From here you can telnet to any of the Messenger machines.

9.      If you want to telnet to the Messenger machines, at the “>” prompt type in “telnet msgr-ns1”, where msgr-ns1 is the machine that you would like to access. 

10.  You will be prompted for the UserID and password for these machines.  Here you need to use the UserID and password for these machines.   Note that the password is not the SecureID passwords.

What are the machines?

Machine Name

Machine IP

Purpose

msgr-s1

209.185.128.171

Staging Server

msgr-ns1

209.185.128.132

NS’s

msgr-ns2

209.185.128.133

msgr-ns3

209.185.128.134

msgr-ns4

209.185.128.135

msgr-ns5

209.185.128.136

msgr-ns6

209.185.128.137

msgr-ns7

209.185.128.138

msgr-ns8

209.185.128.139

msgr-ns9

209.185.128.140

msgr-ns10

209.185.128.141

msgr-ns11

209.185.128.142

msgr-ns12

209.185.128.143

msgr-ns13

209.185.128.144

msgr-ns14

209.185.128.145

msgr-ns15

209.185.128.146

msgr-ns16

209.185.128.147

msgr-ns17

209.185.128.148

msgr-ns18

209.185.128.149

msgr-ns19

209.185.128.150

msgr-ns20

209.185.128.151

msgr-sb1

209.185.128.157

SB’s

msgr-sb2

209.185.128.158

msgr-sb3

209.185.128.159

msgr-sb4

209.185.128.160

msgr-sb5

209.185.128.161

msgr-sb6

209.185.128.177

msgr-dp1

209.185.128.152

DP’s

msgr-dp2

209.185.128.153

msgr-dp3

209.185.128.154

msgr-dp4

209.185.128.155

msgr-dp5

209.185.128.156

msgr-gdb1

209.185.128.167

Friends M-servs

msgr-gdb2

209.185.128.168

msgr-gdb

209.185.128.169

Virtual M-serv address through the Local Director.  Not telnet-able.

msgr-u1

209.185.128.162

Friends U-stores

msgr-u2

209.185.128.163

msgr-u3

209.185.128.164

msgr-u4

209.185.128.165

ldt-bud

 

Local Director

 

What are the key directories on the NS/DP/SB?

Directory

Purpose

Key Files

/home/hotmail/messenger

Main executable directories

·      ns or sb

·      server.conf

·     msgradmin

·      coreread

·      logfilter

·      whichns

/home/hotmail/messenger/conf

Configuration Files

 

(Updates Passport files and new mail templates)

·      friend_cur_machines

·      domainmap.txt

·      msgdomain.conf

·      cvr.csv

·      urllist.txt

·      not_allowed.txt

·      (3) F&F mail templates

·      ContestMail.txt (opt.)

/home/hotmail/messenger/temp

Membership directory search results

·      *.fnd files that are deleted every 15 min

/home/logs

Logs

·      ns.log or sb.log

·      ns.out or sb.out

·      ns.err or ns.err

·      ns.pid or ns.pid

·      sbstatxxxx.txt 

·   nsstatxxxx.txt

·   servmon.log or dpl

/home/hotmail/admin

Admin scripts

·      cleanup.fnd.pl (cleans up stale search results)

·      node monitor files

/tmp

Temporary directories

·      a convenient place to put temporary files

·      everyone has permission to read/write (for FTP, etc.)

·      cleaned up at reboot

How do I transfer files to and from the Messenger machines?

You have to do it in a few steps:

1.      From you PC, you can only zmodem files to the law-l2 machine.

  1. From law-l2, you can ftp to any machine in the cloud.  Note that the reverse is not true, i.e. you can’t ftp from another machine to law-l2.  So you must start ftp on law-l2, and then do a “put” or “get” from there.
  2. When putting files from law-l2 to another machine, you normally can only put files under /tmp directory. You will have to mv the file from /tmp to the proper directory after you logon to that machine.
  3. Under normal situation, you should ftp builds and configuration files to msgr-s1, the staging server, and then from there prop it to all the machines.

About the staging server msgr-s1

The staging sever is where builds and configuration files are prop from.  It contains the build tarballs, as well as the latest configuration files.  There are these directories in the msgr-s1 staging server:

Directory

Purpose

Key Files

/home/hotmail/messenger

where the scripts are kept

·      STOP

·      START

·      STATUS

·      UPGRADE

·      RESTART

·      msgrsrv.dat

·      server.conf

/home/hotmail/messenger/builds

where the build tarballs are kept

·      bldxxxx.tar.gz

/home/hotmail/messenger/conf

where the config files are kept

·      all configuration files except for server.conf

/home/hotmail/messenger/temp

where tarballs are untar and unzipped

·      temporary storage

  This server also contains stat scripts that look like the following:

 

Architecture Overview

DP – Dispatch Server

When users initially contact a Messenger server cloud, a Dispatch Server (DP) handles the packet that they send.  A server cloud may have a single Dispatch Server or it may have several of them.  (When there is more than one Dispatch Server, incoming packets are evenly distributed among them by a Cisco Local Director using a round robin strategy.)  The DP server selects (via a hash function that defines the partitioning strategy for Notification Servers,) the Notification Server (NS) that the user should be using, and tells the user to connect to that NS.  The software that’s installed on a Dispatch Server is actually identical to the software installed on a Notification Server (NS) as described below – the difference is in how the machine is used.  (In other words, any NS is capable of being used as a DP.  If you were to hook an NS machine up to the Cisco Local Director, or modify client software to initiate a connection to an NS machine’s IP address, then that machine would automatically start functioning as a DP.)

NS – Notification Server

Users maintain a persistent connection to a Notification Server (NS) for the duration of their session.  When there is more than one NS, users are partitioned across them, such that each user will always be sent to a particular NS.  (The partitioning scheme can be changed, to handle emergencies such as hardware failures, but usually the partitioning scheme remains static.)  When the user first connects to an NS, the NS performs authentication (by talking to an Authentication M-Serve,) and then handles all messaging between the client and the server cloud except for IM sessions.  When users request an IM session, that session is established by the NS on a Switchboard Server (SB), and the client is provided with connection information for connecting to the IM session.  NS Servers are listed in a configuration file (Server.conf) and that file is used as part of the user-partitioning algorithm.  If you were to add a DP machine to the file it would automatically start functioning as an NS because the bits installed on a DP are identical to those installed on an NS.

SB – Switchboard Server

IM sessions take place on the Switchboard Server (SB).  Switchboard Servers announce their availability by multi-casting a message that is received by the NS machines.  When a user is participating in more than one IM session, they will be connected to more than one Switchboard Server session.

Friends U – Friends U-Store

The Friends U-Store is the storage system for MSN Messenger Service.  This is where Messenger specific information is stored including the user’s friendly name, and the user’s lists (forward, reverse, allow, and block).  When we have more accounts than can be handled on a single U-Store, we spread the accounts across multiple U-Stores.  All U-Store servers provide access to their storage via XFS.

Friends M – Friends M-Serve

The M-Serve acts as an index to the U-Stores for all accounts.  The Friends M-Serve acts as an index to the Friends U-store.  When there is more than one Friends M-Serve, each Friends M-Serve is a full replica of the others.  It is possible to lookup a Messenger entry for any person located in any of the Friends U-Stores by searching in any of the Friends M-Serves.  In other words, while the Friends U-Stores are partitioned to each hold a portion of the Messenger accounts, each Friends M-Serve contains a full index for all of the Friends U-Stores.

Hotmail Servers used by Messenger

Auth U – Authentication U-Store

The Authentication U-Store is the storage system for Hotmail.  This storage contains all of the information concerning a Hotmail account including the password.  MSN Messenger Service looks up this password in order to perform user authentication.  All U-Store servers provide access to their storage via XFS.

Auth M – Authentication M-Serve

The M-Serve acts as an index to the U-Stores for all accounts.  The Auth M-Serve acts as an index to the Hotmail storage system.  When there is more than one Auth M-Serve, each Auth M-Serve is a full replica of the others.  It is possible to lookup a Hotmail entry for any person located in any of the Authentication U-Stores by searching in any of the Authentication M-Serves.  In other words, while the Auth U-Stores are partitioned to each hold a portion of the Hotmail accounts, each Auth M-Serve contains a full index for all of the Auth U-Stores.

Postman

The Postman delivers incoming mail to the Auth U-Store.  When the postman places a message in the user’s inbox, it also sends a notification to the user’s Notification Server (NS) informing the NS that the user has received mail.  In turn, the NS sends a message to the client and the user sees a popup notification.

Membership Directory

The Membership Directory is used to find users by name.  When a user tries to add someone by name, the Add Wizard looks for that name in the Membership Directory.  The results of that search (there can be multiple hits) are returned to the user so that the user may choose the person (by name and location) to whom they want to send Friends & Family (F&F) mail.  We do not disclose account information or e-mail addresses from this directory because it would be a violation of the privacy of people who are in the directory.

Messenger Server Cloud

The Messenger server cloud is located in the Hotmail facilities in San Jose.  This server cloud contains the following machines:

Five (5) Sun Ultra 5 machines (located behind a Cisco Local Director) serve as the production cloud’s Dispatch Servers.  The Local Director receives messages for the Internet address “messenger.hotmail.com” and evenly distributes them among the five dispatch servers.  These machines are named MSGR-DP-1 thru MSGR-DP-5.

Eighteen (18) Sun AXMP machines are used as the production cloud’s Notification Servers.  All Messenger accounts are partitioned across 20 Notification servers, named MSGR-NS-1 thru MSGR-NS-18.  The five Dispatch Servers figure out (via a hash function) which of the eighteen Notification Servers each user should be using, and tells the user to connect to that NS.  

Five (5) Sun AXMP machines are used as the production cloud’s Switchboard Servers.  These machines are named MSGR-SB-1 thru MSGR-SB-5.

Two (2) Sun AXMP machines (located behind a Cisco Local Director) serve as the production cloud’s Friends M-Serve machines.  Whenever one of the NS machines wants to communicate with a Friends M-Serve it will contact the address of the Friends M-Serve Local Director.  The Local Director will evenly distribute the requests among the two Friends M-Server machines.  These machines are MSGR-M-1 and MSGR-M-2.

Five (5) Sun E4500 machines are used as the Friends U-Store.  Each of these machines will store data for 20% of the Messenger accounts.  These machines have large RAID disk arrays and Qualstar tape drives for backup.

The production cloud does not have any Auth M-Serve or Auth U-Store machines that are separate from the Hotmail’s computers.  The Messenger production server cloud will communicate directly with the Auth M-Serve and Auth U-Store machines that are maintained by Hotmail.  Since Hotmail maintains those machines, they are not described here.

After the MSN Messenger test team signs off on a build as being high-enough quality to deploy, our operations staff runs the scripts necessary to move the build from MMSDNS onto the production machines.  This requires several steps, because we’re moving code from outside the Hotmail facility onto machines that are located behind the Hotmail firewall.

Setup and Machine Preparation

DP and NS

  1. Follow the SunOS 2.6 Install for standard Hotmail Backend machines.
  2. Install the following applications:
    1. GZIP: ftp://sunsite.unc.edu/pub/solaris/freeware/sparc/2.6/gzip-1.2.4-sol26-sparc-local
    2. TCSH: ftp://sunsite.unc.edu/pub/solaris/freeware/sparc/2.6/tcsh-6.07.02-sol26-sparc-local.gz
    3. PERL5: ftp://sunsite.unc.edu/pub/solaris/freeware/sparc/2.6/perl-5.005_02-sol26-sparc-local.gz
    4. TOP: ftp://sunsite.unc.edu/pub/solaris/freeware/sparc/2.6/top-3.5beta8-sol26-sparc-local.gz
    5. TRACEROUTE: ftp://sunsite.unc.edu/pub/solaris/freeware/sparc/2.6/traceroute-1.4a5-sol26-sparc-local.gz
    6. System Patches “Generic_105181-14”: ftp://sunsite.unc.edu/pub/sun-info/sun-patches/2.6_Recommended.tar.Z 
    7. Install Qmail, configure for outbound only

3.      Create directory

a.      /home/hotmail

b.      /home/hotmail/messenger

c.      /home/hotmail/messenger/temp

d.      /home/logs

4.      With root privileges, add into /etc/system the line

set tcp:tcp_conn_hash_size=32768

set tcp:tcp_close_wait_interval=60000

set tcp:tcp_keepalive_interval=600000

  1. <TBD: Need to install standard admin scripts>
  2. <TBD: Need to get server build.tar.gz>
  3. <TBD: need to get cron jobs and scripts>
  4. verify the following script:

  

SB

Switchboard machine is almost identical to Dispatch/Notification machines:

  1. Follow the SunOS Install for standard Hotmail Backend machines
  2. Install the following applications:
    1. GZIP: ftp://sunsite.unc.edu/pub/solaris/freeware/sparc/2.6/gzip-1.2.4-sol26-sparc-local
    2. TCSH: ftp://sunsite.unc.edu/pub/solaris/freeware/sparc/2.6/tcsh-6.07.02-sol26-sparc-local.gz
    3. PERL5: ftp://sunsite.unc.edu/pub/solaris/freeware/sparc/2.6/perl-5.005_02-sol26-sparc-local.gz
    4. TOP: ftp://sunsite.unc.edu/pub/solaris/freeware/sparc/2.6/top-3.5beta8-sol26-sparc-local.gz
    5. TRACEROUTE: ftp://sunsite.unc.edu/pub/solaris/freeware/sparc/2.6/traceroute-1.4a5-sol26-sparc-local.gz
    6. System Patches “Generic_105181-14”: ftp://sunsite.unc.edu/pub/sun-info/sun-patches/2.6_Recommended.tar.Z 

3.      Create directory /home/hotmail, /home/hotmail/messenger

4.      With root privileges, add into /etc/system the line

set tcp:tcp_conn_hash_size=32768

set tcp:tcp_close_wait_interval=60000

set tcp:tcp_keepalive_interval=600000

  1. <TBD: Need to install standard admin scripts>
  2. <TBD: Need to get server build.tar.gz>
  3. <TBD: Need to get standard configuration files for SB>

Friends M-serv

Follow standard Hotmail M-serv installation instructions

Friends U-store

Follow standard Hotmail U-store installation instructions

Q: Does the standard U-store installation include Veritas?

Q: what is the standard inode size used?

Network Setup[WYL1]

The following are ports that need to be enabled:

From

To

Port #

Requirement

The following are allowed for incoming connections from Public Internet:

Client

DP

1863 (NSPort)

Allow incoming connections from Internet

Client

NS

1863 (NSPort)

Allow incoming connections from Internet

Client

SB

1863 (SBPort)

Allow incoming connections from Internet

The following are allowed for outgoing connections to Public Internet:

NS

Public Internet

25? (SMTP)

Allow Qmail to send outgoing Email

The following are used for connections to/from Hotmail or other HM-operated sites:

NS