Procedure for Exchange 2010 DAG – Datacenter Failover

Brief

In Our Scenarios we had 10 Mailbox Server, 5 in each site & 2 Public Folder Server one in each site. We had two witnesses Server, the primary was set in DC1 and the alternate was set in DC2, so our primary Site was DC1.

This article is based on When we have Datacenter Activation Coordination Protocol is Enabled (DACP).

Step1. Ensuring that Primary DC Outage has been Declared

Ping Mailbox Server

Ping HUB Server

Ping CAS Server

Step2. Stopping the DAG from the Secondary DC for Primary DC Mailbox Server

Note:

• Ensure that you have Full Permission on the Exchange ORG, there should not be any restriction using RBAC.
• Before running the below command open the Exchange Management Shell “Run as administrator” otherwise the cmdlets won’t load to the PowerShell you may get an error message.
• This has to be ran from any of the Exchange Server From the Secondary DC

Stop-DatabaseAvailAbilityGroup –identity DAG1 -Mailboxserver EXM01.contoso.com -configurationonly

Stop-DatabaseAvailAbilityGroup –identity DAG1 -Mailboxserver EXM02.contoso.com -configurationonly

Stop-DatabaseAvailAbilityGroup –identity DAG1 -Mailboxserver EXM03.contoso.com -configurationonly

Stop-DatabaseAvailAbilityGroup –identity DAG1 -Mailboxserver EXM04.contoso.com -configurationonly

Stop-DatabaseAvailAbilityGroup –identity DAG1 -Mailboxserver EXM05.contoso.com –configurationonly

Step3. Stopping the Cluster Service on every DAG-Member of the secondary data-center

Note:

You might see sometime it will result that Cluster Service is already stopped, but no, check using the Services Console, Ensure Cluster Service is in stopped state.

Net stop clussvc

Step4. Reactivate the mailbox-servers in the standby data

Restore-databaseavailabilitygroup –identity DAG1

Note:

Most of the time you will see database will start mounting after executing this command successfully, and also this will start the cluster service on all the DAG members in the secondary DC. If this happens so then skip the step 5, if cluster service has not started perform the step.

Step5. Starting the Cluster Service on every DAG-Member of the secondary data-center

Net start clussvc

or

Open the services console and choose the Cluster Service and start it.

Step6. All the DB should be mounted to the respective Secondary DAG members

Note:

All the DB should be switched to the respective DAG member successfully. If mailbox-databases don’t switch automatically run the script to move databases to passive

Resume-MailboxDatabaseCopy ‘DATABASENAME\SERVERNAME’

or

Mount-Database -Identity ExchangeServerFQDN\DatabaseName

Step7. Check the Status of the DB

Check the DB Copy Status and Ensure they all are healthy. In My Case I had DB Created with Standard Name DB1 to DB72, So I used the below PowerShell to check the copy of all in one shot.

Get-MailboxDatabaseCopyStatus -Identity DB* | select name, status, SelectcontentIndexState | sort Status | ft –auto

Note:

Mount the databases if not done automatically.

Step8. Move OAB Generation to to Secondary Site Public Folder Server

Move-OfflineAddressBook -Identity “Offline Address Book” -Server EXP11

Step9. Do the Final check for Functionalities i.e.

Note:  

While accessing the Public Folder using Outlook you might see the outlook hanging but that’s ok, you can ignore it as once the connection get thru with new Public Folder Server, you won’t find the slowness.

• Email Flow (Internal/External):
• PUB Folder Access 
• ActiveSync:
• All DB: Mounted
• Free/Busy:
• OAB:
• OWA:
• POP3:
• IMAP:
• Blackberry:

Step9. Primary DC Switchback Steps

Note:

You got to know that the Network Connectivity has been resumed to the primary DC.

Also ensure that you got the final instruction that to proceed with Primary Datacenter Restore for exchange.

Ping Primary Mailbox Server

Ping Primary HUB Server

Ping Primary CAS Server

Step10. Ensure Cluster service has been started on Primary Datacenter Server, if not start it

Note:

You will see that Cluster Service is in Disabled state on all the Primary Datacenter in most cases the cluster service won’t start 

First enable the service by going to services.msc console and then start it

Or

Net start clussvc

If you are getting the error while starting the Cluster Service, then we need to perform cleanup by running the below command.

 

 

Cluster Node EXM01.contoso.com /forcecleanup

Cluster Node EXM02.contoso.com /forcecleanup

Cluster Node EXM03.contoso.com /forcecleanup

Cluster Node EXM04.contoso.com /forcecleanup

Cluster Node EXM05.contoso.com /forcecleanup

Step11. Start the DAG for all the Failed Server

Start-DatabaseAvailabilityGroup -MailboxServer EXM01.contoso.com

Start-DatabaseAvailabilityGroup -MailboxServer EXM02.contoso.com

Start-DatabaseAvailabilityGroup -MailboxServer EXM03.contoso.com

Start-DatabaseAvailabilityGroup -MailboxServer EXM04.contoso.com

Start-DatabaseAvailabilityGroup -MailboxServer EXM05.contoso.com

Step12. Then run the below command from any of the Exchange server

Set-DatabaseAvailabilityGroup –Identity DAG1

Step13. Move the OAB Generation server back to the EXP01 as it was earlier

Move-OfflineAddressBook -Identity “Offline Address Book” -Server EXP01

Step14. Redistribute the Mailbox Database copy across the Primary and Secondary DC

Note:

Ensure all the DBs are mounted and second copies are in health state.

Ensure all the nodes are up and showing good in the Failover Cluster Manager

In My Case I used mine one. You Can use your Own Script to Distribute The DB

Step15. Check all the functionalities

• Email Flow (Internal/External)
• PUB Folder Access:
• ActiveSync:
• All DB: Mounted
• Free/Busy:
• OAB:
• OWA:
• POP3:
• IMAP:
• Blackberry:

 Cheers,

Planning Exchange 2010 Deployment Project

One of the Crucial part before executing any project is Planning, Which is really hard when it comes to Designing a Solution for a kind of  Quite Complex Scenarios.

I was going thru few articles but cant find the detailed one, but finally i referred one book Exchange 2010 Best Practices written by Siegfried Jagott & Joel Stidley

It gives you the complete picture about your Exchange 2010 Project Planning, This is completely based on MOF as well as this also talks about live experience.

As per the Microsoft Operation Framework there is four layers Called “Plan, Deliver, Operate and the operation layer called Manage.

This Exchange Project Plan has touched the each layer of Microsoft Framework, Which talks what are do we need to do when as this is important to know about each of this and what we should talk about.

Plan

In this we are mostly going to ask question with client or gather information in the first meeting.

Business Question

• What are the organization’s strategic business objectives?
• What are the budgetary goals for these types of projects ?
• Are there any internal business roadblocks that could cause delays or objectives to the business requirements?
• Do any internal business processes or departments need extra attention to ensure the success of the project ?
• Which tasks should be handled by current IT staff, consultants, or other outsourced resources?
• What are the business reasons for adopting the particular technology?
• Are there business drivers for the migration or for the new implementation or technology?
• What industry-specific system requirements are needed ?

Technical Question

• What are the most important technology goals and objectives for your organization?
• What are the service-level requirements the messaging system and related services should meet?
• What are the functional requirements for the messaging system?
• Which IT skills and resources are strategic to the organization?
• Which tools and third-party applications need to be included in the design?
• How many users need to be included and where are they located?

Deliver

MOF includes five main steps in this phase: Envision, Project Planning,
Build, Stabilize, and Deploy. But as per real experience this has stretch with four more steps.

Envision

• Step 1: Envision: Identify business and technical requirements.
• Step 2: Assess.
• Step 3: Evaluate the new solution(s) and potential designs.
• Step 4: Build a proof of concept.
• Step 5: Create a design.

Project Planning

• Step 6: Develop the deployment and obtain buyoff.

Build/Stabilize

• Step 7: Implement a pilot. Begin pilot, adjust the plan, and complete deployment.

Deploy

• Step 8: Deploy.
• Step 9: Post-implementation review

Step 1: Envision:

• 1.1 Service and Organizational Level Agreements (SLAs or OLAs)
• 1.2 Operational costs
• 1.3 Network costs
• 1.4 Backup and restore cost and performance improvements
• 1.5 Improved provisioning processes (for example, HR Systems)
• 1.6 Exchange enhancement that enables larger mailboxes
• 1.7 Reducing licensing costs
• 1.8 Return on investment (ROI) and budgetary considerations
• 1.9 Data retention and isolation
• 1.10 Auditing and compliance requirements
• 1.11 Message archiving requirements
• 1.12 Antivirus and anti-spam functionality
• 1.13 Client access
• 1.14 Security requirements
• 1.15 Line-of-business application integration
• 1.16 Performance planning
• 1.17 Capacity planning and management

Step 2.Assess :

• 2.1 Identify and prioritize currently deployed applications by how crucial they are to business functionality.
• 2.2 Identify the clients that are currently in use.
• 2.3 Document service-level and organizational-level agreements in place.
• 2.4 Inventory the hardware currently in use.
• 2.5 Document the network infrastructure design.
• 2.6 Identify where the messaging servers are currently located, the number and size of local mailboxes and public folder replicas, and the number and average size of the e-mail messages sent and received.
• 2.7 Identify the Active Directory Domain Services (AD DS) configuration.
• 2.8 Update any messaging configuration documentation to ensure that all data needed to complete a project plan is available.
• 2.9 Identify the people involved in managing Exchange and all of the dependent services.
• 2.10 Perform a risk assessment
• 2.11 Document the executive escalation path.

Step 3: Evaluate the new solution(s):

• 3.1 Here you Evaluate the Product (Software & Hardware)
• 3.2 Looks about the Feature which would require for the clients
• 3.3 Third Party Prducts (Antivirus, Mobility, Archival)

Step 4: Build a proof of concept:

• 4.1 Prepare :

Verifying functionality is critical in the test environment.
This testing allows time to verify assumptions, verify
functionality, and create a detailed and accurate
deployment design. This is also a good time to learn
more about the product. To prepare for this work, gather
all of the software, hardware, and configuration guides that
are being considered for use in the design.

• 4.2 Deploy Proof of Concept:

Deploy an isolated test environment that is as close as
possible to the expected production environment and
allows testing to be completed. Testing in this phase should
include potential migration scenarios.

• 4.3 Test :

Perform the test scenarios and note the detailed results. Be
sure to note any changes to the test scenarios
because of new or changed features.

• 4.4 Review Test Results:

After the testing is completed, the issues and potential
changes should be reviewed. The reasons for any
unexpected results should also be captured. For instance,
did the test fail because a feature or function didn’t work as
expected, or were there limits reached?
The review should categorize each issue’s criticality and
then each issue should be addressed by changing the
design, opening support cases with the appropriate
vendors, or adjusting the current process to accommodate
the new functionality. Any user behavioral changes should
be documented and later clearly communicated to the
affected users.

Step 5: Create a Design

• 5.1 Define Client Standards
• 5.2 Define Network and Security Designs
• 5.3 Define Antivirus and Anti-spam Design
• 5.4 Define Application Compatibility and Integration
• 5.5 Define Infrastructure Changes
• 5.6 Define and Re mediate Risks
• 5.7 Develop Communication Plan
• 5.8 Develop Marketing and Training Plan
• 5.9 Define Migration Process

Step 6: Develop the deployment and obtain buyoff.

• 6.1 Create Design Milestones
• 6.2 Obtain Project Resources
• 6.3 Define Education and Training Requirements and Communications Plan
• 6.4 Obtain Executive Buy-in

Step 7: Implement a pilot. Begin pilot, adjust the plan, and complete deployment.

• 7.1 Pilot Planning
• 7.2 Implement the Core Exchange 2010 Infrastructure
• 7.3 Pilot Deployment
• 7.4 Evaluate the Pilot Process
• 7.5 Pilot Evaluation

Step 8: Deploy:

• 8.1 Create Migration Plan and Schedule
• 8.2 Complete Deployment Communication and Training
• 8.3 Implement the Exchange 2010 infrastructure
• 8.4 Perform Migration

Step 9: Post-implementation review:

• 9.1 Operate
• 9.2 Operations
• 9.3 Service Monitoring and Control
• 9.4 Customer Service
• 9.5 Problem Management

At Last the its Manage where you can take care for these Points.

Manage

Governance, Risk, and Compliance

Change and Configuration Management

Process Improvement and Management

Cheers,

Ashwini Kumar

 

How to:Re-seed Failed Mailbox Database Copy in Exchange Server 2010

As you can see here DB030 shows as Failed for DAG Member EXM15, to get this fixed we have to reseed the DB.

Step1: Suspend the Failed DB by executing the below command 

Suspend-MailboxDatabaseCopy -Identity “B030/EXM15”

This will ask your confirmation to perform the Suspend for this DB/Server, Select A [Yes to all]

As you can see below

 Step2: Update the Failed Mailbox DB copy from the Active one, by running this shell and Make resume it.

Update-MailboxDatabaseCopy -Identity “B030/EXM15” –DeleteExistingFiles

Note: The duration of the seeding process will depend on the size of the database and the speed of the network. When the update has completed it will automatically resume replication for the database copy.

Just in case if you want the replication to resume it manually use the below command.

Update-MailboxDatabaseCopy -Identity “B030/EXM15” -DeleteExistingFiles -ManualResume