Announcing the Beta Release of the SharePoint Capacity Planning Tool

After several months of work and very little sleep in between we are pleased to announce general availability of the SharePoint Capacity Planning Tool (Beta).


The SharePoint Capacity Planning Tool (Beta) provides a wealth of helpful information when planning a SharePoint Products and Technologies deployment; helping administrators, consultants, and IT Pros determine the potential required hardware investment(s) and topologies to support and meet requirements for availability and performance or even the affect of introducing additonal offices with their own unique bandwidth and latency constraints. 


Using the SharePoint Capacity Planning Tool (Beta) you can build topology models, run simulations, and generate reports that will help you evaluate and identify alternative solutions considering profile-based usage variations, content size and scope, networking technology, and availability requirements.


The SharePoint Capacity Planning Tool (Beta) provides both Windows SharePoint Services 3.0 and Microsoft Office SharePoint Server 2007 models allowing you to explore the unique characteristics of each platform and more effectively and efficiently plan topologies based on their respective infrastructure requirements.


To being using the SharePoint Capacity Planning Tool (Beta) or to learn more visit: https://connect.microsoft.com/programdetails.aspx?ProgramDetailsID=1602


I look forward to hearing your feedback and hope you find this tool invaluable in planning your SharePoint Products and Technologies deployment(s).

Planning for *Additional* Storage with Microsoft SharePoint Products and Technologies – Recycle Bin

This will be the first of a series focusing on those areas of storage that are commonly overlooked when planning storage for Microsoft SharePoint Products and Technologies. In this post we’ll look at the native Recycle Bin in Microsoft Office SharePoint Server 2007 and Windows SharePoint Services 3.0 to understand its fundamental aspects and impacts on storage.

The Recycle Bin enables end-users to easily recover content deleted within a site collection and is configured in two separate and unique stages referred to as the 1st and 2nd stage each with its own configuration properties.

image

The 1st stage Recycle Bin is configured using a numeric representation that defines the duration an item can exist in the Recycle Bin (SPWebApplication.RecycleBinRetentionPeriod Property), the default value of this property is 30 days; while the assumption is often made that this number is unique to the 1st stage Recycle Bin, conversely it applies to both the 1st and 2nd stage Recycle Bins and is based on the initial delete date of the content.

The 2nd stage Recycle Bin is configured on a percentage of quota value or more specifically a quota allocation (SPWebApplication.RecycleBinRetentionPeriod Property) as opposed to a duration as with the 1st stage Recycle Bin. This quota is configured to 50% of the live site quota by default, for example, a Web application with a default quota template of 1 gigabyte applied to a site collection can potentially result in a site collection consuming up to 1.5 gigabytes in a content database or more so, a Web application can be 50% larger than planned.  Though the 2nd stage Recycle Bin can be disabled entirely, you should carefully consider the recovery costs that can arise when content is requested for recovery – many organizations choose to increase the duration an item can exist in the 1st stage as a mitigation strategy.

Windows SharePoint Services 3.0 uses an automated cleanup service based on the initial delete date of a specific item and applies to both the 1st and 2nd stage Recycle Bins enabled as a Timer Job (see illustration).

image

Controlling the size of the 2nd stage Recycle Bin is an essential practice to managing storage for a site collection. The automated cleanup service will purge both the 1st and 2nd stage Recycle Bins when the initial delete date for a specific item or collection of items has reached its configured duration. Reducing the storage allocation for the 2nd stage Recycle Bin to purge items as the configured quota is reached on a more frequent basis can provide for the better management of storage; however, again may increase recovery costs where content becomes necessary to be restored from a conventional backup resource.

Cluster or mirror?

Should I cluster or mirror?  A few short months ago, the answer to that question would have been easy, now with the increasing popularity of database mirroring to achieve redundancy and geographic replication the answer is split…while each solution provides its own unique benefits, those benefits can be distributed across the answer to what are trying to achieve?


Database mirroring replays transaction log records on a standby server and as a result can failover more rapidly in most cases than a traditional SQL cluster with no loss of committed data; however, as a relatively new solution available to SharePoint Products and Technologies database mirroring has the highest operational costs when considering the learning curve required for the server support staff and database administrators.  While a database mirroring partnership can failover automatically through the implementation of a witness server (high availability/synchronous), there are no methods to enable automatic failover of SharePoint Products and Technologies introducing the added cost of developing a mechanism to manage the failover of the SharePoint service in the event the principal server is lost.  In many cases additional hardware will be required to support the witness server role in a mirroring partnership.  Also inherent of database mirroring is the requirement to duplicate storage across the principal and mirror servers; SharePoint Products and Technologies will consume only from one single physical server at a time unless databases are mirrored in a bidirectional manner.  To offset storage costs database mirroring can leverage traditional DAS storage and/or consume its storage from a SAN.


Clustering is the most common database architecture and as such is the least expensive in operational overhead.  Clustering enables running the same application on two or more servers providing a high availability solution if one of the servers fails.  Cluster software (Cluster Service) natively handles the failover process enabling SharePoint Products and Technologies to run uninterrupted on the second server.  Clustering when compared to mirroring offers greater processing power in addition to scalability since it does not carry the same two (2) physical server limitation as database mirroring and shares resources across each member node, reducing storage costs when compared to database mirroring; however, clustering comes with the requirement of shared storage and does not support the DAS storage options of database mirroring.  Geographic redundancy is limited with clustering since technology limits the distance by which two (2) servers can be separated, Fibre Channel for instance provides the greatest overall distancing, but is limited to a few miles.


Database mirroring and clustering both are capable of data replication, commonly through backing up the transaction logs from a database to a different server and applied there on a standby database (log shipping).  Before you consider log shipping in either a mirroring or clustered database architecture you should carefully review the latency between the publishing and subscribing servers.


There are many other considerations to understand prior to implementing either solution as part of your database architecture for SharePoint Products and Technologies that are not mentioned here that should be carefully reviewed and are complimented by an appropriate implementation plan.


In Microsoft IT we have a mixture, though unbalanced, distribution of SQL server clusters and mirror partnerships and by slowly introducing database mirroring in limited locations as one more option in our SharePoint Products and Technologies database architecture we allow allow more efficient growth and sustainability by not overburdening our operations staff with the support and maintenance learning curve and allow them to continue focusing on current deployments while gradually evolving our database mirroring architecture.


For additional information on SQL Server Database Mirroring with SharePoint Products and Technologies, see my three part series:


SQL Server 2005 Database Mirroring and Windows SharePoint Services 3.0/Microsoft Office SharePoint Server 2007 – Part 1 (Introduction, Overview, and basics)


SQL Server 2005 Database Mirroring and Windows SharePoint Services 3.0/Microsoft Office SharePoint Server 2007 – Part 2 (Configuration)


SQL Server 2005 Database Mirroring and Windows SharePoint Services 3.0/Microsoft Office SharePoint Server 2007 – Part 3 (Failover)


Additional Resources


Database Mirroring in SQL Server 2005


Inside Microsoft.com – Getting Started with Database Mirroring


Database Mirroring Best Practices and Performance Considerations


SQL Server 2005 Failover Clustering White Paper


Top Tips for SQL Server Clustering

Stress Testing Microsoft Office SharePoint Server 2007/Windows SharePoint Services 3.0

TechReady5 concluded on Friday and I’m finally returning to work after several customer sessions that immediately followed and one question was shared between the two – what do you recommend or what are you using to monitor performance and how do you determine load and stress when architecting a SharePoint Products and Technologies infrastructure?  The answer is, there are a variety of tools to stress test your Microsoft SharePoint Products and Technologies deployment; let’s cover some them:


SPSiteBuilder

SPSiteBuilder was originally a component of the SharePoint Utility Suite. As many of you know there are no future plans to redevelop this application set and SPSiteBuilder packaged with the SharePoint Utility Suite won’t work with Microsoft Office SharePoint Server 2007/Windows SharePoint Services 3.0. The good news is that making the application compatible with Microsoft Office SharePoint Server 2007/Windows SharePoint Services 3.0 does not require a great deal of effort or a large investment of time.

Let’s step through fixing the code to support Microsoft Office SharePoint Server 2007/Windows SharePoint Services 3.0:

First locate the line:


globalAdmin = new SPGlobalAdmin();
virtualServer = globalAdmin.OpenVirtualServer(uri);


SPGlobalAdmin is maintained for backward comparability; replace the globalAdmin variable with Farm and the SPGlobalAdmin() method with new SPFarm().


Farm = new SPFarm();


Now we need to replace the OpenVirtualServer() method since it is obsolete and use the Lookup() method of the WebApplication object.


WebApp = WebApplication.Lookup(uri);


Now locate the line:


if (!virtualServer.Config.Prefixes.Contains(strPrefix))
virtualServer.Config.Prefixes.Add(strPrefix, Microsoft.SharePoint.Administration.SPPrefixType.WildcardInclusion);


This enumeration member is obsolete in Windows SharePoint Services 3.0 because it is no longer necessary to tell Windows SharePoint Services which URL paths should not be handled by Windows SharePoint Services. So we should replace it with the WebApp variable we specified earlier.




if (!webApp.Prefixes.Contains(strPrefix))
webApp.Prefixes.Add(strPrefix, Microsoft.SharePoint.Administration.SPPrefixType.WildcardInclusion);
}



Finally to complete the retrofit we need to replace all instances of globalAdmin with Farm and all instances of virtualServer with WebApp and ensure we are referencing Microsoft.SharePoint.dll from a Windows SharePoint Services 3.0 build and recompile.


MOSSDW.EXE


MOSSDW.EXE is a performance testing tool that provides data population/stress testing capabilities for Microsoft Office SharePoint Server 2007.  If your plans include only data population, consider retrofitting SPSiteBuilder to support Microsoft Office SharePoint Server 2007/Windows SharePoint Services 3.0 as described above.


WSSDW.EXE


WSSDW.EXE is a performance testing tool that populates data for testing deployments of Windows SharePoint Services 3.0 (see also SPSiteBuilder and MOSSDW.EXE).


Excel Services Ocracoke Performance Testing Sample Scripts


Excel Services Ocracoke Performance Testing Sample Scripts are performance testing scripts used in conjunction with MOSSDW.EXE that provide data population/stress testing capabilities for Excel Services.


Internet Information Services (ISS) 6.0 Resource Kit Tools


The IIS 6.0 Resource Kit Tools can help you administer, secure, and manage IIS; using the Web Capacity Analysis Tool Version 5.2 you can stress test your application.


IIS Diagnostic Toolkit


The IIS Diagnostic Toolkit is a compiled set of tools aimed at reducing the overall time to resolve problems with Internet Information Services (IIS) products.  Use Debug Diagnostics 1.0 to run diagnostic tests for web servers hosting SharePoint Products and Technologies.


Web Application Stress Tool


The Web Application Stress Tool is designed to realistically simulate multiple browsers requesting pages from a web site. You can use this tool to gather performance and stability information about your web application. This tool simulates a large number of requests with a relatively small number of client machines.


SQLIO.EXE Disk Subsystem Benchmark Tool

The I/O system is important to the performance of SQL Server. When configuring a new server for SQL Server or when adding or modifying the disk configuration of an existing system, it is good practice to determine the capacity of the I/O subsystem prior to deploying SQL Server.  SQLIO.EXE can be used to determine the I/O capacity of a given configuration.  An example of what we may see on a typical day during peak usage on a x64 A/P cluster hosting 100+ databases and supports 8 unique server farms, each with a local Shared Service Provider is:


IOs/sec:  7808.43

MBs/sec:    15.25

sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file c:testfile.dat with 2 threads (0-1) using mask 0x0 (0)
2 threads writing for 360 secs to file c:testfile.dat
using 8KB sequential IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 100 MB for file: c:testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 3434.19
MBs/sec: 26.82
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 4
Max_Latency(ms): 282


SQLIOsim

SQLIOsim simulates the I/O patterns of Microsoft SQL Server 2005, of SQL Server 2000, and of SQL Server 7.0. The I/O patterns of these versions of SQL Server resemble one another.  You can use SQLIOsim to simulate read, write, checkpoint, backup, sort, and read-ahead activities for Microsoft SQL Server 2005  (see also SQLIO.EXE Disk Subsystem Benchmark Tool).

Performance/System Monitor

This is a standard component of the Windows operating system. You can use it to monitor performance objects and counters and instances of various hardware and software components.

Joel Oleson has a good list of counters for measuring Web front-end and backend performance – http://blogs.msdn.com/joelo/archive/2007/01/16/good-list-of-performance-counters.aspx.

NOTE At minimum consider monitoring PhysicalDiskDisk Transfers per second or LogicalDiskDisk Transfers per second measuring the amount of IO reads and writes per second to the database drives (see illustration).


Microsoft Operations Manager (MOM)


You can install MOM agents on individual servers that collect data and send it a centralized MOM server. The data is stored in the MOM database, which can be a dedicated SQL Server or the Microsoft SQL Server 2000 Desktop Engine (MSDE) version of Microsoft SQL Server. MOM is suitable for collecting large amounts of data over a long period of time.   MOM packs to consider for a typical SharePoint Products and Technologies deployment include:



Microsoft Operations Manager Packs


Microsoft Windows SharePoint Services
Microsoft SharePoint Portal Server 2003
Microsoft SQL Server 2000 & 2005
Microsoft Internet Information Server 6
Microsoft Cluster Service
Microsoft Windows 2003 Server


Web Site Monitoring


Web Sites and Services MP


Management Pack Catalog – http://www.microsoft.com/technet/prodtechnol/mom/catalog/catalog.aspx?vs=2005.


Stress tools such as Application Center Test (ACT)


You can use tools such as ACT to simulate clients and collect data during the duration of a test.


Network Monitor (NetMon)


You use NetMon to monitor the network traffic. You can use it to capture packets sent between client and server computers. It provides valuable timing information as well as packet size, network utilization, addressing, and routing information and many other statistics that you can use to analyze system performance.


SQL Profiler


Identify slow and inefficient queries, deadlocks, timeouts, recompilations, errors, and exceptions for any database interactions.  For example, one measure we implement is identifying queries exceeding two (2) seconds excluding search.  Profiling can be used when reports of latency are received in conjunction with monitoring the web front-end components of the server farm.


SQL Query Analyzer


This tool is also installed with SQL Server. You can use it to analyze the execution plans for SQL queries and stored procedures. This is mostly used in conjunction with the SQL Profiler.


SQLDiag


Collects valuable information about the configuration of the computer running SQL Server, the operating system, and the information that is reported to the SQL Server error logs.


 


SharePoint Products and Technologies Performance and Capacity Planning Resources


Plan for availability (Windows SharePoint Services)


Plan for availability (Microsoft Office SharePoint Server 2007)


Plan for performance and capacity (Microsoft Office SharePoint Server 2007)


Determining the hardware requirements for a single farm (Microsoft Office SharePoint Server 2007)


Performance and capacity planning (Windows SharePoint Services)


Performance and capacity planning factors (Microsoft Office SharePoint Server 2007)


Estimate performance and capacity requirements for Windows SharePoint Services collaboration environments (Microsoft Office SharePoint Server 2007)


Microsoft Office SharePoint Server 2007 on HP ProLiant Servers – Performance Summary


White Paper: Intel Performance Testing of Windows SharePoint Services

SQL Server 2005 Database Mirroring and Windows SharePoint Services 3.0/Microsoft Office SharePoint Server 2007 – Part 3 (Failover)

Database mirroring is increasing in popularity and becoming an integral part of high availability and disaster recovery solutions within the SharePoint Products and Technologies arena. I’ve spent much of the last few months building labs, testing scenarios, impacts on platforms whether it be WSS or MOSS.  This post is the third and final on my series [SQL Server 2005 Database Mirroring and Windows SharePoint Services/Microsoft Office SharePoint Server 2007].

We know with asynchronous mirroring we can automatically manage SQL Server 2005 failover by introducing the witness role in our server farms, the most challenging question to answer is how to manage web front-end failover.  In this post I will outline several possible solutions for managing SharePoint Products and Technologies failover in a mirrored database architecture.

Solution # 1 Network Load Balancing

In a high-availability environment, you can create a NLB cluster to easily route client requests from the original principal server to a promoted mirror server without having to update each client directly. NLB setup and configuration are fairly painless; however, it comes with the requirement that each server be on the same subnet and the active mirror remain in a suspended state in the load balancing rotation.  With this in mind you should carefully consider the limitations when using native NLB, for example you will not be able to geo-cluster in most circumstances. 

Solution # 2 Manual Failover

Manual failover requires the least in respect to infrastructure and configuration; however, has the highest operational costs and without proper management and monitoring can present the largest client impact in the event the of a failover. The steps required to instantiate a manual failover of SharePoint Products and Technologies are described in the first part of this series.

Solution #3 Custom Solutions (Example)

Custom solutions can be implemented monitoring the state of the principal and mirror servers redirecting the application calling the original principal server to the new principal server – typically implemented through a Windows service.

For example, using an aliasing scheme where SharePoint Products and Technologies accesses the SQL database server through an alternate name;  SharePoint Products and Technologies accessing the alternate name is redirected to the proper physical name of the principal server. Since SharePoint Products and Technologies is only aware of the alternate name and never accesses the physical name of either node it does not need to be failover aware.

From a logical perspective in the example solution proposed SharePoint Products and Technologies would make the request to the SQL alias, for example contoso, requests for contoso are intercepted and modified to reference the active principal node referenced in the web front-end server Registry as contoso1.  In a failover scenario, the Windows service would detect the principal role is on the new principal server, previously the mirror server, and replace the web front-end server Registry entry for contoso1 with the new principal physical server contoso2.