Ghosts in the Machine?

Ghosted and unghosted pages are references not new to Microsoft Office SharePoint Server 2007, but have received increased interest as a result of their impact on upgrading from previous versions and more recently, the ability to manage pages in an unghosted state.


Introduction


Ghosted is the preferred state of pages in a site collection, ghosted pages refer to site definition files cached in memory on the server at process startup of IIS.  By caching site definition files in memory performance and scalability are improved by reducing data storage and retrieval requirements.  Ghosted pages can be reused as a result across one or many site collections on the Web application.


Unghosted pages are most commonly the result of customization through Microsoft FrontPage and/or Microsoft Office SharePoint Designer and are stored in the corresponding content database.  The contents of unghosted pages are routed through safe mode parsing in ASP.NET, which prevents server-side code from executing, and which depends entirely on the Safe Controls list specified in the web.config file of the wwwroot directory to determine which controls can be rendered at run time.  When upgrading unghosted pages in SharePoint Portal Server 2003 and or Windows SharePoint Services 2.0, some functionality is lost until the page is reset to its site definition (ghosted) and can include security trimming, navigation (Site Actions), Recycle Bin, and other core Microsoft Office SharePoint Server 2007 functionality.  The performance penalty of unghosted pages in Microsoft Office SharePoint Server 2007 is less evident when compared to previous versions due to enhancements in the .NET 2.0 runtime.


So how do pages become unghosted?


In most cases an unghosted page is the result of customization through Microsoft FrontPage and/or Microsoft Office SharePoint Designer 2007 – browser based modifications such as the manipulation of Web Parts will not cause the page to be unghosted.  In a change over Microsoft FrontPage, Microsoft Office SharePoint Designer users are presented with a prompt to indicate the page they are working with will no longer be associated with its specified site definition (see illustration).



Reghosting


In many cases users can resolve minor customizations through resetting the page to its site definition through both the Windows SharePoint Services 3.0 user interface and Microsoft Office SharePoint Designer; however, changes made within Web Part Zones will be retained after synchronizing the page with its site definition.  [Site Settings | Look and Feel | Reset to site definition]. 




In some circumstances a page cannot be reghosted, for example if the page was not based on an existing page, was created from a blank page, imported from another application or editor or (most commonly) was associated with a site definition no longer available to the server farm – this can occur if a site definition was retired or a database migration approach was implemented and the upgrade definition not made available to the target server farm.  In many cases you can simply copy a known good <page>.aspx from an alternate location and retrofit it to restore functionality.


C#


Reghosting can be also accomplished through by a server farm administration calling the RevertAllDocumentContentStreams method of the Microsoft.SharePoint namespace on each SPWeb object.


Sample Code

        static void Main(string[] args)
{
string siteUrl = “”;
SPSite Site = new SPSite(siteUrl);
foreach (SPWeb Web in Site.AllWebs)
{
Web.RevertAllDocumentContentStreams();
}
}

This code snippet is provided under the Microsoft Permissive License.


Microsoft Office SharePoint Designer 2007



  1. In Office SharePoint Designer 2007, open the Web site where the customized page resides.
  2. Right-click the page that you want to reset to the site definition, and then click Reset to Site Definition on the shortcut menu.

NOTE A copy of the customized page is placed in the same directory where the reset page resides and is named file name_copy(1), where file name is the original file name – when reghosting a page through the Windows SharePoint Services 3.0 user interface a backup copy of the page is not created.



MasterPages


If the customization was implemented on a MasterPage, the MasterPage and attached pages can be reset to their site definition through Microsoft Office SharePoint Designer 2007:


In Office SharePoint Designer 2007, open the Web site where the customized master page resides.




  1. Right-click the master page that you want to reset, such as default.master, and then click Reset to Site Definition on the shortcut menu.


    Note   By default, in the Folder List, master pages are located in the masterpages folder, which is in the _catalogs folder in the site.

    A warning message informs you that the master page’s contents will be overwritten, but that a backup copy of the current page will be created.


  2. Click Yes.

Ghosted Pages and Upgrade


Ghosted pages, though, in many cases can be upgraded safely depending on the extent of their customization (see introduction for caveats) should be considered for reghosting at the time they are upgraded.  The gradual upgrade approach allows administrators to reghost (reset to site definition) pages during the upgrade process and enables the reghosting of pages at a more granular level.  Reghosting in a gradual approach can be applied through the Upgrade user interface or optionally using the STSADM -o command line argument -Reghost in the upgrade operation, for example STSADM -o upgrade -sidebyside -url http://www.contoso.com -sitelistpath <pathtoxml> -reghost.  This granularity allows you to decide where you would like to reghost and what customizations should be retained during the upgrade.  The database migration approach and inplace approaches both permit the upgrade of unghosted pages preserving the customizations; however, these pages will be upgraded in their existing state and will need to be reghosted post-upgrade to enable Microsoft Office SharePoint Server 2007 functionality.


Locating Unghosted Pages


Now that we understand ghosted vs. unghosted pages and their implication on upgrade, how do we determine where unghosted pages exist?


PRESCAN


PRESCAN.EXE has two primary purposes:



  1. It parses and saves List definitions with the associated Lists.  SharePoint Portal Server 2003 Service Pack 2 already incorporates this feature whenever a list is modified; however, this process should be completed for all Lists, so prescan calls the SharePoint Portal Server 2003 Service Pack 2 method to persist that data.
  2. PRESCAN.EXE will report on common issues that will result in a failed upgrade; therefore, running PRESCAN.EXE, addressing reported issues, and resolving those issues, and re-running PRESCAN.EXE to verify those fixes is a best practice when planning a Microsoft Office SharePoint Server 2007/Windows SharePoint Services 3.0 upgrade.  The most commonly detected issues are:

    • Database Orphans This is a class of issue where an object exists, but the pointer with the parent object is broken and/or corrupt.   Classic examples include situations where a site exists in the content database; however, does not exist in the configuration database and a web that points to a site collection that no longer exists. 
    • Missing Site Definitions This issue is rare at best ad exists when a site collection has been removed/deleted – sites under this classification will not be upgraded and in addition those sites will not render in SharePoint Portal Server 2003/Windows SharePoint Services 2.0.

PRESCAN can be used to identify custom Site Definitions, FrontPage customizations, Web Parts, etc.

PRESCAN.EXE will report and summarize any unghosted pages occurring in your environment (see illustrations).



The example above illustrates unghosted pages discovered by PRESCAN.EXE and reported in the application log file.



The example above illustrates a summary of unghosted pages.


SQL


Alternatively SQL can be leveraged (not recommended on production databases) to generate a report of unghosted pages in your content databases (see example).



SELECT


     Docs.DirName,

     Docs.LeafName

FROM Docs WHERE

     (Docs.Type = 0)

AND

     (Docs.SetupPath IS NOT NULL) AND

     (dbo.Docs.DocFlags & 64 = 64)


Microsoft Office SharePoint Designer 2007

You can also run a report in Office SharePoint Designer 2007 to list all of the customized pages in your site.


  1. In Office SharePoint Designer 2007, open the Web site for which you want to run the report.
  2. On the Site menu, point to Reports, point to Shared Content, and then click Customized Pages.

    The report opens with all pages in the site listed, and the Customized column indicates whether content has been customized for that page.


  3. To display only pages that have been customized, click the down arrow to the right of the Customized column, and then click Yes.

    The report now displays only pages that have been customized.


Additional Resources


Joel Oleson’s To Ghost or Not to Ghost, That is the Question


MSDN – Custom Site Defintiions


Microsoft.com – SharePoint Application Templates


This Blog – PRESCAN


Microsoft.com – Upgrade Considerations for Customized Sites


Technet – Determine How to Handle Customizations

Stress Testing Microsoft Office SharePoint Server 2007/Windows SharePoint Services 3.0

TechReady5 concluded on Friday and I’m finally returning to work after several customer sessions that immediately followed and one question was shared between the two – what do you recommend or what are you using to monitor performance and how do you determine load and stress when architecting a SharePoint Products and Technologies infrastructure?  The answer is, there are a variety of tools to stress test your Microsoft SharePoint Products and Technologies deployment; let’s cover some them:


SPSiteBuilder

SPSiteBuilder was originally a component of the SharePoint Utility Suite. As many of you know there are no future plans to redevelop this application set and SPSiteBuilder packaged with the SharePoint Utility Suite won’t work with Microsoft Office SharePoint Server 2007/Windows SharePoint Services 3.0. The good news is that making the application compatible with Microsoft Office SharePoint Server 2007/Windows SharePoint Services 3.0 does not require a great deal of effort or a large investment of time.

Let’s step through fixing the code to support Microsoft Office SharePoint Server 2007/Windows SharePoint Services 3.0:

First locate the line:


globalAdmin = new SPGlobalAdmin();
virtualServer = globalAdmin.OpenVirtualServer(uri);


SPGlobalAdmin is maintained for backward comparability; replace the globalAdmin variable with Farm and the SPGlobalAdmin() method with new SPFarm().


Farm = new SPFarm();


Now we need to replace the OpenVirtualServer() method since it is obsolete and use the Lookup() method of the WebApplication object.


WebApp = WebApplication.Lookup(uri);


Now locate the line:


if (!virtualServer.Config.Prefixes.Contains(strPrefix))
virtualServer.Config.Prefixes.Add(strPrefix, Microsoft.SharePoint.Administration.SPPrefixType.WildcardInclusion);


This enumeration member is obsolete in Windows SharePoint Services 3.0 because it is no longer necessary to tell Windows SharePoint Services which URL paths should not be handled by Windows SharePoint Services. So we should replace it with the WebApp variable we specified earlier.




if (!webApp.Prefixes.Contains(strPrefix))
webApp.Prefixes.Add(strPrefix, Microsoft.SharePoint.Administration.SPPrefixType.WildcardInclusion);
}



Finally to complete the retrofit we need to replace all instances of globalAdmin with Farm and all instances of virtualServer with WebApp and ensure we are referencing Microsoft.SharePoint.dll from a Windows SharePoint Services 3.0 build and recompile.


MOSSDW.EXE


MOSSDW.EXE is a performance testing tool that provides data population/stress testing capabilities for Microsoft Office SharePoint Server 2007.  If your plans include only data population, consider retrofitting SPSiteBuilder to support Microsoft Office SharePoint Server 2007/Windows SharePoint Services 3.0 as described above.


WSSDW.EXE


WSSDW.EXE is a performance testing tool that populates data for testing deployments of Windows SharePoint Services 3.0 (see also SPSiteBuilder and MOSSDW.EXE).


Excel Services Ocracoke Performance Testing Sample Scripts


Excel Services Ocracoke Performance Testing Sample Scripts are performance testing scripts used in conjunction with MOSSDW.EXE that provide data population/stress testing capabilities for Excel Services.


Internet Information Services (ISS) 6.0 Resource Kit Tools


The IIS 6.0 Resource Kit Tools can help you administer, secure, and manage IIS; using the Web Capacity Analysis Tool Version 5.2 you can stress test your application.


IIS Diagnostic Toolkit


The IIS Diagnostic Toolkit is a compiled set of tools aimed at reducing the overall time to resolve problems with Internet Information Services (IIS) products.  Use Debug Diagnostics 1.0 to run diagnostic tests for web servers hosting SharePoint Products and Technologies.


Web Application Stress Tool


The Web Application Stress Tool is designed to realistically simulate multiple browsers requesting pages from a web site. You can use this tool to gather performance and stability information about your web application. This tool simulates a large number of requests with a relatively small number of client machines.


SQLIO.EXE Disk Subsystem Benchmark Tool

The I/O system is important to the performance of SQL Server. When configuring a new server for SQL Server or when adding or modifying the disk configuration of an existing system, it is good practice to determine the capacity of the I/O subsystem prior to deploying SQL Server.  SQLIO.EXE can be used to determine the I/O capacity of a given configuration.  An example of what we may see on a typical day during peak usage on a x64 A/P cluster hosting 100+ databases and supports 8 unique server farms, each with a local Shared Service Provider is:


IOs/sec:  7808.43

MBs/sec:    15.25

sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file c:testfile.dat with 2 threads (0-1) using mask 0x0 (0)
2 threads writing for 360 secs to file c:testfile.dat
using 8KB sequential IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 100 MB for file: c:testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 3434.19
MBs/sec: 26.82
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 4
Max_Latency(ms): 282


SQLIOsim

SQLIOsim simulates the I/O patterns of Microsoft SQL Server 2005, of SQL Server 2000, and of SQL Server 7.0. The I/O patterns of these versions of SQL Server resemble one another.  You can use SQLIOsim to simulate read, write, checkpoint, backup, sort, and read-ahead activities for Microsoft SQL Server 2005  (see also SQLIO.EXE Disk Subsystem Benchmark Tool).

Performance/System Monitor

This is a standard component of the Windows operating system. You can use it to monitor performance objects and counters and instances of various hardware and software components.

Joel Oleson has a good list of counters for measuring Web front-end and backend performance – http://blogs.msdn.com/joelo/archive/2007/01/16/good-list-of-performance-counters.aspx.

NOTE At minimum consider monitoring PhysicalDiskDisk Transfers per second or LogicalDiskDisk Transfers per second measuring the amount of IO reads and writes per second to the database drives (see illustration).


Microsoft Operations Manager (MOM)


You can install MOM agents on individual servers that collect data and send it a centralized MOM server. The data is stored in the MOM database, which can be a dedicated SQL Server or the Microsoft SQL Server 2000 Desktop Engine (MSDE) version of Microsoft SQL Server. MOM is suitable for collecting large amounts of data over a long period of time.   MOM packs to consider for a typical SharePoint Products and Technologies deployment include:



Microsoft Operations Manager Packs


Microsoft Windows SharePoint Services
Microsoft SharePoint Portal Server 2003
Microsoft SQL Server 2000 & 2005
Microsoft Internet Information Server 6
Microsoft Cluster Service
Microsoft Windows 2003 Server


Web Site Monitoring


Web Sites and Services MP


Management Pack Catalog – http://www.microsoft.com/technet/prodtechnol/mom/catalog/catalog.aspx?vs=2005.


Stress tools such as Application Center Test (ACT)


You can use tools such as ACT to simulate clients and collect data during the duration of a test.


Network Monitor (NetMon)


You use NetMon to monitor the network traffic. You can use it to capture packets sent between client and server computers. It provides valuable timing information as well as packet size, network utilization, addressing, and routing information and many other statistics that you can use to analyze system performance.


SQL Profiler


Identify slow and inefficient queries, deadlocks, timeouts, recompilations, errors, and exceptions for any database interactions.  For example, one measure we implement is identifying queries exceeding two (2) seconds excluding search.  Profiling can be used when reports of latency are received in conjunction with monitoring the web front-end components of the server farm.


SQL Query Analyzer


This tool is also installed with SQL Server. You can use it to analyze the execution plans for SQL queries and stored procedures. This is mostly used in conjunction with the SQL Profiler.


SQLDiag


Collects valuable information about the configuration of the computer running SQL Server, the operating system, and the information that is reported to the SQL Server error logs.


 


SharePoint Products and Technologies Performance and Capacity Planning Resources


Plan for availability (Windows SharePoint Services)


Plan for availability (Microsoft Office SharePoint Server 2007)


Plan for performance and capacity (Microsoft Office SharePoint Server 2007)


Determining the hardware requirements for a single farm (Microsoft Office SharePoint Server 2007)


Performance and capacity planning (Windows SharePoint Services)


Performance and capacity planning factors (Microsoft Office SharePoint Server 2007)


Estimate performance and capacity requirements for Windows SharePoint Services collaboration environments (Microsoft Office SharePoint Server 2007)


Microsoft Office SharePoint Server 2007 on HP ProLiant Servers – Performance Summary


White Paper: Intel Performance Testing of Windows SharePoint Services

SQL Server 2005 Database Mirroring and Windows SharePoint Services 3.0/Microsoft Office SharePoint Server 2007 – Part 3 (Failover)

Database mirroring is increasing in popularity and becoming an integral part of high availability and disaster recovery solutions within the SharePoint Products and Technologies arena. I’ve spent much of the last few months building labs, testing scenarios, impacts on platforms whether it be WSS or MOSS.  This post is the third and final on my series [SQL Server 2005 Database Mirroring and Windows SharePoint Services/Microsoft Office SharePoint Server 2007].

We know with asynchronous mirroring we can automatically manage SQL Server 2005 failover by introducing the witness role in our server farms, the most challenging question to answer is how to manage web front-end failover.  In this post I will outline several possible solutions for managing SharePoint Products and Technologies failover in a mirrored database architecture.

Solution # 1 Network Load Balancing

In a high-availability environment, you can create a NLB cluster to easily route client requests from the original principal server to a promoted mirror server without having to update each client directly. NLB setup and configuration are fairly painless; however, it comes with the requirement that each server be on the same subnet and the active mirror remain in a suspended state in the load balancing rotation.  With this in mind you should carefully consider the limitations when using native NLB, for example you will not be able to geo-cluster in most circumstances. 

Solution # 2 Manual Failover

Manual failover requires the least in respect to infrastructure and configuration; however, has the highest operational costs and without proper management and monitoring can present the largest client impact in the event the of a failover. The steps required to instantiate a manual failover of SharePoint Products and Technologies are described in the first part of this series.

Solution #3 Custom Solutions (Example)

Custom solutions can be implemented monitoring the state of the principal and mirror servers redirecting the application calling the original principal server to the new principal server – typically implemented through a Windows service.

For example, using an aliasing scheme where SharePoint Products and Technologies accesses the SQL database server through an alternate name;  SharePoint Products and Technologies accessing the alternate name is redirected to the proper physical name of the principal server. Since SharePoint Products and Technologies is only aware of the alternate name and never accesses the physical name of either node it does not need to be failover aware.

From a logical perspective in the example solution proposed SharePoint Products and Technologies would make the request to the SQL alias, for example contoso, requests for contoso are intercepted and modified to reference the active principal node referenced in the web front-end server Registry as contoso1.  In a failover scenario, the Windows service would detect the principal role is on the new principal server, previously the mirror server, and replace the web front-end server Registry entry for contoso1 with the new principal physical server contoso2.

Just Published – August 2007

Building Simple Custom Approval Workflows with InfoPath 2007 Forms – http://msdn2.microsoft.com/en-us/library/bb629921.aspx


Creating and Editing Custom Document Information Panels from Office SharePoint Server 2007 – http://msdn2.microsoft.com/en-us/library/bb684925.aspx


Integrating Siebel CRM with Office SharePoint Server 2007 – http://msdn2.microsoft.com/en-us/library/bb613692.aspx