Search Scale and Resiliency Improvements in SharePoint Server 2016 IT Preview

Improvements in performance and scale in SharePoint Server 2016 IT Preview search now allow search to scale up to 500 million items, an increase of 250 million items SharePoint Server 2013 [ based on 10 million items per index partition.  In SharePoint Server 2013 and SharePoint Server 2016 IT Preview each index partition contains a subset of the whole search index. If the number of indexed items is high in relation to how much memory the server has, affects the query response time negatively.  SharePoint Server 2016 IT Preview has been improved to support 20 million items per index partition, where 500 million items are based on 25x index partitions.

In addition to improving the search scale boundaries as described above, SharePoint Server 2016 IT Preview makes a number of general performance optimizations in the query path to reduce overall latency, implements changes in merge scheduling reducing peak disk and memory usage from ~210% steady state consumption to ~140% enabling larger index partitions while avoiding a large resource buffer.

While improvements in scale and performance allow substantially larger index partitions, as the number of partitions increase, so does the need for resiliency.  SharePoint Server 2016 IT Preview implements new failover logic for serving queries between index replicas which improves the ability to respond to overload to include immediate recovery when the overload is removed.

To learn more about SharePoint Server 2016 IT Preview see also

Navigating List View Thresholds in SharePoint Server 2016 IT Preview


In SharePoint Server 2013 the List View Threshold specified the maximum number of list or library items that a database operation, such as a query, can process at the same time outside the daily time window set by the administrator during which queries are unrestricted. In SharePoint 2013 the List View Threshold is set to 5,000 or 20,000 for users and auditors respectively. Typically a users’ initial experience with List View Threshold is when it has been exceeded, the resultant error: The number of items in this list exceeds the list view threshold, which is 5000 items” as documented at

The List View Threshold controls Front End and Backend interaction to prevent disruptions in service to other users whose Site Collections are contained in the same Content Database as the executing query which prompts the threshold.  For additional information see also List and library limits at

The List View Threshold and subsequent boundary of 5,000 was designed to mitigate lock escalation within SQL Server, I.e. lock escalation occurs when a single Transact-SQL statement acquires at least 5,000 locks on a single non-partitioned table or index – effectively a row lock escalates to a table lock, blocking all subsequent requests – or otherwise converts many fine-grain locks into fewer coarse-grain locks preventing other operations from completing. Each view request within a List is effectively the result of a query against one or more database tables.

In SharePoint Server 2016 IT Preview databases are no longer subject to lock escalation; however, a List View Threshold is enabled (configurable) on a per Web Application basis.  In addition SharePoint Server 2016 IT Preview provides several new capabilities designed to mitigate potential performance degradation related to queries which may impact performance for other users.

List View Auto-Indexing

In SharePoint Server 2016 IT Preview a Timer Job (Large list column index management) examines the views in Lists that exceed 2,500 items. In the event a view definition would benefit from a column index, one is programmatically created. For example, if a view includes a filter for “WHERE A=1 AND B=2”, the Timer Job will create an index on either column A or column B. The specific choice depends on the other view definitions in the list, with the goal of minimizing the number of indexes created.


List View Auto-Indexing is applicable to Lists which are enabled for automatic management of indices which is the default configuration for Lists in SharePoint Server 2016 IT Preview. 


Automatic Index Management allows the Timer Job to maintain column indices on Lists to provide optimal query performance within views associated with the List.

Increased Effective Threshold

The List View Threshold is not only applicable to List views, but also includes any database operation which potentially involves scanning of a large number of rows (non-List view operations).

An example of a non-List view operation would be a scenario in which a non-privileged user is unable to create a column index in a List which exceeds 5,000 items as the result of that operation would require both the reading of more than 5,000 items from one database table and writing more than 5,000 items into another, in this case the NameValuePair index table.  Under these case SharePoint Server 2016 IT Preview improves support for a subset of these operations such as creating or removing a column index, permissions inheritance, and deleting a column.

Query Engine Improvements

In addition improvements have been made to the query engine on large Lists to enable it to better anticipate when a query should be throttled by recognizing specific query patterns that rely on built-in indexes such as Item Id, Document GUID, File Path, etc. and allow those operations.

Document Library Views

Improvements in out of the box Document Library views have been improved to address throttling related to sort ordering.  For example, the default view on a Document Library is to sort folders before files.  In a large List scenario, this can result in the view to the throttled as it’s necessary for SQL to scan the entire List to find all of the folders in order to satisfy the sort criteria.  In SharePoint Server 2016 IT Preview the folder first sort criteria is omitted in the event it would result in throttling of the query. 

To learn more about SharePoint Server 2016 IT Preview see also

Quick Steps to Diagnose Performance Issues in SharePoint 2010

Make the most of the Developer Dashboard

Monitoring Latency and SQL Server Round Trips

Among the information provided through the Developer Dashboard is information about page latency and database queries surfaced under Execution Timeout in the user interface in the Web Server section. Using the information provided in this field you can determine whether or not a recently introduced feature or particular page is exceeding acceptable performance thresholds in your environment. As an example, general performance guidance dictates a single round trip for a typical page in addition to a single round trip per Web Part on the page. Wiki and layout pages will result in higher overall roundtrips when compared to a typical page at 2 to 2-3 roundtrips respectively. The Web Part round trip metric applies equally at 1 round trip per Web Part.

Latencies exceeding .5 seconds should also be considered suspect when your investigating possible performance issues. The most useful information when attempting the diagnose higher than expected latency is located under the Show or hide additional tracing information link.

Developers seeking to monitor the performance characteristics and resource usage of their custom code can wrap their code with a SPMonitoredScope.

Custom pages can also leverage the developer dashboard through simply adding the required controls.  The first control renders the launch icon to enable the developer dashboard OnDemand.  The second control specified where on the page the Developer Dashboard will render and the final control is responsible for capturing all of the operations occurring during the rendering of the page, with that said it is important to ensure that this control is at the top of the page.  To learn more about the Developer Dashboard see also Welcome to the Developer Dashboard.

        NavigateUrl="BLOCKED SCRIPTToggleDeveloperDashboard()"
        Width=16 />

<div id="DeveloperDashboard" class="ms-developerdashboard" />

<SharePoint:DeveloperDashboard runat="server" />

Make the most of the SharePoint Health Analyzer

Identify Server Configuration Issues

As you investigate possible performance issues in your environment a quick and simple check to determine the potential cause of issues is to check the SharePoint Health Analyzer through SharePoint 2010 Central Administration to determine whether or not environmental or configuration issues are causing the problem.

Make the most of ULS

Error Handling

Correlation Ids are perhaps the single most improved feature in SharePoint 2010. Correlation Ids allow you to quickly isolate additional information about issues occurring in your environment in the ULS logs.

Additional Tools

Visual Round Trip Analyzer

Bottleneck Detection Counters

Planning a 100,000+ My Site personal site deployment? Notes…

This topic has frequented my Inbox over the past months, planning a My Site personal site deployment for 100,000+ users.  In this post we’ll examine the issues and potential solutions.

First it is important to understand that while a My Site personal site is a site collection in basic form, it’s instantiation via a browser request or managed code results in significantly greater overhead than a traditional collaboration-type site collection.  The majority of overhead is presented through the numerous data building operations that occur both on the private and public pages associated with the template, private.aspx and person.aspx respectively.

The My Site private page (private.aspx) is considered to be a private view that hosts information specific to you (or the site collection owner).  This information is presented in part based on your membership in a specific audience in many configurations.  The private page also serves as a host to your documents, alerts and subsequent alert results.  Private.aspx also provides the capability to view your Inbox, calendar, and other Exchange views.

The My Site personal site public page is intended to present information to be shared with other users and contains information such as personal and professional interests, shared links and documents.  The public page also provides out of the box functionality that can be used to determine organizational similarities between the user requesting your profile and themselves (also known as the Organization Hierarchy Web Part).  Other comparative Web Parts include In Common with You and Memberships.

The result of these two unique views presented in a My Site personal site is approximately a 50% more costly public view when compared to the private view as a result of resource intensive operations required to build and present the profile information for a specific user requesting the My Site personal site public page.

The majority of the information that is presented on the public page is derived from the profile database, since that content is isolated to a single database there are no options to spread the load across multiple database servers.  The database is the smallest unit of file system representation for SharePoint and cannot be logically partitioned (clustered indexing strategy comes in play here); however, you can physically partition the database and stripe the data files across dedicated physical disks to gain parallelism which will help with data access I/O.


A distributed topology provides the most efficient mechanism to support a large number of My Site personal sites and is commonly comprised of two or more independent server farms supporting the associated Web applications from unique geographic regions.  For example, server farm A in Seattle hosts the My Site host Foo, and an audience defined in the Shared Services Provider for domain users in the North America region is configured to use the My Site host Foo as their trusted My Site host (Figure 1) whereas server farm B in Dublin hosts the My Site host Bar, and an audience (Figure 2) defined in the Shared Services Provider for domain users in the EMEA region is configured to use the My Site host Bar as their trusted My Site host.  This topology generally requires a parent Shared Services Provider to which server farms A and B are consumers, a local Shared Services Provider can be configured; however, certain profile replication mechanisms should be in place to keep the SSP’s synchronized.  In either scenario when user B who is a member of an EMEA based domain users group, e.g. EuropeDomain Users requests Foo, the audience membership is read and subsequently the user is redirected to Bar, this process conversely applies to user C who is a member of a North America based domain users group, e.g. AmericaDomains Users.


Figure 1 Trusted My Site host locations (Shared Services Provider administration)


Figure 2 Configure Trusted My Site host location and audience

*An important consideration is the requirement to have two or more unique URI’s to supported the distributed topology, for example http://mysite in North America and http://mysiteemea in EMEA.


Consider RAID 0+1 or 1+0 sets to supported the Shared Services Provider database and define a subsequent scale-up strategy.

Consider running profile synchronization operations during non-core business hours.

Consider removing costly Web Parts and determine their overall business value, this may include Organizational Hierarchy, In Common with You, and Memberships.

Consider a distributed topology where possible.

Remember the sizeable fraction of visits to a My Site personal site are from that of the site collection owner themselves.

RPS – Throughput can decline as the number of site collections in a given content database increase.

Consider Kerberos where possible to improve authentication performance and reduce overhead.

Plan for capacity and ensure a proper governance plan is endorsed and implemented.


About My Site

About Audiences

Plan for software boundaries (Office SharePoint Server)

Planning and Monitoring SQL Server Storage for SharePoint: Performance Recommendations and Best Practices