SharePoint

Search Scale and Resiliency Improvements in SharePoint Server 2016 IT Preview

Improvements in performance and scale in SharePoint Server 2016 IT Preview search now allow search to scale up to 500 million items, an increase of 250 million items SharePoint Server 2013 [http://technet.microsoft.com/en-us/library/cc262787.aspx#Search%5D based on 10 million items per index partition.  In SharePoint Server 2013 and SharePoint Server 2016 IT Preview each index partition contains a subset of the whole search index. If the number of indexed items is high in relation to how much memory the server has, affects the query response time negatively.  SharePoint Server 2016 IT Preview has been improved to support 20 million items per index partition, where 500 million items are based on 25x index partitions.

In addition to improving the search scale boundaries as described above, SharePoint Server 2016 IT Preview makes a number of general performance optimizations in the query path to reduce overall latency, implements changes in merge scheduling reducing peak disk and memory usage from ~210% steady state consumption to ~140% enabling larger index partitions while avoiding a large resource buffer.

While improvements in scale and performance allow substantially larger index partitions, as the number of partitions increase, so does the need for resiliency.  SharePoint Server 2016 IT Preview implements new failover logic for serving queries between index replicas which improves the ability to respond to overload to include immediate recovery when the overload is removed.

To learn more about SharePoint Server 2016 IT Preview see also https://technet.microsoft.com/en-us/library/cc303422(v=office.16).aspx.

Standard
SharePoint

Navigating List View Thresholds in SharePoint Server 2016 IT Preview

Overview

In SharePoint Server 2013 the List View Threshold specified the maximum number of list or library items that a database operation, such as a query, can process at the same time outside the daily time window set by the administrator during which queries are unrestricted. In SharePoint 2013 the List View Threshold is set to 5,000 or 20,000 for users and auditors respectively. Typically a users’ initial experience with List View Threshold is when it has been exceeded, the resultant error: The number of items in this list exceeds the list view threshold, which is 5000 items” as documented at https://support.microsoft.com/en-us/kb/2759051/.

The List View Threshold controls Front End and Backend interaction to prevent disruptions in service to other users whose Site Collections are contained in the same Content Database as the executing query which prompts the threshold.  For additional information see also List and library limits at https://technet.microsoft.com/en-us/library/cc262787.aspx.

The List View Threshold and subsequent boundary of 5,000 was designed to mitigate lock escalation within SQL Server, I.e. lock escalation occurs when a single Transact-SQL statement acquires at least 5,000 locks on a single non-partitioned table or index – effectively a row lock escalates to a table lock, blocking all subsequent requests – or otherwise converts many fine-grain locks into fewer coarse-grain locks preventing other operations from completing. Each view request within a List is effectively the result of a query against one or more database tables.

In SharePoint Server 2016 IT Preview databases are no longer subject to lock escalation; however, a List View Threshold is enabled (configurable) on a per Web Application basis.  In addition SharePoint Server 2016 IT Preview provides several new capabilities designed to mitigate potential performance degradation related to queries which may impact performance for other users.

List View Auto-Indexing

In SharePoint Server 2016 IT Preview a Timer Job (Large list column index management) examines the views in Lists that exceed 2,500 items. In the event a view definition would benefit from a column index, one is programmatically created. For example, if a view includes a filter for “WHERE A=1 AND B=2”, the Timer Job will create an index on either column A or column B. The specific choice depends on the other view definitions in the list, with the goal of minimizing the number of indexes created.

ColumnIndexManagement

List View Auto-Indexing is applicable to Lists which are enabled for automatic management of indices which is the default configuration for Lists in SharePoint Server 2016 IT Preview. 

AutomaticIndexManagement

Automatic Index Management allows the Timer Job to maintain column indices on Lists to provide optimal query performance within views associated with the List.

Increased Effective Threshold

The List View Threshold is not only applicable to List views, but also includes any database operation which potentially involves scanning of a large number of rows (non-List view operations).

An example of a non-List view operation would be a scenario in which a non-privileged user is unable to create a column index in a List which exceeds 5,000 items as the result of that operation would require both the reading of more than 5,000 items from one database table and writing more than 5,000 items into another, in this case the NameValuePair index table.  Under these case SharePoint Server 2016 IT Preview improves support for a subset of these operations such as creating or removing a column index, permissions inheritance, and deleting a column.

Query Engine Improvements

In addition improvements have been made to the query engine on large Lists to enable it to better anticipate when a query should be throttled by recognizing specific query patterns that rely on built-in indexes such as Item Id, Document GUID, File Path, etc. and allow those operations.

Document Library Views

Improvements in out of the box Document Library views have been improved to address throttling related to sort ordering.  For example, the default view on a Document Library is to sort folders before files.  In a large List scenario, this can result in the view to the throttled as it’s necessary for SQL to scan the entire List to find all of the folders in order to satisfy the sort criteria.  In SharePoint Server 2016 IT Preview the folder first sort criteria is omitted in the event it would result in throttling of the query. 

To learn more about SharePoint Server 2016 IT Preview see also https://technet.microsoft.com/en-us/library/cc262787.aspx.

Standard
Administration, SharePoint

Quick Steps to Diagnose Performance Issues in SharePoint 2010

Make the most of the Developer Dashboard

Monitoring Latency and SQL Server Round Trips

Among the information provided through the Developer Dashboard is information about page latency and database queries surfaced under Execution Timeout in the user interface in the Web Server section. Using the information provided in this field you can determine whether or not a recently introduced feature or particular page is exceeding acceptable performance thresholds in your environment. As an example, general performance guidance dictates a single round trip for a typical page in addition to a single round trip per Web Part on the page. Wiki and layout pages will result in higher overall roundtrips when compared to a typical page at 2 to 2-3 roundtrips respectively. The Web Part round trip metric applies equally at 1 round trip per Web Part.

Latencies exceeding .5 seconds should also be considered suspect when your investigating possible performance issues. The most useful information when attempting the diagnose higher than expected latency is located under the Show or hide additional tracing information link.

Developers seeking to monitor the performance characteristics and resource usage of their custom code can wrap their code with a SPMonitoredScope.

Custom pages can also leverage the developer dashboard through simply adding the required controls.  The first control renders the launch icon to enable the developer dashboard OnDemand.  The second control specified where on the page the Developer Dashboard will render and the final control is responsible for capturing all of the operations occurring during the rendering of the page, with that said it is important to ensure that this control is at the top of the page.  To learn more about the Developer Dashboard see also Welcome to the Developer Dashboard.

  <Sharepoint:DeveloperDashboardLauncher
        ID="DeveloperDashboardLauncher"
        NavigateUrl="BLOCKED SCRIPTToggleDeveloperDashboard()"
        runat="server"
        ImageUrl="/_layouts/images/fgimg.png"
        Text="<%$Resources:wss,multipages_launchdevdashalt_text%>"
        OffsetX=0
        OffsetY=78
        Height=16
        Width=16 />

<div id="DeveloperDashboard" class="ms-developerdashboard" />

<SharePoint:DeveloperDashboard runat="server" />

Make the most of the SharePoint Health Analyzer

Identify Server Configuration Issues

As you investigate possible performance issues in your environment a quick and simple check to determine the potential cause of issues is to check the SharePoint Health Analyzer through SharePoint 2010 Central Administration to determine whether or not environmental or configuration issues are causing the problem.

Make the most of ULS

Error Handling

Correlation Ids are perhaps the single most improved feature in SharePoint 2010. Correlation Ids allow you to quickly isolate additional information about issues occurring in your environment in the ULS logs.

Additional Tools

Visual Round Trip Analyzer

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=119f3477-dced-41e3-a0e7-d8b5cae893a3&DisplayLang=en

Bottleneck Detection Counters

http://msdn.microsoft.com/en-us/library/ms804032.aspx

Standard
Uncategorized

Planning a 100,000+ My Site personal site deployment? Notes…

This topic has frequented my Inbox over the past months, planning a My Site personal site deployment for 100,000+ users.  In this post we’ll examine the issues and potential solutions.


First it is important to understand that while a My Site personal site is a site collection in basic form, it’s instantiation via a browser request or managed code results in significantly greater overhead than a traditional collaboration-type site collection.  The majority of overhead is presented through the numerous data building operations that occur both on the private and public pages associated with the template, private.aspx and person.aspx respectively.


The My Site private page (private.aspx) is considered to be a private view that hosts information specific to you (or the site collection owner).  This information is presented in part based on your membership in a specific audience in many configurations.  The private page also serves as a host to your documents, alerts and subsequent alert results.  Private.aspx also provides the capability to view your Inbox, calendar, and other Exchange views.


The My Site personal site public page is intended to present information to be shared with other users and contains information such as personal and professional interests, shared links and documents.  The public page also provides out of the box functionality that can be used to determine organizational similarities between the user requesting your profile and themselves (also known as the Organization Hierarchy Web Part).  Other comparative Web Parts include In Common with You and Memberships.


The result of these two unique views presented in a My Site personal site is approximately a 50% more costly public view when compared to the private view as a result of resource intensive operations required to build and present the profile information for a specific user requesting the My Site personal site public page.


The majority of the information that is presented on the public page is derived from the profile database, since that content is isolated to a single database there are no options to spread the load across multiple database servers.  The database is the smallest unit of file system representation for SharePoint and cannot be logically partitioned (clustered indexing strategy comes in play here); however, you can physically partition the database and stripe the data files across dedicated physical disks to gain parallelism which will help with data access I/O.


Topology


A distributed topology provides the most efficient mechanism to support a large number of My Site personal sites and is commonly comprised of two or more independent server farms supporting the associated Web applications from unique geographic regions.  For example, server farm A in Seattle hosts the My Site host Foo, and an audience defined in the Shared Services Provider for domain users in the North America region is configured to use the My Site host Foo as their trusted My Site host (Figure 1) whereas server farm B in Dublin hosts the My Site host Bar, and an audience (Figure 2) defined in the Shared Services Provider for domain users in the EMEA region is configured to use the My Site host Bar as their trusted My Site host.  This topology generally requires a parent Shared Services Provider to which server farms A and B are consumers, a local Shared Services Provider can be configured; however, certain profile replication mechanisms should be in place to keep the SSP’s synchronized.  In either scenario when user B who is a member of an EMEA based domain users group, e.g. EuropeDomain Users requests Foo, the audience membership is read and subsequently the user is redirected to Bar, this process conversely applies to user C who is a member of a North America based domain users group, e.g. AmericaDomains Users.


clip_image002


Figure 1 Trusted My Site host locations (Shared Services Provider administration)


clip_image002[5]


Figure 2 Configure Trusted My Site host location and audience


*An important consideration is the requirement to have two or more unique URI’s to supported the distributed topology, for example http://mysite in North America and http://mysiteemea in EMEA.


Planning


Consider RAID 0+1 or 1+0 sets to supported the Shared Services Provider database and define a subsequent scale-up strategy.


Consider running profile synchronization operations during non-core business hours.


Consider removing costly Web Parts and determine their overall business value, this may include Organizational Hierarchy, In Common with You, and Memberships.


Consider a distributed topology where possible.


Remember the sizeable fraction of visits to a My Site personal site are from that of the site collection owner themselves.


RPS – Throughput can decline as the number of site collections in a given content database increase.


Consider Kerberos where possible to improve authentication performance and reduce overhead.


Plan for capacity and ensure a proper governance plan is endorsed and implemented.


Resources


About My Site
http://office.microsoft.com/en-us/sharepointserver/HA011605561033.aspx


About Audiences
http://office.microsoft.com/en-us/sharepointserver/HA011603031033.aspx


Plan for software boundaries (Office SharePoint Server)
http://technet.microsoft.com/en-us/library/cc262787.aspx


Planning and Monitoring SQL Server Storage for SharePoint: Performance Recommendations and Best Practices
http://go.microsoft.com/fwlink/?LinkID=105623&clcid=0x409

Standard
Uncategorized

Bill’s 5-min. SharePoint Performance Recommendations

Performance recommendations and guidance is something I receive comment and question on quite frequently, typically in hallway conversation or in passing at conferences – so in the spirit of those occurrences I’ve decided to compile a quick list of SharePoint performance recommendations that can be conveyed verbally in five minutes or less.


Governance


Do limit the number of site collections/content database, I’m adamantly opposed to the “airline booking model” and much prefer what I like to refer to as the “accounting model” of database management, for example, if you know your maximum allowable site collection quota will be 5GB, and would like to keep your content databases at 100GB, logically you can host no more than 20 site collections per database and while this can result in a large number of content databases, you avoid site collection proliferation…with this model you should also take into account growth and set aside 5-10% to support schema changes, etc.  *Remember to size your content databases to the desired size as they are created – SharePoint Products and Technologies will provision the content database at xMB and configure growth to occur incrementally at 1MB chunks.  A smaller number of site collections in a content database not only benefits efficiency of operations, but can also mitigate the exposure of any locking that may occur within that database to a small subset of users as opposed to a large population.


Caching and Compression


Do consider and encourage site output caching, BLOB caching, and/or HTTP compression where possible – do be sure to closely monitor processor utilization where using HTTP compression.


Windows Server 2008


Do consider Windows Server 2008, the Next Generation TCP/IP Stack brings a number of benefits to bolster performance including receive window auto-tuning, compound TCP, improved routing path detection and recovery, and more.  There is a very informative whitepaper available “Enhanced Network Performance with Microsoft Windows Vista and Windows Server 2008“.  In regards to Windows Server 2008 SharePoint Products and Technologies, check back frequently as I observe our own results at Microsoft.


64-bit


Do consider 64-bit, the benefits here are too many to detail, but consider larger data processing chunks, larger address space, etc.


Do consider proactively addressing garbage collection on 64-bit machines, custom code and other external factors can potentially litter the address space with unintended assemblies and permanent memory allocations, while most is addressed with “managed memory” you may still experience conditions in which there is a Web server response delay until the buffer can be moved in memory.  Depending on the nature of the code you should consider monitoring .NET CLR Exceptions, Memory, and Loading Performance.  So you are faced with the decision to either proactively manage consumption through scheduled recycling, or monitor garbage collection activity dynamically and recycle when garbage collection activity has exceeded a pre-defined threshold – of these decisions the former implies configuring an arbitrary memory threshold and the latter understanding your individual requirements based on Web server configuration (I.e. resident services, hardware, load, etc.) which is the preferred methodology since you in effect will maximize the available memory while ensuring your Web servers remain responsive and mitigating availability issues as the result of too frequent a recycle in the case of configuring or designating a threshold.


Wide Area Networks


Consider WAN acceleration to manage traffic generated by satellite or remote offices and/or data replication scenarios.


Adjust IIS timeout settings to accommodate large file uploads by remote users on slow links, VPN, etc.


Storage Design and Database Architecture


Storage design and database architecture are other potential performance bottlenecks, carefully consider database distribution – where clustering, databases should be distributed across two or more instances – and with the underlying storage, provide as many spindles as possible to your data LUNs.


Authentication


Do consider Kerberos authentication where possible, significantly reducing the number of round trips per page versus NTLM.


More reading…


About Performance and Capacity Planning (Windows SharePoint Services)


Tools for Performance and Capacity Planning (Windows SharePoint Services)


Estimate Performance and Capacity Planning Requirements for Windows SharePoint Services Collaboration Environments


Posts Tagged “Performance” on this Blog

Standard
Uncategorized

Just Published – Whitepaper: Database Maintenance for Microsoft SharePoint Products and Technologies

I’m pleased to announce the general availability of my most recent whitepaper, “Database Maintenance for Microsoft SharePoint Products and Technologies”.  This whitepaper describes the recommended maintenance strategies for the databases that host content and configuration settings for SharePoint Products and Technologies.


Read more…

Standard