Search Scale and Resiliency Improvements in SharePoint Server 2016 IT Preview

Improvements in performance and scale in SharePoint Server 2016 IT Preview search now allow search to scale up to 500 million items, an increase of 250 million items SharePoint Server 2013 [http://technet.microsoft.com/en-us/library/cc262787.aspx#Search%5D based on 10 million items per index partition.  In SharePoint Server 2013 and SharePoint Server 2016 IT Preview each index partition contains a subset of the whole search index. If the number of indexed items is high in relation to how much memory the server has, affects the query response time negatively.  SharePoint Server 2016 IT Preview has been improved to support 20 million items per index partition, where 500 million items are based on 25x index partitions.

In addition to improving the search scale boundaries as described above, SharePoint Server 2016 IT Preview makes a number of general performance optimizations in the query path to reduce overall latency, implements changes in merge scheduling reducing peak disk and memory usage from ~210% steady state consumption to ~140% enabling larger index partitions while avoiding a large resource buffer.

While improvements in scale and performance allow substantially larger index partitions, as the number of partitions increase, so does the need for resiliency.  SharePoint Server 2016 IT Preview implements new failover logic for serving queries between index replicas which improves the ability to respond to overload to include immediate recovery when the overload is removed.

To learn more about SharePoint Server 2016 IT Preview see also https://technet.microsoft.com/en-us/library/cc303422(v=office.16).aspx.

SharePoint Server 2016 IT Preview Image in Windows Azure Platform Gallery

SharePoint Server 2016 IT Preview is now available in the Windows Azure Platform Gallery.  The image is based on the current Windows Server 2012 R2 Datacenter release and includes a complete installation of SharePoint Server 2016 IT Preview.

Getting Started

To get started with SharePoint Server 2016 IT Preview on Windows Azure navigate to the Windows Azure Portal (https://manage.windowsazure.com/) and sign-in with your credentials.

Select + New.

Select Compute, click Virtual Machine and then click From Gallery

On the Create Virtual Machine Dialog click Platform Images and then click SharePoint Server 2016 IT Preview.

MSAzure

Click -> Next.

Complete the remaining details to provision your virtual machine.

NOTE

Extra Large, A6, or A7 are the recommended virtual machine sizes recommended for use with SharePoint Server 2016 IT Preview.

The virtual machine contains a complete installation of SharePoint Server 2016 IT Preview and expires on February 23, 2016.  SharePoint Server 2016 IT Preview and prerequisites are preinstalled with links to SharePoint 2016 Central Administration, the SharePoint 2016 Management Shell, and SharePoint Products Configuration on the Start screen. SQL Server 2014 with Service Pack 1 or SQL Server 2016 CTP 2.2 is required to complete the configuration.

This image can be configured using the available roles in the SharePoint Products Configuration Wizard.

Resources

SharePoint Deployment on Windows Azure Virtual Machines
http://www.microsoft.com/en-us/download/details.aspx?id=34598

SharePoint 2013 on Windows Azure Infrastructure
http://msdn.microsoft.com/en-us/library/windowsazure/dn275958.aspx

Installing SharePoint 2013 on Windows Azure Infrastructure Services
http://msdn.microsoft.com/en-us/library/windowsazure/dn275959.aspx

Automate Windows Azure SharePoint Deployments
https://github.com/WindowsAzure/azure-sdk-tools-samples/wiki/Automated-Deployment-of-SharePoint-2013-with-Windows-Azure-PowerShell

SharePoint and Windows Azure Development Kit
http://www.microsoft.com/en-us/download/details.aspx?id=24398

Navigating List View Thresholds in SharePoint Server 2016 IT Preview

Overview

In SharePoint Server 2013 the List View Threshold specified the maximum number of list or library items that a database operation, such as a query, can process at the same time outside the daily time window set by the administrator during which queries are unrestricted. In SharePoint 2013 the List View Threshold is set to 5,000 or 20,000 for users and auditors respectively. Typically a users’ initial experience with List View Threshold is when it has been exceeded, the resultant error: The number of items in this list exceeds the list view threshold, which is 5000 items” as documented at https://support.microsoft.com/en-us/kb/2759051/.

The List View Threshold controls Front End and Backend interaction to prevent disruptions in service to other users whose Site Collections are contained in the same Content Database as the executing query which prompts the threshold.  For additional information see also List and library limits at https://technet.microsoft.com/en-us/library/cc262787.aspx.

The List View Threshold and subsequent boundary of 5,000 was designed to mitigate lock escalation within SQL Server, I.e. lock escalation occurs when a single Transact-SQL statement acquires at least 5,000 locks on a single non-partitioned table or index – effectively a row lock escalates to a table lock, blocking all subsequent requests – or otherwise converts many fine-grain locks into fewer coarse-grain locks preventing other operations from completing. Each view request within a List is effectively the result of a query against one or more database tables.

In SharePoint Server 2016 IT Preview databases are no longer subject to lock escalation; however, a List View Threshold is enabled (configurable) on a per Web Application basis.  In addition SharePoint Server 2016 IT Preview provides several new capabilities designed to mitigate potential performance degradation related to queries which may impact performance for other users.

List View Auto-Indexing

In SharePoint Server 2016 IT Preview a Timer Job (Large list column index management) examines the views in Lists that exceed 2,500 items. In the event a view definition would benefit from a column index, one is programmatically created. For example, if a view includes a filter for “WHERE A=1 AND B=2”, the Timer Job will create an index on either column A or column B. The specific choice depends on the other view definitions in the list, with the goal of minimizing the number of indexes created.

ColumnIndexManagement

List View Auto-Indexing is applicable to Lists which are enabled for automatic management of indices which is the default configuration for Lists in SharePoint Server 2016 IT Preview. 

AutomaticIndexManagement

Automatic Index Management allows the Timer Job to maintain column indices on Lists to provide optimal query performance within views associated with the List.

Increased Effective Threshold

The List View Threshold is not only applicable to List views, but also includes any database operation which potentially involves scanning of a large number of rows (non-List view operations).

An example of a non-List view operation would be a scenario in which a non-privileged user is unable to create a column index in a List which exceeds 5,000 items as the result of that operation would require both the reading of more than 5,000 items from one database table and writing more than 5,000 items into another, in this case the NameValuePair index table.  Under these case SharePoint Server 2016 IT Preview improves support for a subset of these operations such as creating or removing a column index, permissions inheritance, and deleting a column.

Query Engine Improvements

In addition improvements have been made to the query engine on large Lists to enable it to better anticipate when a query should be throttled by recognizing specific query patterns that rely on built-in indexes such as Item Id, Document GUID, File Path, etc. and allow those operations.

Document Library Views

Improvements in out of the box Document Library views have been improved to address throttling related to sort ordering.  For example, the default view on a Document Library is to sort folders before files.  In a large List scenario, this can result in the view to the throttled as it’s necessary for SQL to scan the entire List to find all of the folders in order to satisfy the sort criteria.  In SharePoint Server 2016 IT Preview the folder first sort criteria is omitted in the event it would result in throttling of the query. 

To learn more about SharePoint Server 2016 IT Preview see also https://technet.microsoft.com/en-us/library/cc262787.aspx.

Fast Site Collection Creation in SharePoint Server 2016 IT Preview

Fast Site Collection Creation is a new capability in SharePoint Server 2016 IT Preview that improves Site Collection creation performance by reducing Feature activation overhead. Similar to the approach associated with MinRole Site Collections that support Fast Site Collection Creation are Feature-optimized.

Fast Site Collection Creation is a mechanism designed to improve provision performance of Site Collections through instrumenting a copy operation using SPSite.Copy at the Content Database level.

The primary vehicle in addressing this scenario is the enablement of a Site Master or master copy of the Site Collection for an enabled Template that provides the source Site Collection when a request is made for creating a new Site Collection.

Enabling Fast Site Collection Creation is achieved using the new Enable-SPWebTemplateForSiteMasterWindows PowerShell.

To access the SharePoint Management Shell

  1. Start the SharePoint Management Shell.
    • For Windows Server 2012:
      • On the Start screen, click SharePoint Management Shell.

If SharePoint Management Shell is not on the Startscreen:

      • Right-click Computer, click All apps, and then click SharePoint Management Shell.

To enable Fast Site Collection Creation enter Enable-SPWebTemplateForSiteMaster -Template <TEMPLATENAME> -CompatibilityLevel 15at the prompt.

Once Fast Site Collection Creation is enabled a new Site Master will need to be created for the respective Template(s). To create a new Site Master enter New-SPSiteMaster -ContentDatabase <CONTENTDB> -Template <TEMPLATENAME> at the prompt.

Provisioning of new Site Collections using Fast Site Collection Creation is achieved through including the new parameter –CreateFromSiteMasterwith the New-SPSite Windows PowerShell CmdLet as shown in the example below:

New-SPSite http://www.contoso.com/sites/<SITE&gt; -Template <TEMPLATE> -ContentDatabase <CONTENTDB> -CompatibilityLevel 15 -CreateFromSiteMaster -OwnerAlias <OWNER>

Since Site Collections are now created by copying at Content Database level with Fast Site Collection Creation enabled, the Feature Activation that usually happens during regular Site Collection creation does not occur.

For Developers who have Feature-based customization using Site Collection provisioning logic, if there are features that perform special processing based on the current Site's information, these will be incorrect. To mitigate this scenario, Features should be marked to be activated post-copy, so their activation code executes after the Site is created.

To learn more about Windows PowerShell in SharePoint Server 2016 IT Preview see also https://technet.microsoft.com/EN-US/library/ee806878(v=office.16).aspx.

Sensitive Information Types in SharePoint Server 2016 IT Preview

One of the key improvements to eDiscovery in SharePoint Server 2016 IT Preview is the introduction on Sensitive Information Types to eDiscovery.

In SharePoint Server 2016 IT Preview Data Loss Prevention is now built into Enterprise Search. It allows you to search for sensitive content in your existing eDiscovery Center, keeping content in place and enabling you to search in real time.  SharePoint Server 2016 IT Preview provides a wide range of sensitive information types from different industry segments and geographies, such as credit card numbers, Social Security numbers (SSNs), bank account numbers, and other types, many of which you may already be using to search for sensitive content in email. These sensitive information types are detected based on pattern matching and are easy to set up.

Sensitive Information Types are defined by patterns that can be identified by Regular Expressions or a Function and are available for use within Data Loss Prevention policies. Sensitive Information Types improve on simple pattern matching by supporting the use of corroborative evidence such as keywords and checksums to identify sensitive information stored in SharePoint Server 2016 IT Preview in addition to including native confidence level and proximity logic that is used in the evaluation process.

Using the US Social Security Number sensitive information type as an example:

Patterns are identified in one of two ways, Formatted and Unformatted as shown below:

Formatted:

  • Nine digits in the format ddd-dd-dddd OR ddd dd dddd

Unformatted:

  • Nine digits in the format ddddddddd

For proximity it can be 85, 75, 65, or 55 percent. Using 85% as an example:

A Data Loss Prevention policy is 85% confident that it's detected this type of sensitive information if, within a proximity of 300 characters:

  • The function Func_ssn finds content that matches the pattern.
  • At least one of the following is true:
  • A keyword from Keyword_ssn is found.
  • The function Func_us_date finds a date in the right date format.
  • The function Func_us_address finds an address in the right date format.

The specific native keywords based on this type include:

Social Security

Social Security#

Soc Sec

SSN

SSNS

SSN#

SS#

SSID

In this example we have a document that contains the following information in Microsoft Word .docx format with a document name of Candidate Profile for Garth Fort containing the unformatted text:

Candidate Profile for Garth Fort

Background check completed on 8/26/2015

Social Security Number: 123-45-6789

Expires: 8/26/2018

A new eDiscovery Case is created within the eDiscovery Center as Credit Cards and SSNs and a corresponding query mapped to those types:

SensitiveType=”Credit Card Number” OR SensitiveType=”U.S. Social Security Number (SSN)”

In this example the document, once crawled, is discovered as having met the conditions of the Sensitive Information Type and presented as a result in the eDiscovery Case where additional actions can be taken against the content such as Export.

clip_image002

To learn more about available Sensitive Information Types see also https://technet.microsoft.com/en-us/library/jj150541(v=exchg.160).aspx.