As most of my blogs, this blog is also an outcome of an interesting engagement with a client. While trying to help one of my clients to recover from a hardware failure, I learned something about a trace flag. Let us learn about backup or restore history.

My client was trying to restore a database from the backup. Even though it was a small database, SQL was executing restore for a very long time and as soon as they cancelled it, the output was as follows:

36 percent processed.
73 percent processed.
100 percent processed.
Processed 328 pages for database ‘MyDatabase’, file ‘MyDatabase_Logical_File’ on file 1.
Processed 8 pages for database ‘MyDatabase’, file ‘MyDatabase_Logical_File1’ on file 1.
Processed 8 pages for database ‘MyDatabase’, file ‘MyDatabase_Logical_File1’ on file 1.
Processed 3 pages for database ‘MyDatabase’, file ‘MyDatabase_log’ on file 1.
Processed 0 pages for database ‘MyDatabase’, file ‘MyDatabase_log1’ on file 1.
Processed 0 pages for database ‘MyDatabase’, file ‘MyDatabase_log1’ on file 1.
Msg 3009, Level 16, State 1, Line 2
Could not insert a backup or restore history/detail record in the msdb database. This may indicate a problem with the msdb database. The backup/restore operation was still successful.
Problems recording information in the msdb..suspect_pages table were encountered. This error does not interfere with any activity except maintenance of the suspect_pages table. Check the error log for more information.
RESTORE DATABASE successfully processed 347 pages in 0.578 seconds (4.683 MB/sec).
Msg 3204, Level 16, State 1, Line 2
The backup or restore was aborted.
Query was canceled by user.

When they checked the database, it was there and they were able to see all the data in it. This is the time they contacted me and wanted advice on what is happening.
I tried reproducing the same issue in my lab. To simulate logging issue to restore the history table in the msdb database, I have taken lock on the table using below statement.

begin tran
update [msdb].[dbo].[restorehistory] with (tablock)
set replace = 1

Note that, I have not issued rollback/commit and have tablock hint to cause locking. In another query window, I started to restore my database. I used below command

USE [master]
RESTORE DATABASE [Foo] FROM DISK = N'foo.bak'
WITH FILE = 1, NOUNLOAD, STATS = 20
GO

20 percent processed.
40 percent processed.
60 percent processed.
80 percent processed.
100 percent processed.
Processed 63872 pages for database ‘Foo’, file ‘Foo’ on file 1.
Processed 3 pages for database ‘Foo’, file ‘Foo_log’ on file 1.

< <<<<<<<<<<<<< Stuck here >>>>>>>>>>>>>>>>>>>
< <<<<<<<<<<<<< Below appeared when I cancelled the query >>>>>>>>>>>>>>>>

Msg 3009, Level 16, State 1, Line 3
Could not insert a backup or restore history/detail record in the msdb database. This may indicate a problem with the msdb database. The backup/restore operation was still successful.
Problems recording information in the msdb..suspect_pages table were encountered. This error does not interfere with any activity except maintenance of the suspect_pages table. Check the error log for more information.
RESTORE DATABASE successfully processed 63875 pages in 14.750 seconds (33.831 MB/sec).
Msg 3204, Level 16, State 1, Line 3
The backup or restore was aborted.
Query was canceled by user.

When I checked the database, it was there. This means only logging didn’t happen in the MSDB database. Note that this is only a simulation of the issue and the logging might go wrong because of any reason. One of the reasons could be that MSDB database is not available.

WORKAROUND/SOLUTION

If you are not bothered about the history of logging messages in MSDB, then we can use undocumented Trace Flag 3001. (Referred to my websites)

While doing above, I still had the same blocking, but due to trace flag, the logging didn’t happen to MSDB tables and restore completed without error.
Have you ever heard of such trace flags?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: Msg 3009, Level 16 – Could not Insert a Backup or Restore History Detail Record in the msdb Database

Received an email “Need your urgent help On Demand, our MSDB Database has grown too big and we need help to check our MSDB.” I have been an independent consultant for a while and one of the services I provide is “On Demand (50 minutes)” service. This service is very helpful for organizations who are in need immediate help with their performance tuning issue. Though, I have set working ours for my regular clients, every single day, I keep two hours available for this particular offering. This way, I can make sure that anyone who urgently needs my help, can avail the same. Click here to read more about it.

Well, I immediately allocated one of the two hours which I keep available to the individual who sent me an email.

So, I joined call with them and I looked into various table/objects in MSDB. We have found that the objects that are occupying most of the storage are named queue_messages_*

WORKAROND/SOLUTION

The resolution was to delete data from the queue in MSDB. They had no idea who create the queue and how the messages were there. All they were looking for ways to clear the queue. First, we took backup of MSDB database and after that I cleared data from queue_message. You need to make sure that these messages are not important for you. Here is the script which I used.

declare @conversation_handle uniqueidentifier
select top 1 @conversation_handle = conversation_handle from sys.conversation_endpoints
while @@rowcount = 1
begin
     end conversation @conversation_handle with cleanup
     select top 1 @conversation_handle = conversation_handle from sys.conversation_endpoints
end

Well, once we ran above script the database had free up some good amount of space. Though we all know database shrinking is not a good option, this was a good case when we could shrink database and gain additional space. We ran following script on the database and it freed up quite a lot of empty space.

USE [msdb]
GO
DBCC SHRINKDATABASE(N'MSDB')
GO
USE [msdb]
GO
DBCC SHRINKFILE (N'MSDBData' , 0, TRUNCATEONLY)
GO

After running above script, our msdb database which was of 18 GB came down to only 600 MB.

I strongly encourage everyone to read my blog post about SQL SERVER – Shrinking Database is Bad – Increases Fragmentation – Reduces Performance, which explains why shrinking database is not a good idea and how it adversely affect performance. As I said this was a unique scenario where we have to shrink the database.

Well, that’s it. After 50 minutes of consultancy, my client was very happy as we were able to bring back their database to state which made them very happy.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – MSDB Database Uncontrolled Growth Due to Queue_messages. How to Clear All Messages From a Queue?

Recently, one of my clients reported a weird issue to me. They explained that when they right click on the database, and try to read its properties they were getting the following error message. Let us learn about error related to could not load file or assembly.

Here is the text of the message:

Could not load file or assembly ‘SqlManagerUi, Version=14.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91’ or one of its dependencies. The system cannot find the file specified. (mscorlib)

I asked to check if they can open ANY properties window of ANY object in SSMS? They tested and found that none of the additional windows, like creating database, properties were opening. When I searched, I found that last part of the error is the real issue. In our case we are seeing “The system cannot find the file specified”.

We partially know the file name as it should be having SqlManagerUi in the name. Whenever there is a file missing issue, I always use Process Monitor

WORKAROUND/SOLUTION

In process monitor, I filtered for SSMS.exe and found that SSMS was looking for SQLManagerUI.dll under C:\Program Files (x86)\Microsoft SQL Server\140\Tools\Binn\ManagementStudio and it was missing from there. Note that the client’s SSMS version was SQL 2016. If you are using an older version, you might see path like C:\Program Files\Microsoft SQL Server\110\Tools\Binn\VSShell\Common7\IDE

When we compared this with a working machine, we also found that a huge number of files were missing from that folder. The common causes could be a (a) Viruses or other malware, (b) File system corruption and (c) Failing hard drive.

Finally, we had to repair the shared features, after that the issues were resolved. Here are the steps to repair the instance https://docs.microsoft.com/en-us/sql/database-engine/install-windows/repair-a-failed-sql-server-installation

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Could Not Load File or Assembly ‘SqlManagerUi, Version=14.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91’ or One of its Dependencies

Sometime ago I wrote a blog on identity column value jump issue, you can read it from here SQL SERVER –  Jump in Identity column after restart! In this blog post we will learn how to resolve the error when we alter column from int to bigint.

One of my clients contacted me for consultation and they have exhausted Integer range. Due to this, they wanted to know the possible ways to change the existing column to bigint data type.

SOLUTION/WORKAROUND

If there was no primary key on this table, we could have used ALTER TABLE… ALTER COLUMN syntax to modify the data type like below.

ALTER TABLE OneTable ALTER COLUMN ID bigint

In case of primary key or FK dependency, it would fail with below error:

Msg 5074, Level 16, State 1, Line 1
The object ‘PK_OneTable’ is dependent on column ‘ID’.
Msg 4922, Level 16, State 9, Line 1
ALTER TABLE ALTER COLUMN ID failed because one or more objects access this column.

You can refer https://docs.microsoft.com/en-us/sql/t-sql/statements/alter-table-transact-sql for more detailed syntax and limitations.

Here are the solutions which I have shared with them.

Solution 1

1. Create a new bigint column in the table
2. Update that new column with the values from the int column
3. Delete the int column
4. Rename the bigint column

Here is the sample:

CREATE TABLE [dbo].[OneTable](
	[ID] [int] NOT NULL PRIMARY KEY,
	[FName] [nchar](10) NULL,
	[Lname] [nchar](10) NULL,
)
GO

ALTER TABLE OneTable ADD NewColumn BIGINT NOT NULL
GO
UPDATE OneTable SET NewColumn=ID
GO
ALTER TABLE [dbo].[OneTable] DROP CONSTRAINT [PK_OneTable] WITH ( ONLINE = ON )
GO
ALTER TABLE OneTable DROP COLUMN ID
GO
USE [Foo]
GO
ALTER TABLE [dbo].[OneTable] ADD  CONSTRAINT [PK_OneTable] PRIMARY KEY CLUSTERED
(
	[NewColumn] ASC
)
GO

EXEC sp_rename 'OneTable.NewColumn', 'ID', 'COLUMN'

This approach has a problem that column order would change, and the application might break, if it depends on the column order. Honestly, you should not write an application which depends on the column order. You can read it here: How to Add Column at Specific Location in Table? – Interview Question of the Week #126

Here is  the output, which we get when we run sp_rename

Caution: Changing any part of an object name could break scripts and stored procedures.

 Solution 2

1. Created a new table with new datatype.
2. Move data to the new table
3. Drop old table
4. Renamed new table

Both solutions need downtime so need to be done during the maintenance window.

Do you have any other solution? Please comment and share with other readers.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – ALTER Column from INT to BIGINT – Error and Solutions

Disasters are always bad. Recovering from disaster is a skill and needs a lot of planning and practice. One of my clients had a disaster and built a new machine to restore the backups. They tried a few attempts to install SQL Server but were having issues. Finally, they contacted me for consultation to fix rule, reporting services catalog database file existence.

I uninstalled SQL Server and started the installation again. At configuration rules checking screen, we got two items listed as failed.

Here is the text of the message when we click on the “Failed” hyperlink.

Rule “Reporting Services Catalog Database File Existence” failed.
The Reporting Services catalog database file exists. Select a Reporting Services files-only mode installation.

AND

Rule “Reporting Services Catalog Temporary Database File Existence” failed.
The Reporting Services catalog temporary database file exists. Select a Reporting Services files-only mode installation.

I searched on internet and found that its due to earlier files of ReportServer and ReportServerTempDB databases which are created as a part of SSRS installation. I wanted to check setup logs to see the path. Here is the snippet from Detail.txt file

Microsoft.SqlServer.Configuration.RSExtension.DoesTempCatalogExistBlocker
2017-06-30 12:23:09 RS: Initializing the check for exisiting catalog database and temp db files.
2017-06-30 12:23:09 RS: Getting the SQL data path
2017-06-30 12:23:09 RS: Current RS install mode is ‘DefaultNativeMode’
2017-06-30 12:23:09 RS: Using SQl data path E:\Program Files\Microsoft SQL Server\MSSQL12.SQL2014\DATA
2017-06-30 12:23:09 RS: Getting the RS database name
2017-06-30 12:23:09 RS: Using default database name.
2017-06-30 12:23:09 RS: Getting the RS database name
2017-06-30 12:23:09 RS: Using default database name.
2017-06-30 12:23:09 RS: Catalog Temp db file exists
2017-06-30 12:23:09 Slp: Evaluating rule : RS_DoesCatalogTempDBExist
2017-06-30 12:23:09 Slp: Rule running on machine: SQLCDBSVR
2017-06-30 12:23:09 Slp: Rule evaluation done : Failed
2017-06-30 12:23:09 Slp: Rule evaluation message: The Reporting Services catalog temporary database file exists. Select a Reporting Services files-only mode installation.

Now, using log we can see exact behavior of SQL Server Rule check and the location which we need to target.

WORKAROUNS/SOLUTION

If you get same error, try and look at SQL Setup logs and search for “DoesCatalogExistBlocker” and “DoesTempCatalogExistBlocker” and you should see path there. If we look at that path, we should see files like below.
• ReportServer.mdf
• ReportServer_log.LDF
• ReportServerTempDB.mdf
• ReportServerTempDB_log.LDF.
In case of named instance, files would be having instance name as well. In my case, the instance name was SQL2014, so files were as below.

As a safety measure, take a backup and remove the files from this location. Then we can use “rerun” button available on the same screen below the green bar. This should take care of making failed checks as Passed.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: Rule “Reporting Services Catalog Database File Existence” Failed

Applying a patch in SQL Server is a planned process which is followed across various companies. Most of the companies take time to apply the patch to make sure the stability of the patch can be validated. Let us see an error related to patch installer.

One of the client was trying to apply the patch on production and was under tremendous pressure due to the small amount of allocated downtime. They didn’t waste much time and contacted me to see what I can offer in a short amount of time.

Since it was urgent, I quickly join GoToMeeting and looked at the logs of failure.

Instance SAPPROD overall summary:
Final result: The patch installer has failed to update the shared features. To determine the reason for failure, review the log files.
Exit code (Decimal): -2068052374
Exit facility code: 1212
Exit error code: 1642
Exit message: The patch installer has failed to update the shared features. To determine the reason for failure, review the log files.
Start time: 2017-06-29 17:14:46
End time: 2017-06-29 17:15:20
Requested action: Patch
Log with failure: C:\Program Files\Microsoft SQL Server\100\Setup Bootstrap\Log\20170629_171254\SAPPROD\SqlWriter_Cpu64_1.log

To decode 1642, I have used NET command.

Here is the text of the message shown in the above image.

The upgrade cannot be installed by the Windows Installer service because the program to be upgraded may be missing, or the upgrade may update a different version of the program. Verify that the program to be upgraded exists on your computer and that you have the correct upgrade.

MSI (s) (C8:78) [17:15:18:787]: Windows Installer installed an update. Product Name: Microsoft SQL Server VSS Writer. Product Version: 10.50.1600.1. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: {8B9F68F5-8191-4F48-B6C5-4A98858642AF}. Installation success or error status: 1642.
MSI (s) (C8:78) [17:15:18:787]: Note: 1: 1708
MSI (s) (C8:78) [17:15:18:787]: Product: Microsoft SQL Server VSS Writer — Installation failed.
MSI (s) (C8:78) [17:15:18:787]: Windows Installer installed the product. Product Name: Microsoft SQL Server VSS Writer. Product Version: 10.50.1600.1. Product Language: 1033. Manufacturer: Microsoft Corporation. Installation success or error status: 1642.

Above lines also tell error 1642. So, we are sure that this is due to some missing patch files.

SOLUTION/WORKAROUND

In above logs, we know that issue is with “Microsoft SQL Server VSS Writer” so I went to Control Panel > Programs and Features > Microsoft SQL Server VSS Writer > right click > Repair. It failed as the location to where it was pointing did not show and sqlwriter.msi. We mounted base media and browsed to the file which it was asking for.

Using above trick, we successfully repaired the SQL Server VSS Writer. After this we applied the same patch which was failing earlier. This time, it was successful. We went to Control Panel > Programs and Features > View Installed Updates > in the updates for Microsoft SQL Server found that this hotfix was installed.

There might be other reasons of same error message, but in my case above worked. So you may want to search more on internet if the above trick doesn’t fix the issue.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – The Patch Installer has Failed to Update the Shared Features

Microsoft SQL Server has given an option to connect to SQL Server when normal connections can’t be made because of any reasons. This feature is called as Dedicate Administrator Connection (DAC). You can read more about DAC over here. Diagnostic Connection for Database Administrators

During one of my online classes, I showed a demo to show how DAC can be useful in doing some basic diagnostics to find the cause of any issues. One of my students tried it after my class and he was not able to connect. Here is the error message which he shared with me.

A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 43 – An error occurred while obtaining the dedicated administrator connection (DAC) port. Make sure that SQL Browser is running, or check the error log for the port number) (Microsoft SQL Server, Error: -1)

We verified that SQL Browser was running, and it was a local connection. I asked to check my favorite file to look for any errors related to DAC. Interestingly, we found below

2017-07-05 18:24:10.84 Server Error: 17182, Severity: 16, State: 1.
2017-07-05 18:24:10.84 Server TDSSNIClient initialization failed with error 0x2, status code 0x22. Reason: Unable to retrieve dynamic TCP/IP ports registry settings for Dedicated Administrator Connection. The system cannot find the file specified.

2017-07-05 18:24:10.84 Server, Dedicated admin connection support was not started because of error 0x2, status code: 0x1. This error typically indicates a socket-based error, such as a port already in use.

There are two messages here. First one says: “The system cannot find the file specified” and based on complete message, it looks like SQL is looking for a certain registry key. A second message is a side effect of the first message and not because of the port used by any other process.

SOLUTION/WORKAROUND

I compared it with my system and found that under below the key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL13.SQL2016\MSSQLServer\SuperSocketNetLib\AdminConnection\Tcp

We were not seeing TcpDynamicPorts key, which is precisely reported by SQL ErrorLog. So, we created key as shown below.

On my server MSSSQL13 is for SQL Server 2016 and SQL2016 is the instance name. If you have default instance of SQL Server 2016 then it would be MSSQL13.MSSQLSERVER. Based on your SQL Server version and instance name you need to create appropriate key.

Have you solved any issue using DAC? Please share via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – An Error Occurred While Obtaining the Dedicated Administrator Connection (DAC) Port

One of my clients wanted to secure their SQL Server. One of the rules was that wanted to avoid running SQL Server service with an account which is part of local “Administrators” group. They wanted to provide the minimal permissions needed for the SQL server service account for SQL Server to function.

Below are the respective articles for each version that outline the permissions assigned to the service accounts:

Configure Windows Service Accounts and Permissions (SQL Server 2016 and later)
https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/configure-windows-service-accounts-and-permissions

Configure Windows Service Accounts and Permissions (SQL Server 2014)
https://msdn.microsoft.com/en-US/library/ms143504(SQL.120).aspx#Windows

Configure Windows Service Accounts and Permissions (SQL Server 2012)
https://msdn.microsoft.com/en-US/library/ms143504(SQL.110).aspx#Windows

Setting Up Windows Service Accounts (SQL Server 2008 R2)
http://msdn.microsoft.com/en-us/library/ms143504(v=sql.105).aspx#Review_NT_rights

Setting Up Windows Service Accounts (SQL Server 2008)
http://msdn.microsoft.com/en-us/library/ms143504(v=sql.100).aspx#Review_NT_rights

Setting Up Windows Service Accounts (SQL Server 2005)
http://msdn.microsoft.com/en-us/library/ms143504(v=sql.90).aspx#Review_NT_rights

When I spoke to my client, they already removed service account from the Administrators group. As per article, we have permissions in security policy, but still we were not able to start SQL Service. By looking further into the Event Viewer, under Application Log we see the error :

“initerrlog: Could not open error log file ”. Operating system error = 3(The system cannot find the path specified.)”.

SQL SERVER – Event 17058 – initerrlog: Could not Open Error Log File

If we provide incorrect path, we generally see below message.

initerrlog: Could not open error log file ‘C:\BadPath\Errorlog.txt’. Operating system error = 3(The system cannot find the path specified.).”

Notice the blank path in an error message.  Later, we used one of my favorite tools called as Process Monitor and ran the tool to look for registry read activity. We knew that the service was queried for the for the start-up parameters which are saved in the registry and we wanted to check how it does it. We saw we were getting Access Denied for the Account: Americas\SAPPROD_SQLSvc on the registry hive:

HKLM\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQLServer\Parameters

Seeing that, we went to the registry hive and checked if the service account had permission and found that the user did not have permission, we also saw that the there was a group, SQLServerMSSQLUser$MPNA01$MSSQLSERVER

WORKAORUND/SOLUTION

We were able to fix the issue by adding Service Account Americas\SAPPROD_SQLSvc to the group SQLServerMSSQLUser$MPNA01$MSSQLSERVER

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Unable to Start SQL When Service Account is Removed From Local Administrators Group. Why?

There are many rules in SQL Server setup which are checked to make sure that the user is not seeing failure at the end of the installation. Most of the rules are generally passed, but few of them are known to generate warnings. One of such rule is Microsoft .NET Application Security which can be seen below.

If we click on a hyperlink, we would see below.

Here is the text of the message.

“The computer cannot access the Internet. There might be delays in starting a .NET application like Management Studio. If navigate to http://crl.microsoft.com/pki/crl/products/MicrosoftRootAuthority.crl and are prompted to download the MicrosoftRootAuthority.crl file you should not have .NET security validation issues. It is not necessary to download the MicrosoftRootAuthority.crl file.”

In my case, the machine, on which I was installing SQL Server, was not having a connection to the internet, so I was getting the warning.

WORKAROUND / SOLUTION

As explained in the warning message, if we have SSMS installed on this machine, we might see slowness in opening it. While I don’t know how to bypass the warning, but I do know trick to launch SSMS faster.

Based on my limited knowledge of .NET, the reason why it is slow is because .Net Runtime tries to contact crl.microsoft.com to ensure that the certificate validity. If then there is no direct route out to the internet to crl.microsoft.com to validate the certificate(s) and it eventually times out after 30 to 45 seconds.

To bypass above, we can make changes to the HOSTS file. By this we can force the certificate checks to crl.microsoft.com to route to the local host which immediately fails and the certificate checks are ignored. Here are the steps

1. Press the keys [Win] + [R]
2. Enter the following..

notepad %systemroot%\system32\drivers\etc\hosts

3. Append the following..

127.0.0.1    crl.microsoft.com

4. Save the file.

This will not bypass the warning, but an issue which might be caused in the future would be resolved.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Setup Rule: Microsoft .NET Application Security. The computer cannot access the Internet

One of my clients contacted me for On Demand (55 minutes) as they believed that this was a simple issue. John was honest enough to confess the mistake which he has done, which lead to error related to invalid object name.

John was trying to troubleshoot a deadlock issue, and he found that this specific server doesn’t have a system_health session in extended events.

So, he found that the definition of the session is defined in U_tables.sql file from “Install” folder.

C:\Program Files\Microsoft SQL Server\MSSQL13.SQLSERVER2014\MSSQL\Install

He executed the script, but it failed with below the messages.

This file creates all the system tables in master.
drop view spt_values ….
Creating view ‘spt_values’.
Msg 208, Level 16, State 1, Procedure spt_values, Line 56
Invalid object name ‘sys.spt_values’.
sp_MS_marksystemobject: Invalid object name ‘spt_values’
Msg 15151, Level 16, State 1, Line 61
Cannot find the object ‘spt_values’, because it does not exist or you do not have permission.
drop table spt_monitor ….
Creating ‘spt_monitor’.
Grant Select on spt_monitor
Insert into spt_monitor ….

Now, there was a bigger problem. A lot of places in SSMS, he started seeing below errors.

An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)
Invalid object name ‘master.dbo.spt_values’. (Microsoft SQL Server, Error: 208)

John knew that he should create the view ‘master.dbo.spt_values’ using below but was unable to.

WORKAROUND/SOLUTION

To create master.dbo.spt_values the reference is needed to sys.spt_values. This can’t be accessed by normal connection. There are two ways

1. Start SQL Server in single user mode, which would need downtime.
2. Connect using Dedicate Administrator Connection (DAC). You can read more about DAC over here. Diagnostic Connection for Database Administrators

After making connection run below script.

create view spt_values as
select name collate database_default as name,
	number,
	type collate database_default as type,
	low, high, status
from sys.spt_values
go
EXEC sp_MS_marksystemobject 'spt_values'
go
grant select on spt_values to public
go

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Invalid Object Name ‘master.dbo.spt_values’ in Management Studio

Earlier I wrote a blog on same error message which you can read from below link. SQL SERVER – FIX: Msg 35250, Level 16, State 7 – The Connection to the Primary Replica is Not Active. The Command Cannot be Processed

One of my blog readers emailed me that he is getting same error 35250 (The Connection to the Primary Replica is Not Active. The Command Cannot be Processed.)

But he did not see “Database Mirroring login attempt by user” in the SQL Server ERRORLOG. He told me that my blog didn’t help me.

I reached him back via email and provided few steps to check. First, I asked to run below query to get port and service account details.

SELECT tep.NAME 'Endpoint Name'
	  ,sp.NAME 'Owner'
	  ,tep.state_desc 'State'
	  ,tep.port 'Port'
FROM sys.tcp_endpoints tep
INNER JOIN sys.server_principals sp ON tep.principal_id = sp.principal_id
WHERE tep.type = 4

Here was the output and this tells me that they are using the default port for Always On availability groups.

hadr_endpoint, database_mirroring, started, 5022

As a next step, I asked him to do two more things.

1. Telnet <Primary replica> 5022 from secondary replica via command line (cmd.exe).
2. Telnet <Secondary replica> 5022 from primary replica via command line (cmd.exe).

Telnet from the primary to the secondary worked, but it failed from the secondary to the primary. He was smart enough and tried to ping the IP address, but that did not work as well.

WORKAROUND/SOLUTION

Based on our test we concluded that this was an issue with network component. My client engaged his networking experts and found the cause of ping failure and fixed it.

Have you seen such error and found some more solution?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: Msg 35250 – The Connection to the Primary Replica is Not Active. The Command Cannot be Processed

While preparing for a demo for an upcoming session, I came across an error which I didn’t see earlier. Here is the error in the UI and it is related to Backup Log Operation.

I am also trying to add a file to the database using below command.

USE [master]
GO
ALTER DATABASE [SQLAuth] ADD FILE ( NAME = N'SQLAuth_LogNew', FILENAME = N'F:\DATA\SQLAuth_LogNew.ndf' , SIZE = 8192KB , FILEGROWTH = 65536KB ) TO FILEGROUP [PRIMARY]
GO

And it also failed with following error.

If I change the file name, it works. As mentioned in error message I performed a log backup, and I was able to add the file again.

ROOT CAUSE

While working yesterday, I already added the file and dropped it. As per error message, if I need to reuse the same logical name of the database file, I must take a log backup. Here is the simple reproduction of the error.

CREATE DATABASE [DropFileTest]
GO
USE [master]
GO
-- take a full backup to make it real full else it would act as simple.
BACKUP DATABASE [DropFileTest] TO  DISK = N'DropFileTest.bak'
GO
-- add a new LDF file
USE [master]
GO
ALTER DATABASE [DropFileTest] ADD LOG FILE ( NAME = N'NewLDF',
FILENAME = N'F:\LOG\NewLDF.ldf' , SIZE = 8192KB , FILEGROWTH = 65536KB )
GO
-- reomve the file.
USE [DropFileTest]
GO
ALTER DATABASE [DropFileTest]  REMOVE FILE [NewLDF]
GO
-- Adding same file again and it should fail.
USE [master]
GO
ALTER DATABASE [DropFileTest] ADD LOG FILE ( NAME = N'NewLDF',
FILENAME = N'F:\LOG\NewLDF.ldf' , SIZE = 8192KB , FILEGROWTH = 65536KB )
GO

If you run above, you can notice the same error.

WORKAROUND/FIX

As mentioned in the error message, take a transaction log backup of the database. In the above demo script, we can do below before adding a file and it should work.

BACKUP LOG [DropFileTest] TO  DISK = N'DropFileTest.trn'
GO
-- Adding same file again and it should work this time.
USE [master]
GO
ALTER DATABASE [DropFileTest] ADD LOG FILE ( NAME = N'NewLDF', FILENAME = N'F:\LOG\NewLDF.ldf' )
GO

Have you seen such interesting error?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Msg 1833 – File Cannot be Reused Until After the Next BACKUP LOG Operation

Today the progress made in technology is astounding and beyond our imaginations, but still there are clients who can’t move to higher, latest and greatest version of SQL so easily. They are still stuck with SQL 2005 and SQL 2008 SQL servers. In this blog post we will learn how to fix RequireKerberos Error During SQL 2008 Installation on Windows 2008 R2 Cluster.

One of my client was trying to install SQL Server 2008 on Windows 2008 R2 cluster. While the setup was almost at the end, here was the error popped up.

The following error has occurred:
There was an error setting private property ‘RequireKerberos’ to value ‘1’ for resource ‘SQL Network Name (DELHISQLPRD)’. Error: Value does not fall within the expected range.

He was not sure what was wrong with his cluster. He contacted me and since I have worked with many other clients and this was seen earlier. I told him that this is a known issue (List of known issues when you install SQL Server on Windows 7 or on Windows Server 2008 R2) and Microsoft has fixed it in SP1 for SQL Server 2008

WORKAROUND/SOLUTION

As mentioned in knowledge base article by Microsoft, the issue is fixed in SP1 of SQL 2008. We need to use “Slipstream media” to install SQL Server 2008. Slipstream media mean include patch (like service pack) into the original installation. We need to follow below article to create slipstream media.

https://support.microsoft.com/en-us/help/955392/how-to-update-or-slipstream-an-installation-of-sql-server-2008

We made sure that we uninstalled SQL using “Remove Node” option before using slipstream media to install SQL cluster again. This time it worked like a charm.

Have you seen similar setup issues which you solved? Please comment and share with other readers.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: RequireKerberos Error During SQL 2008 Installation on Windows 2008 R2 Cluster

Sometimes some behaviors in SQL Server are unexpected and that’s the right time when an expert need to be engaged. In this blog post we will learn about how automatic startup sometimes does not work in SQL Server Services.

One of my client told that they had a business down situation for few hours. Based on their findings they realized that there was a restart of SQL Server machine due to power failure. They are OK with that, but their expectation was that SQL Service should have automatically started after machine reboot which didn’t happen. I asked to share the screenshot of SQL Server Configuration Manager.

By looking at the screenshot it’s clear that they have already configured “Start Mode” of SQL Service for Automatic. Which means their expectation was correct.

Next, we looked into Event Logs to see any interesting information. We found Event ID 5719 via source NETLOGON.

Log Name: System
Source: NETLOGON
Event ID: 5719
Description:
The MSSQLSERVER service was unable to log on as FORTUNE\SQLSvc with the currently configured password due to the following error: There are currently no logon servers available to service the logon request. To ensure that the service is configured properly, use the Services snap-in in Microsoft Management Console (MMC)

It was not very difficult to search below knowledge base article from Microsoft which explained the cause of such message. https://support.microsoft.com/en-in/help/139410/err-msg-there-are-currently-no-logon-servers-available

ROOT CAUSE

Based on the error messages we can conclude that in this scenario the SQL Service didn’t start automatically because SQL Server was unable to communicate to domain controller (DC). When I checked with client about above and asked about network connectivity to DC, they replied – “Unfortunately, domain controller was also rebooted due to power failure”.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Automatic Startup Not Working for SQL Server Service

While preparing for a demo for a client I was hit with an interesting error in AlwaysOn Availability Group backup and restore scenario. In my lab setup, I had four nodes SQLAUTH1 to SQLAUTH4 and I have always on high availability turned on SQL Server 2016. I played and did several operations with database and found that I have bloated the transaction log file to 100 GB. And now I need to shrink the transaction log on SQLAUTH1. As soon as I did that, I was welcomed with below message about database cannot be shrunk until all secondaries have moved past the point.

The log for database ‘SQLAuth’ cannot be shrunk until all secondaries have moved past the point where the log was added.

It’s worth mentioning that I did add files in the database so above message is valid. I even took transaction log backup, but no luck. File usage was still shown as 99%.

WORKAROUND/SOLUTION

I looked around at AlwaysOn Dashboard and found that one of the secondary was not getting synchronized. Since it was a lab machine I decided to delete the AG and the thing came back to normal.

But in production, if you can’t afford to delete and reconfigure Always On availability group, then you need to find the cause why secondary is not getting synchronized with primary. In theory, the transaction log files on primary would keep on growing due to secondary not synchronizing with primary.  If you find that the data movement is suspended, then you may want to resume it and find the possible cause in the SQL Server ERRORLOG file

SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location

One of the possible case I can think of is a change of the service account.

SQL SERVER – FIX: Msg 35250, Level 16, State 7 – The Connection to the Primary Replica is Not Active. The Command Cannot be Processed

STILL STUCK?

If you are not able to find the cause, feel free to contact me and use my consulting services. As you might know that I have been an independent consultant for a while and one of the services I provide is “On Demand (50 minutes)” service. This service is very helpful for organizations who are in need immediate help with their performance tuning issue. Though, I have set working ours for my regular clients, every single day, I keep two hours available for this offering. This way, I can make sure that anyone who urgently needs my help, can avail the same. Click here to read more about it.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: The Log for Database Cannot be Shrunk Until All Secondaries Have Moved Past the Point Where the Log was Added

Many companies do hardening of servers to make sure they are protected from any known vulnerabilities. It is important to test all the application in test server before doing hardening. Let us learn about how to fix Eventlog Service not starting.

Recently, one of my existing clients contacted me and informed that they are not able to start SQL Server Agent Service after running a hardening script. Without wasting much time, I joined the session with them and started looking at the logs.

I asked to start SQL Server Agent from configuration manager. It failed with an unhelpful error message

The request failed or the service did not respond in a timely fashion. Consult the event log or other applicable error logs for details.

When I checked the LOG folder, there was no SQLAgent.out generated. I thought of checking the event viewer and found below the message.

But is that related to SQL Server Agent startup issue? I ran the SQLAgent executable in console mode using below command

SQLAGENT.EXE" -i SQL2016 -c -v

This failed with an error:

The EventLog service has not been started
2017-08-03 09:24:45 – ? [098] SQLServerAgent terminated (normally)

When I checked “Windows Event Log” Service, it was stopped, and I was unable to start it. I was getting access denied error when trying to start.

Windows could not start the Windows Event Log service on Local Computer. Error 5: Access is denied.

I used Process Monitor tool and found that we had “Access Denied” on C:\Windows\System32\winevt\Logs\System.evtx file. This is the System Event Log. When we checked the properties, we found “Read-only” was checked which was not the case with other machines.

As soon as we removed the checkbox, we were able to start SQL Server Agent service.

Have you ever encountered such issues due to hardening? Please share via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – SQL Agent Not Starting. The EventLog Service has Not Been Started

One of my clients reported an interesting issue and I was needed to wear detective cap to solve it. The mystery, for which they hired me was as below for Database Files Initial Size Changing Automatically.

Pinal,
We see an interesting issue, and we believe there is something not right with our environment. As per best practices from one of your blog, we have configured the initial file size of the database files to big enough for next 1 year data growth.

We are noticing that the file size is not getting set and it changes to smaller size every night.  Would you be able to suggest us something?

MY INVESTIGATION

I asked them if there is any maintenance plan which runs every night and does the shrink of the database, like below?

And they said that they have no maintenance plan doing such activity.

Here are the various data points I looked at.

1. SELECT * FROM database_files

2. SELECT * FROM databases

3. sp_helpdb SAPDB

When I looked at sys.databases, I could see is_auto_shrink_on was set to 1.

You can run below query in you production database and make sure auto shrink is set to 0.

SELECT is_auto_shrink_on, * FROM sys.databases

ROOT CAUSE

As you can see above, we concluded that the behavior which they saw was because Auto Shrink property was set to ON for many databases. Since this was one of an active database, auto shrink was kicked as compared to other databases. This article discusses the use of both Auto options.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Database Files Initial Size Changing Automatically

If you have ever contacted me via email, you would know that I am very active in replying to emails. Many of the emails are for suggestions and I don’t get much time to help everyone, but I do reply to them letting them know the alternatives. If you are following my blog, you would know that I do provide “On Demand” services to help critical issues. This blog is an outcome of one of such short engagement. This is how it started and it is about RPC Server.

Received an email
Need your urgent help “On Demand”!

We are trying to install SP2 on the passive node of a SQL Server 2016 cluster. This patch has already worked on one server, but now we’re getting RPC too busy errors.

We are under strict timelines to finish this activity. Can you please help us quickly?

Without spending much time, I asked them to join GoToMeeting and got started. When they showed me the screen, the SQL Server setup was on the error “Failed to retrieve data for this request”.

I asked to cancel the setup and share the setup logs to see exact error.

Microsoft.SqlServer.Management.Sdk.Sfc.EnumeratorException: Failed to retrieve data for this request. —> Microsoft.SqlServer.Configuration.Sco.SqlRegistryException: The RPC server is too busy to complete this operation.

WORKAROUND/SOLUTION

As we can see above, SQL setup is failing to do an activity on remote node. I immediately recalled a similar blog which I wrote earlier having same remote node symptoms, but the error message was different.

SQL SERVER – Microsoft.SqlServer.Management.Sdk. Sfc.EnumeratorException: Failed to Retrieve Data for This Request

We checked “Remote Registry Service” on remote node and sure enough, it was in “stopped” state. As soon as we started, we were able to move forward and finish the activity in less than the scheduled time.

If you are having any quick issue to resolve, you can also avail the same kind of services. Click here to read more about it.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Cluster Patching: The RPC Server is Too Busy to Complete This Operation

This blog explains what needs to be done when there are tools which can monitor and report issue about SPNs. Once a client informed that SCOM (System Center Operations Manager) is connected to the databases on SQL Server and raising below warning regarding the SQL Server (Service Principal Name) SPNs:

SQL Server cannot authenticate using Kerberos because the Service Principal Name (SPN) is missing, misplaced, or duplicated.

Service Account: SQLAUTHORITY\ProdSQLSrv

Missing SPNs: MSSQLSvc/SAPSQLSERVER.SQLAUTHORITY.NET:SAP, MSSQLSvc/ SAPSQLSERVER.SQLAUTHORITY.NET:1433

Here is the screenshot for the alert.

Whenever I see errors/warning related to SPN, I always try to use Kerberos Configuration Manager tool.

WORKAROUND/SOLUTION

It is important to note that this is an alert not a failure because SPN issue will not cause every connection to fail. It means that any application, which is trying to use Kerberos authentication is going to fail with errors:

1. Login failed for user ‘NT AUTHORITY\ANONYMOUS LOGON’
2. Cannot generate SSPI context

I downloaded the tool and installed in on the server. As soon we are launching the tool, we can just hit connect. The best piece about this tool is that it can help in finding missing SPN and provide script to run or fix it directly if you have permission. Basically, it can

• Gather information on OS and Microsoft SQL Server instances installed on a server.
• Report on all SPN and delegation configurations on the server.
• Identify potential problems in SPNs and delegations.
• Fix potential SPN problems.

If you are not a fan of a installing tool on the server then you can use SETSPN.exe to set the correct SPNs. Based on above error, here is the command.

setspn -A MSSQLSvc/ SAPSQLSERVER.SQLAUTHORITY.NET:1433 SQLAUTHORITY\ProdSQLSrv

Note that we need to run above line from an elevated command prompt with an account with domain admin permissions. We ran the command and after a minute we got the message all was fine. And as expected, the alerts don’t come back anymore.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SCOM – Alert: SQL Server Cannot Authenticate Using Kerberos Because the Service Principal Name (SPN) is Missing, Misplaced, or Duplicated

After installing the latest version of SSMS (17.2), I was not able to connect to SQL Server Integration Services SSIS on my local machine.

Here is the text of the error message.

Connecting to the Integration Services service on the computer “LOCALHOST” failed with the following error: “The specified service does not exist as an installed service.”.

This error can occur when you try to connect to a SQL Server 2005 Integration Services service from the current version of the SQL Server tools. Instead, add folders to the service configuration file to let the local Integration Services service manage packages on the SQL Server 2005 instance.

This error can occur when you try to connect to a SQL Server 2005 Integration Services service from the current version of the SQL Server tools. Instead, add folders to the service configuration file to let the local Integration Services service manage packages on the SQL Server 2005 instance.

I checked services.msc to make sure it was running.

Above image confirmed that error message “The specified service does not exist as an installed service” is totally misleading.

WORKAROUND/SOLUTION

After trying various steps given on various sites, I was looking at the documentation and found https://docs.microsoft.com/en-us/sql/integration-services/service/integration-services-service-ssis-service#manage-the-service This says:

To connect directly to an instance of the legacy Integration Services, Service, you have to use the version of SQL Server Management Studio (SSMS) aligned with the version of SQL Server on which the Integration Services Service is running. For example, to connect to the legacy Integration Services, Service running on an instance of SQL Server 2016, you have to use the version of SSMS released for SQL Server 2016

Which means that, I need to download and install older version of SQL Server Management Studio. (16.5.3).

As of today, the download link is http://go.microsoft.com/fwlink/?LinkID=840946 which might change in the future so follow below steps.

1. Go to https://docs.microsoft.com/en-us/sql/ssms/sql-server-management-studio-changelog-ssms
2. Search for “SSMS 16.5.3” and you should reach to the below image.
   
3. Click on the hyperlink to download

Hope this would help someone’s time as I wasted close to 30 min.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL Server Management Studio (SSMS) – Unable to Connect to SSIS – The Specified Service Does Not Exist as an Installed Service