There have been many situations where rather than performance tuning, I am contacted for an installation issue. Due to my knowledge by looking at logs and pinpointing at the error message, I am able to fix them as well. In this blog post we will learn how to fix Error Installing Microsoft Visual Studio 2010 Shell.

Here is another blog which is related to patching issue which I was asked to fix recently. In the UI, the failure was caused while installing Visual Studio 2010 Shell. I looked at log pointed which was VC10Redist_Cpu64_1.log

Here is the content of the file.

******* RunEngine:
******* Product: C:\Program Files\Microsoft SQL Server\110\Setup Bootstrap\Update Cache\KB3045318\GDR\redist\VisualStudioShell\VC10SP1\vc_red.msi
******* Action:
******* CommandLine: **********
MSI (s) (A8:84) [13:58:01:399]: Note: 1: 2203 2: C:\Program Files\Microsoft SQL Server\110\Setup Bootstrap\Update Cache\KB3045318\GDR\redist\VisualStudioShell\VC10SP1\vc_red.msi 3: -2147287037
MSI (s) (A8:84) [13:58:01:399]: MainEngineThread is returning 3
MSI (s) (A8:64) [13:58:01:415]: User policy value ‘DisableRollback’ is 0
MSI (s) (A8:64) [13:58:01:415]: Machine policy value ‘DisableRollback’ is 0
MSI (s) (A8:64) [13:58:01:415]: Incrementing counter to disable shutdown. Counter after increment: 0
MSI (s) (A8:64) [13:58:01:415]: Note: 1: 1402 2: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Installer\Rollback\Scripts 3: 2
MSI (s) (A8:64) [13:58:01:415]: Decrementing counter to disable shutdown. If counter
MSI (c) (90:CC) [13:58:01:415]: MainEngineThread is returning 3

This was the only information in the log. If we have a look at the initial stage of the setup file, we see an error related to C:\Program Files\Microsoft SQL Server\110\Setup Bootstrap\Update Cache\KB3045318\GDR\redist\VisualStudioShell\VC10SP1\vc_red.msi

I checked the file and it was not there in the location mentioned above. We were not sure how it was missing. May be someone deleted the file due to space issue on C drive? For the path, it looks like the SQL setup caches the media locally for un-installation purpose?

WORKAROUND/SOLUTION

We download patch from KB3045318 again and looked for the file. It was found under VC10SP1\x64\vc_red.msi  folder. The fix the above issue was that we had to copy vc_red.msi directly into VC10SP1 so that the final path looks like — C:\Program Files\Microsoft SQL Server\110\Setup Bootstrap\Update Cache\KB3045318\GDR\redist\VisualStudioShell\VC10SP1\vc_red.msi

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Error Installing Microsoft Visual Studio 2010 Shell

Along with my performance consulting, I also assist many clients in fixing some quick issue. You can refer for various consulting services which I provide. This blog is an outcome of one such engagement where the client has deployed standalone SQL Server on two machines, created windows cluster between them and wanted to then use AlwaysOn availability group feature. This blog is about Primary Replica not being active.

Here is the text of the message (for search engines)

Joining database on secondary replica resulted in an error. (Microsoft.SqlServer.Management.HadrTasks)
——————————
ADDITIONAL INFORMATION:
Failed to join the database ‘SQLAuthority’ to the availability group ‘ProdAG’ on the availability replica ‘SRV2’. (Microsoft.SqlServer.Smo)
——————————
An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)
——————————
The connection to the primary replica is not active. The command cannot be processed. (Microsoft SQL Server, Error: 35250)
——————————

We get the same error even if we try to run JOIN command via T-SQL.

Msg 35250, Level 16, State 7, Line 1
The connection to the primary replica is not active. The command cannot be processed.

We checked ERRORLOG on both the replicas and found below the messages

Database Mirroring login attempt by user ‘DomainName\svc_sql.’ failed with error: ‘Connection handshake failed. The login ”DomainName\svc_sql’ does not have CONNECT permission on the endpoint. State 84.’. [CLIENT: 192.168.1.11]

SOLUTION/WORKAROUND

The error message is clear that the service account is not able to connect to SQL Server on database mirroring / AlwaysOn AG port. To fix the issue we need to run below command

GRANT CONNECT ON ENDPOINT::Hadr_endpoint TO [DomainName\svc_sql];

Once above was done, the issue was resolved and we were able to join the database to AG.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – FIX: Msg 35250, Level 16, State 7 – The Connection to the Primary Replica is Not Active. The Command Cannot be Processed

I must confess that consulting has taught me much more than what I have learned in my regular job. Freelancing comes with advantages and disadvantages. For me, I am looking at advantages and I am happy with my decision. Let us see how to fix an error related to Login Failed and Deleting Collection.

Whenever there is an upgrade of SQL version, it is expected that the server would go to the script upgrade mode and would finish it as a part of database recovery. There are many situations where I have seen server getting stuck to script upgrade mode. This is the blog which I have written long ago to troubleshoot such issues.

SQL SERVER – Login Failed For User – Reason Server is in Script Upgrade Mode

One of the client followed above article and shared information about their SQL script upgrade mode.

2017-05-16 01:29:18.66 spid7s Creating [dbo].[fn_sysutility_mi_get_batch_manifest] function
2017-05-16 01:29:18.78 spid7s Deleting collection set “Utility Information”…
2017-05-16 01:29:42.21 Logon Error: 18401, Severity: 14, State: 1.
2017-05-16 01:29:42.21 Logon Login failed for user ‘India\SQLSvc’. Reason: Server is in script upgrade mode. Only administrator can connect at this time. [CLIENT: ]

I have asked to share information about below tables as they were found in the script

use msdb
go
select COUNT(*) from sysssislog(nolock)
go
select COUNT(*) from syscollector_execution_log_internal(nolock)
go

There were having a huge number of records 400K+

WORKAROUND/SOLUTION

To allow connections to SQL, we need to start SQL services with trace flag 902. Here are the steps which helped my client.

• Keep the SQL Server Service running with the T902 startup parameter
• Backup MSDB database and remove the data of the above two tables.
• Remove the -T902 parameter.
• Restart the SQL Server service to allow the upgrade script to finish.

I was not able to spend much time to identify why those tables had a huge amount of records. It may be due to a lot of SSIS packages? May be due to excessive logging? Since the issue was resolved by manually cleaning the tables, they never wanted to spend more time and money to find the cause of the error.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Login Failed – Server is in Script Upgrade Mode – Deleting Collection Set “Utility Information”

A few months ago, I wrote a blog which tells us reasons about why we should use tools provided with the product rather than generic tools. Yeah, I am talking about services.msc vs. SQL Server Configuration manager.

Why to Use SQL Server Configuration Manager Over Services applet (services.msc)? – Interview Question of the Week #112

Once of the blog reader reported below error.

SQL Server Configuration Manager
—————————
Cannot connect to WMI provider. You do not have permission or the server is unreachable. Note that you can only manage SQL Server 2005 and later servers with SQL Server Configuration Manager.
Invalid class [0x80041010]

I contacted him via email provided and he also shared the image of the message.

While searching on the internet, there were various blogs talking about below solution.

1. Find the *.mof file for SQL Server 100 (SQL2008) or 110 (SQL2012) or 120 (SQL2014) or 130 (SQL2016)
2. Use command “mofcomp” to compile the mof file.

We tried that and it didn’t succeed.

An error occurred while processing item 10 defined on lines 73 – 79 in file C:\Program Files (x86)\Microsoft SQL Server\110\Shared\sqlmgmproviderxpsp2up.mof:
Error Number: 0x80070005, Facility: Win32
Description: Access is denied.
Compiler returned error 0x80070005

There are two important things in above failure. Error message – “Access is denied” and the line number. If you get error for 4 – 7 then most likely you have not run CMD using “Run As Administrator” but if you get error for some other line (like we got for 73-79) then the problem would be something else.

SOLUTION/WORKAROUND

Since it was an access denied error, I downloaded and run a process monitor and looked for access denied. We found below interesting key related to WMI.

svchost.exe 1023 RegOpenKey HKCR\CLSID\{73E709EA-5D93-4B2E-BBB0-99B7938DA9E4}\LocalServer32 ACCESS DENIED

Now, we know the key which is showing access denied.

We checked permission and found that Administrator has full permission on the above key. But access denied was for Network Service account so we added Network service on the above key and then ran the above command (MOFCOMP) again. Since permissions were fixed, we could run the above command successfully. After that SQL Server Configuration Manager just opened fine and we were able to manage the SQL related services successfully.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Configuration Manager – Cannot Connect to WMI Provider. You Do Not Have Permission or The Server is Unreachable

Whenever you are into a business of “fixing the issue” – you should always find in error message first. The whole approach of solving the issue changes if we go behind the wrong error message. Let us learn about the error Backup Failure.

Once of my client told that their backups are failing randomly. If you are a DBA, you would agree with me that there can be many different reasons of backup failure, mostly permission or space issue. So, I asked them to send me email with exact issue and the error message from a SQL Server ErrorLog file. So now, they reported that – Our backups are “randomly” failing to backup up a database to a file share will with below errors in ERRORLOG file.

2017-05-11 11:36:23.10 spid109 Error: 18204, Severity: 16, State: 1.
2017-05-11 11:36:23.10 spid109 BackupDiskFile::CreateMedia: Backup device ‘\\ozone-fs\Backups\DBName.bak’ failed to create. Operating system error 1240(The account is not authorized to log in from this station.).
2017-05-11 11:36:23.10 Backup Error: 3041, Severity: 16, State: 1.
2017-05-11 11:36:23.10 Backup BACKUP failed to complete the command BACKUP DATABASE DBName. Check the backup application log for detailed messages.

The first observation from error message is that it’s a share not a local drive. I was curious to know what kind of share it is? They informed that this is Scale-Out File Server (SOFS) configuration with CA (Continuously Available) shares. When I searched for the error message, it was talking about SMB and some registry level change, but had nothing to do with CA shares.

I had no idea about this feature so I did some research about this feature and found something interesting. Read the article here.

My thoughts:

Based on my research and learning, for the SQL backups, we should not use Scale-Out File Server (SOFS) configuration with CA (Continuously Available) shares, in favor of General Use File Server Cluster. Here is the reference.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER- Error: 18204, Severity: 16 – Backup Failure – Operating System Error 1240 (The Account is not Authorized to Log in From This Station)

There are many times when internet has answers to many questions. To get the right answer, you need to search for the right keyword and it comes from practice. Let us learn how we can fix the error Unable to Change “Shared Feature Directory” Under Feature Selection.

One of my blog readers pinged on Skype and told that they need quick assistance as they are unable to proceed with SQL Server installation. They were trying to install SQL Server 2014, but they were unable to change the shared feature installation directory in the setup.

As we can see above that both the textbox and browse button were disabled and it was stopping them to choose the desired path.

WORKAROUND/SOLUTION

I was able to find the cause for them. As per my search on the internet, if there are any shared components installed on a server in the default path, then we will not be able to change the default shared future path. It is not just a SQL version which we are installing. Even if we have any other instance shared directory installed in other drives. We will not be able to change the path during the next installation. So, what can be done? Well, if you really want to install the shared components in some other drives, then the currently installed drive, then we need to uninstall currently installed instances and have to install the new instance in different drives.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Unable to Change “Shared Feature Directory” Under Feature Selection

One of my client was running into the below error running xp_logininfo on some of their users.  It was interesting that when they run the script to capture user information it works for the same user in another domain. They asked me if there is anything specific they can look for to find out the difference between the two users in the different domains?

Msg 15404, Level 16, State 4, Server BRSQL3, Procedure xp_logininfo, Line 43
Could not obtain information about Windows NT group/user ‘INDIA\SQLGroup’, error code 0x2147.
Msg 15404, Level 16, State 4, Server VISQL3, Procedure xp_logininfo, Line 43
Could not obtain information about Windows NT group/user ‘EMEA\SQLGroup’, error code 0x2147.

I converted hex code 0x2147 to decimal (using the calculator as below)

And it was error number 8519 which means “A global group cannot have a cross-domain member.” As per the message, it seems like something to do with the type of group they were trying to add.

The error message helped them and they were able to find what was mentioned in the error message.

ANOTHER VARIATION

Below is another flavor of the error which you might see.

Msg 15404, Level 16, State 19, Procedure xp_logininfo, Line 62
Could not obtain information about Windows NT group/user ‘ASIA\SQL_svc’, error code 0x5.

The above error is generally reported in SQL Agent job failure. Notice that error code is 0x5 which is in hexadecimal. This is equivalent to decimal 5 = Access is denied.

TIP AND TRICK

You need not remember the message text. If you now any message ID for windows error, you can convert it to text using net helpmsg command.

The above can be fixed by changing the owner in job properties to ‘SA’ and it should fix 0x5 error.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – xp_logininfo – Msg 15404: Could not obtain information about Windows NT group/user, error code 0x2147

There are many DBAs who like to automate the deployment of SQL. It is not fun to use wizard in all the installations. Imagine that setup media is kept in a remote share and all you need to do is provide a batch file to run and it would take care of installation. Let us learn How to Install SSMS From Command Line?

One of my client was trying to do the same. They wanted to deploy SQL Server Management Studio, SSMS, via command line. They contacted me and informed that installation via command is not working and they are getting an error. I asked to share command and the log file.

Here is the command to install Basic SSMS (no advanced features)

\\INDIA-FS1\Softwares\Microsoft\”SQL Server”\”SQL 2014″\”Enterprise Edition”\setup.exe /ACTION=Install /Q /IACCEPTSQLSERVERLICENSETERMS /FEATURES=Conn,BC,SSMS

Here was the error on command line (in my lab server for SQL 2016)

The following error occurred:
The specified value for setting ‘MEDIALAYOUT’ is invalid. The expected values are:

WORKAROUND/SOLUTION

My client was using double quotes to make sure that spaces are handled. But instead of putting quotes around each folder having space, they need to put the complete setup.exe path in double quotes. Here is the modified version of the command line.

“\\INDIA-FS1\Softwares\Microsoft\SQL Server\SQL 2014\Enterprise Edition\setup.exe” /ACTION=Install /Q /IACCEPTSQLSERVERLICENSETERMS /FEATURES=Conn,BC,SSMS

After using the above command, they were able to install SSMS and other components.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – How to Install SSMS From Command Line? Error: The Specified Value for Setting ‘MEDIALAYOUT’ is Invalid.

While I work on performance tuning most of the time, I do provide consultancy on issues which can be resolved quickly. These are the engagement where I get a lot of content for my blogs as well. Let us learn about the error related to redo log record.

One of my clients contacted me when they were not able to start their SQL Service. Without wasting any time, I got engaged with them and ask to share Errorlog to see the cause. Below is the error which I could see from the ERRORLOG.

2017-05-22 04:22:55.97 spid7s Starting up database ‘master’.
2017-05-22 04:22:55.98 spid7s Error: 3456, Severity: 21, State: 1.
2017-05-22 04:22:55.98 spid7s Could not redo log record (2579:456:5), for transaction ID (0:151685), on page (1:375), database ‘master’ (database ID 1). Page: LSN = (2579:424:5), type = 1. Log: OpCode = 4, context 2, PrevPageLSN: (2579:368:3). Restore from a backup of the database, or repair the database.

Since the master database was having corruption, there were not many choices left. The error message clearly says what we need to do. The message generally comes when there is a corruption in the transaction log file of the database. In our case, it was master database, which is one of the system database and due to that SQL was not getting started.

WORKAROUND/SOLUTION

To come out of the situation, below are the options.

• Restore from backup
• Rebuild System DBs
• Reinstall SQL Server

If we choose to restore from backup, then first we need to rebuild system databases from setup.exe and then perform disaster recovery procedures. Below is the command which can be used to rebuild system databases. We have put instance name as MSSQLServer because it was a default instance. For named instance, we need to use instance name. Make sure to take backup of existing MDF and LDF at a safe location before rebuilding.

setup /ACTION=REBUILDDATABASE /QUIET /INSTANCENAME=MSSQLSERVER /SQLSYSADMINACCOUNTS=”Domain\LoginName” /SAPWD=P@ssw0rd

It completed successfully and SQL was up and running. Now, this SQL server was as fresh as a new installation with no information about the earlier configuration of SQL Server. My client was having a hard luck as they were not having a backup of any system databases. All they had was MDF and LDF file of their user database. We tried to attach the user database but got the below error.

CREATE FILE encountered operating system error 5(Access is denied.) while attempting to open or create the physical file ‘S:\MSSQL2016\MSSQL14.MSSQLSERVER\MSSQL\Data\AX_PROD.mdf’. (Microsoft SQL Server, Error: 5123)

Then we went to the file properties and saw that this file was orphaned file as there was no owner. We provided SQL Startup account as the owner of this file. Once permission was set correctly, we were able to attach the database files and bring user database online.

Later, we stopped SQL Service and replaced database files for MSDB and model database and started SQL again. This time they were lucky and MSDB was clean so all information in MSDB came back. They were able to see the jobs etc. There were not many logins in the master database so they were able to recreate them and application was back online.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – FIX : Error: 3456, Severity: 21 – Could not redo log record (xx:xx:xx), for transaction ID (x:xx), on page (x:xx), database ‘master’

There are some weird situations which I see in SQL Server. This is also a one of the surprising situation. One of my clients reported via below screen shot. Let us see the issue of Master Database in Single User Mode.

Interesting enough, there was no issue with SQL functionality. It was acting just normally which means that we were able to make connections to SQL Server without any issues. They said that there are no users complains and users were able to get multiple connections to the Master as well.

We tried to switch it back to multi_user both in T-SQL and GUI, we get the same error:

Msg 5058, Level 16, State 5, Line 1
Option ‘MULTI_USER’ cannot be set in database ‘master’.

Since, it was interesting, I asked to check sp_helpdb for master and check the same in sys.database catalog views.

Both of the above have conflicting information. While this is an unexpected behavior, how do we come out of this situation?

WORKAROUND/SOLUTION

Here is the solution I offered them.

• Take the backup of master database using the backup database command.
• Restored the backup taken in the previous step as a user database. Let’s say master_one.
• The use ALTER command and get master_one in multiuser mode.
• Now take the backup of that user database (master_one) and then restored it in master using the below command. To restore the master database, we need to start SQL in single user mode.

Restore database master from disk= 'C:\Temp\Master_one.bak' with replace

Once they followed the above steps, the master database was back to multi user mode. My client was not sure how they ended up in such situation.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Master Database in Single User Mode – A Serious Issue

Along with performance tuning, I do provide consultancy services for deployments, including AlwaysOn availability group deployments. One of my clients who had a multi GB database wanted to configure AlwaysOn availability group and add a replica for disaster recovery purpose as well. They already had a secondary replica in the same data center for high availability purpose. Let us learn about AlwaysOn Join Error.

Since its not advisable to use UI for such big database to initialize the secondary, they were initializing secondary by will manual backup / restore of backup from primary followed by “join” option in the wizard. Whenever they tried this, below was the error.

Msg 1408, Level 16, State 211
The remote copy of database “SQLAUTH” is not recovered far enough to enable database mirroring or to join it to the availability group. You need to apply missing log records to the remote database by restoring the current log backups from the principal/primary database.

They contacted me because they were sure that there are no backups taken on primary replica, even then, somehow, they were seeing the error 1408 while trying to join the database to AG. They knew that the error is reported when there is a log backup taken from primary replica and there is a mismatch of LSN between the restored copy of secondary and current copy on the primary.

WORKAROUND/SOLUTION

While they knew that the error is due to the LSN mismatch, they missed one of the core concepts of backup taken from the AlwaysOn availability database.

It was easy to explain and demonstrate that they error would appear when a log backup is taken, after the full back-up from primary, even on a secondary replica. I used my below blog to find the history of the backup of the database.

When we ran the same query on the current secondary replica (remember, they already had one secondary) we found that log backups are taken regularly on that replica. As soon as we restored all the pending log backups to new secondary and tried join option again – it worked!

So whenever you see such errors, always use the script from my blog and find out the log backups which are taken. These log backups can happen even on secondary replica and would still maintain the chain with another log backup on other replicas.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – AlwaysOn Join Error – Msg 1408, Level 16 – The Remote Copy of Database “SQLAUTH” is Not Recovered Far Enough to Enable Database Mirroring or to Join it to the Availability Group

In SQL Server 2012 onwards, the cluster health check detection logic has been enhanced. Instead of the traditional pull mechanism of the cluster (IsAlive and LooksAlive), SQL Server (version 2012 onwards) uses a push mechanism to detect the health of the SQL instance. This is done by special stored procedure called sp_server_diagnostics. We should remember that the failover mechanism for the AlwaysOn FCI and the AlwaysOn Availability Groups is same.

While troubleshooting, it is very important to know which log to look at along with the basics of a feature. Recently, one of my clients was having trouble in bringing SQL Server AlwaysOn availability group resource online.

Whenever I am stuck with a cluster related issue, I always look at cluster log. You can refer my previous blog about how to generate cluster logs. SQL SERVER – Steps to Generate Windows Cluster Log?

Now let us change see the error logs.

Now let us inspect them heavily and you will find following error.

The User Does Not Have Permission to Perform this Action. (297)

I have removed date-time column from the output to provide the clarity. If we look at series of messages, you would notice that cluster has made a connection to SQL Server. After this it executes below statement

exec sp_server_diagnostics 10

As we can see in the next line that this execution failed with error.

The user does not have permission to perform this action

Due to above error diagnostic health check failed and SQL Server will not be able to come online in a cluster. Same error can also appear in the AlwaysOn availability group as well.

Now the real question is which user? And how to fix this issue?

WORKAROUND/SOLIUTION

The account which is used to connect to SQL Server from a cluster is a local system account. My client informed that due to hardening they have modified default permissions in SQL Server.

To fix this issue, we can add VIEW SERVER STATE permission to the SYSTEM account.

use [master]
GO
GRANT VIEW SERVER STATE TO [NT AUTHORITY\SYSTEM]
GO

Once done, the issue was resolved and SQL came online in the cluster as well.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – sp_server_diagnostics – The User Does Not Have Permission to Perform this Action. (297)

One of my client was unable to install SQL Server 2014 on a Windows 2012 Cluster. They were using mount points with below the structure. Let us learn about installation error.

• M:\Data_MP
• M:\Log_MP
• M:\System_MP
• M:\Temp_MP
• M:\Backup_MP

Over here, M was 10GB local drive, Data_MP is a SAN disk mounted in the root drive. As we can see they have configured disks from the SAN as mount points. The validation is getting successful, but the SQL installation is failing with below error.

Error description: The resource ‘Cluster Disk 1’ could not be moved from cluster group ‘Available Storage’ to cluster group ‘SQL Server (MSQLSERVER)’. Error: There was a failure to call cluster code from a provider. Exception message: Generic failure . Status code: 183. Description: Cannot create a file when that file already exists

When I looked at the Detail.txt file for more details, the message is same.

WORKAROUND/SOLUTION

There are two workarounds which I found for them

1. Pre-create a Windows Cluster group/Role in Cluster and add the disks to that group before starting the setup. During the setup wizard, choose this group for SQL to be installed.
2. Mount the mount points in a folder instead of Root drive like below
   • M:\UserDB_Data\Data_MP
   • M:\UserDB_Log\Log_MP
   • M:\SystemDB\System_MP
   • M:\TempDB\Temp_MP
   • M:\Backup\Backup_MP

Over here, M was 10GB local drive, UserDB_Data is a folder inside M and Data_MP is SAN disk mounted inside folder.

Hope this would help someone in the world!

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Installation Error – Status Code: 183 Description: Cannot Create a File When that File Already Exists

So far, I have written many blogs and I still feel that there are many errors and scenarios which are still uncovered. Whenever I see a new error and find a new solution, I always share it via this blog.  Let us see how we can fix errors related to sp_xp_cmdshell_proxy_account.

One of my clients was using below blogs to use proxy to run xp_cmdshell.

SQL SERVER – Fix: The EXECUTE Permission was Denied on the Object ‘xp_cmdshell’, Database ‘mssqlsystemresource’,schema ‘sys’.

SQL SERVER – Enable xp_cmdshell using sp_configure

They informed that they are getting errors while using sp_xp_cmdshell_proxy_account. Here is the first error:

Msg 15137, Level 16, State 1, Procedure sp_xp_cmdshell_proxy_account, Line 1 [Batch Start Line 5] An error occurred during the execution of sp_xp_cmdshell_proxy_account. Possible reasons: the provided account was invalid or the ‘##xp_cmdshell_proxy_account##’ credential could not be created. Error code: 1326(The user name or password is incorrect.), Error Status: 0.

I was able to reproduce the error by running below (i.e. by giving wrong password)

EXEC sp_xp_cmdshell_proxy_account 'domain\user', 'WrongPassword'

Now, I have given the right password as below

EXEC sp_xp_cmdshell_proxy_account 'domain\user', 'RightPassword'

But got below error.

Msg 15137, Level 16, State 1, Procedure sp_xp_cmdshell_proxy_account, Line 1 [Batch Start Line 6] An error occurred during the execution of sp_xp_cmdshell_proxy_account. Possible reasons: the provided account was invalid or the ‘##xp_cmdshell_proxy_account##’ credential could not be created. Error code: 5(Access is denied.), Error Status: 0.

WORKAROUND/SOLUTION

In both errors above, even though error message is the same, the Error Code is different. First one has 1326 which was due to incorrect password. The second one has error code of 5, which means access denied. To fix this error, I should launch SSMS as Run as Administrator as shown below.

The above image is taken from the Windows 2016 Operating system, start menu. You might have a different interface, but you get an idea.

Let me know what you think of this blog post by leaving comments below.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Msg 15137, Level 16 – Error Related to sp_xp_cmdshell_proxy_account

Recently, one of my client was trying to troubleshoot the below error which they were getting while trying to start the SQL Server service under the domain account. The interesting thing which they told me was that they are using Read Only Domain Controller (RODC) which I have heard earlier as well. Initially they told that they had an issue with installation when service account was used so they used LocalSystem account and installed SQL Server. But now, as per company standards, they need to use a domain account as the service account.

I asked them to share ERRORLOG with me. Here is the link how you can find the error log location: SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location

Here are the errors which I found in Error Log.

2017-06-12 11:55:21.46 spid10s Unable to initialize SSL encryption because a valid certificate could not be found, and it is not possible to create a self-signed certificate.
2017-06-12 11:55:21.46 spid10s Error: 17182, Severity: 16, State: 1.
2017-06-12 11:55:21.46 spid10s TDSSNIClient initialization failed with error 0x80092004, status code 0x80. Reason: Unable to initialize SSL support. Cannot find object or property.
2017-06-12 11:55:21.46 spid10s Error: 17182, Severity: 16, State: 1.
2017-06-12 11:55:21.46 spid10s TDSSNIClient initialization failed with error 0x80092004, status code 0x1. Reason: Initialization failed with an infrastructure error. Check for previous errors. Cannot find object or property.
2017-06-12 11:55:21.46 spid10s Error: 17826, Severity: 18, State: 3.
2017-06-12 11:55:21.46 spid10s Could not start the network library because of an internal error in the network library. To determine the cause, review the errors immediately preceding this one in the error log.
2017-06-12 11:55:21.46 spid10s Error: 17120, Severity: 16, State: 1.
2017-06-12 11:55:21.46 spid10s SQL Server could not spawn FRunCommunicationsManager thread. Check the SQL Server error log and the Windows event logs for information about possible related problems.

WORKAROUND/SOLUTION

Based on my research with SQL Server, RODC and encryption, I found that this is reported by many clients on various forums and as per them it started after installation of KB 2992611 or KB 3000850. To work around this issue, you will need to perform the following registry key changes.

We need to create the following key:

• Path : HKLM\Software\Microsoft\Cryptography\Protect\Providers\df9d8cd0-1501-11d1-8c7a-00c04fc297eb
• Name: [ProtectionPolicy] (without square brackets !)
• Value : 1 (DWORD)

When I searched my own blog history, I found that I another blog which talks about the setup issue SQL SERVER – System. Security. Cryptography. CryptographicException – There Was an Error Generating the XML Document

The solution in that blog is also same.

Have you encountered same error on RODC? Please comment and let me know.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Startup Issue – Unable to Use Domain Account as Service Account When Read Only Domain Controller (RODC) is Involved

Many customers want to do in-place upgrade of SQL Server instance to avoid the hardware cost and downtime. There are various pros and cons for doing in-place upgrade or parallel upgrade and one need to evaluate them before making a final choice. Let us see an error related to Upgrade Failure.

My client contacted me for an issue where in-place upgrade of a clustered instance was failing. I always believe that we should always start with error message while troubleshooting any issue. So, I asked for setup logs to investigate. If you are familiar with setup logs, there are two basic files, Summary.txt and Detail.txt. When I looked at Summary.txt, I found below.

Reason for failure: An error occurred for a dependency of the feature causing the setup process for the feature to fail.
Next Step: The upgrade process for SQL Server failed. To continue the upgrade process, use the following information to resolve the error. Next, uninstall SQL Server by using this command line: setup /q /action=uninstall /instanceid=MSSQLSERVER /features=SQLENGINE,FULLTEXT,REPLICATION. Then, run SQL Server Setup again.

SOLUTION/WORKAROUND

The information which is useful here is “Reason for failure”, “Error description” and “Next Step”. By looking at above information, it’s easy to come to the conclusion that there is something wrong with SSAS resource in the cluster. When we looked into the cluster, we found that there was no Analysis services resource on the cluster. We followed the Microsoft article to create SSAS resource manually in the failover cluster manager. We created a generic service resource and added the dependency on SQL Network name and disks. After that, we could bring SSAS resource online in a cluster.

Now, as mentioned in the log, we ran below command

setup /q /action=uninstall /instanceid=MSSQLSERVER /features=SQLENGINE, FULLTEXT, REPLICATION

I think this command would have taken care of removing various leftovers caused due to last unsuccessful installation.

After this, we ran setup.exe again and upgrade went fine.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Upgrade Failure – The Cluster Resource is Not Online. Bring the Analysis Services Server Online Before Starting the Upgrade Process

In our real life, there are lots of dependencies and same is true with SQL Server as well. This blog also talks about inability to drop a login due to a dependency. Let us see the error related to login is the owner of the 1 job(s).

When you try to drop a login from SQL Server, you might run into the error. Here is what I received when I was trying to drop a login called “foo” from the SQL Server Management Studio.

This login is the owner of 1 job(s). You must delete or reassign these jobs before the login can be dropped. (Microsoft SQL Server, Error: 15170)

If the same action is tried from below T-SQL:

USE [master]
GO
DROP LOGIN [foo]
GO

Then we get this:

Msg 15170, Level 16, State 1, Line 4
This login is the owner of 1 job(s). You must delete or reassign these jobs before the login can be dropped.

WORKAROUND/SOLUTION

As mentioned in the error message, this is due to dependency of job on the account which we are trying to delete. We need to find out such jobs so that suggested action can be taken.

SELECT name,*
FROM msdb..sysjobs
WHERE owner_sid = SUSER_SID('foo')

You need to replace “foo” which the user name which you are trying to delete. The output of the query would show the jobs which are owned by this login. The next action would be to change the job owner as shown below.

Have you faced any similar errors? If this blog has helped you in learning something new, please comment as let me know.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: Msg 15170, Level 16, This login is the Owner of 1 Job(s). You Must Delete or Reassign these Jobs Before the Login can be Dropped

With Microsoft Azure pitching in the market, I am getting few clients who want me to assist them for errors related to Azure as well. As a part of growth, I do learn them and try to help the clients. Let us learn about Block Blobs in this blog post.

My client was migrating their on-premise SQL Severs to SQL running under Azure virtual machines. They have decided to move database via backup restore.

• Source SQL Server (On-Premise): SQL 2012 Standard.
• Destination SQL Server (SQL on Azure VM): SQL 2014 Enterprise

While restoring from the Azure storage using “Restore … from URL” command, we were seeing below error in ERRORLOG.

BackupVirtualDeviceFile::DetermineFileSize: SetPosition(0,EOF) failure on backup device ‘https://sqlauth storagehdd.blob.core.windows.net/sqlbackup/DBName.bak’. Operating system error The specified URL points to a Block Blob. Backup and Restore operations on Block Blobs are not permitted.

BackupVirtualDeviceFile::DetermineFileSize: SetPosition(0,EOF) failure on backup device ‘https://sqlauth storagehdd.blob.core.windows.net/sqlbackup/DBName.bak’. Operating system error The specified URL points to a Block Blob. Backup and Restore operations on Block Blobs are not permitted.

I asked the method to transfer the data from the source server to Azure. They told that they have used AzCopy to copy the data from the local machine to Azure storage. So essentially, database backup was taken locally on On-Prem machine and then AzCopy was used to copy the backup files. They have not used “BACKUP … to URL”

Transfer data with the AzCopy on Windows

I looked into documentation of SQL 2012 and SQL 2014 and found below.

SQL Server Backup and Restore with Windows Azure Blob Storage Service

Blob: A file of any type and size. There are two types of blobs that can be stored in the Windows Azure Blob storage service: block and page blobs. SQL Server backup uses page Blobs as the Blob type. Blobs are addressable using the following URL format: https://<storage account>.blob.core.windows.net/<container>/<blob>

Separately, in SQL Server 2014, we were able to see the same limitation listed, but under a different topic: SQL Server Backup to URL

Blob: A file of any type and size. There are two types of blobs that can be stored in the Windows Azure Blob storage service: block and page blobs. SQL Server backup uses page Blobs as the Blob type. Blobs are addressable using the following URL format: https://<storage account>.blob.core.windows.net/<container>/<blob>

SOLUTION / WORKAROUND

It looks like, the backup file was turned into a block blob, and that explains the cause of the error during restore. I think the above is implying that the transfer tool (AZCopy) may have been configured to use block blob to transfer the backup file by default. After reading further about the tool, I found there is a parameter called /BlobType:Page. Once we used that parameter, restore worked like a charm.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Unable to Restore from URL – The Specified URL Points to a Block Blob. Backup and Restore Operations on Block Blobs are not Permitted

In the recent past, I have written a blog about Ransomware. Here is the link. SQL SERVER – How to Protect Your Database from Ransomware?. In this blog post we are going to discuss about system database.

One of my client was hit with Ransomware and they contacted me. Luckily, they had database backups in another location and all they wanted was to build a new server and go back to the old state. They have reinstalled same version of operating system and installed the same version of SQL Server. Now, to go back to a previous state, they wanted to restore all databases from backup. While restoring master, they were getting error related to SQL version.

Msg 3168, Level 16, State 1, Line 1
The backup of the system database on the device cannot be restored because it was created by a different version of the server (10.50.2550) than this server (10.50.2500).
Msg 3013, Level 16, State 1, Line 1
RESTORE DATABASE is terminating abnormally.

The error message is about version mismatch. If you get the same error while restoring master (or msdb or models), you need to make a note of two things.

1. The version on which backup was taken. (First build number in error message)
2. Current version of SQL Server. (Second build number on error message.

WORKAROUND/SOLUTION

To restore any system database backup, #1 and #2 should match. If they don’t, then SQL would raise error message during restore. You should refer to http://sqlserverbuilds.blogspot.in and find the desired build number and download it.

• If the current bid is lower and apply the desired patch.
• If the current bid is higher than uninstall the patch (or patches) till both become same.

Hope this would help. If you have more ideas and thoughts, please share via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Msg 3168, Level 16 – The Backup of the System Database on the Device Cannot be Restored Because it was Created by a Different Version of the Server

I have been dealing with a client who observed that SQL services not starting up after a server reboot. When I asked them to check event logs, they found below timeout error.

Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7009
User: N/A
Computer: SQLMachine
Description:
Timeout (30000 milliseconds) waiting for the SQL Server (MSSQLSERVER) service to connect.

They also informed me another interesting fact. Once the reboot is completed, they can login to machine and when they start SQL Service manually, it starts fine. So, it looks like the system is very busy during startup of the machine and within 30 seconds there was no response from SQL to service control manager. Due to this the error message was reported in the event log. “Timeout (30000 milliseconds) waiting for the SQL Server (MSSQLSERVER) service to connect.”

WORKAROUND/SOLUTION

If this issue happens only during startup and service starts fine while starting manually then we can try to increase the timeout from 30 seconds to 60 seconds. Here are the steps to change the timeout value.

1. Click start->Run-> type regedit, and then click ok.
2. Browse through the registry and then click the following registry sub key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control
3. Right-click on Control, point to New, and then click on DWORD Value.
4. In the New Value #1 box, type ServicesPipeTimeout, and then press ENTER.
5. Right-click ServicesPipeTimeout, and then clock Modify.
6. Click Decimal, type the number of milliseconds that we want to wait until the service times out, and then click ok. For my client, we changed it to wait 60 seconds [60000 milliseconds])
7. Quit Registry Editor, and then restart the computer.

For my client, there were many services and system performance was slow during startup. Adding the key helped them in getting the service started on reboot.

Have you ever used such trick to bypass performance issue? Please leave a comment.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Error: Timeout (30000 milliseconds) Waiting for the SQL Server (MSSQLSERVER) Service to Connect