Few of my clients are still running older versions of SQL Server and they have their own reasons to not to move to the new SQL version. In this blog, we would learn about the fix of an error for which my client contacted me. My client was trying to apply the patch to a SQL Server 2008 R2 clustered instance which was running on Window Server 2012 and getting the error: Unable to retrieve the Cluster Service – Not clustered or the cluster service is up and online.

First, I asked to share Detail.txt to understand the exact error. Here is the information available. I have added a bullet point for clarity.

• Slp: Unable to retrieve the Cluster Service
• Slp: System.InvalidOperationException: The given key was not present in the dictionary. —> System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
• ThrowHelper.ThrowKeyNotFoundException()
• Collections.Generic.SortedList`2.get_Item(TKey key)
• SqlServer.Chainer.Infrastructure.ServiceContainer.GetService(Type serviceType)
• — End of inner exception stack trace —
• SqlServer.Chainer.Infrastructure.ServiceContainer.GetService(Type serviceType)
• SqlServer.Chainer.Infrastructure.ServiceContainer.GetService[T]()
• SqlServer.Chainer.Infrastructure.ServiceContainer.get_Cluster()
• SqlServer.Configuration.SetupExtension.RunDiscoveryAction.ExecuteAction(String actionId)

One of the causes of the above error is found earlier which I shared via below blog. SQL SERVER – Patch Rule Failure: Not Clustered or the Cluster Service is Up and Online

The blog above should be used when the cluster is in a bad shape. But in the current situation, my client had cluster working fine. It was just the SQL patch setup which was not able to talk to the cluster.

WORKAROUND/SOLUTION

I knew that there are articles from Microsoft which talks about but I was not getting any hit with the error which I saw in Detail.txt

Finally, we found that “Failover Cluster Automation Server” feature was not installed. This is default setting when install windows cluster service, but the feature is needed when installing the SQL Server 2008 R2 patch. Here is the PowerShell command to add the feature.

Add-WindowsFeature -Name RSAT-Clustering-AutomationServer

Finally, I found KB which talks about the solution.

Here is the snippet from the article.

To resolve this issue, do one of the following:

• Enable the Failover Cluster Automation Server feature on each node by using Server Manager. In Server Manager, expand Remote Server Administration Tools, expand Feature Administration Tools, expand Failover Clustering Tools, and then click to select Failover Cluster Automation Server.

Here is the PowerShell screenshot while installing it. I have used below command to view the feature.

Get-WindowsFeature RSAT-Cluster*

Hope this blog help in finding the solution of patch failure. Please comment and let me know.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Patch Failure – Unable to Retrieve the Cluster Service

Once upon a time, I was trying to remove TempDB database additional files, but it was not working. In this blog I would explain the steps needed fix Msg 2555: Cannot move all contents of the file to other places to complete the emptyfile operation.

THE SITUATION

My client installed SQL Server using an automated method which kept tempdb database files on C Drive. As a good practice, he decided to add new files to T drive and remove files from C drive. While removing the files from C drive, they encountered below error.

Msg 2555, Level 16, State 1, Line 18
Cannot move all contents of file “tempdata1” to other places to complete the emptyfile operation.

The command which they were running was below.

USE [tempdb]
GO
DBCC SHRINKFILE (N'tempdb1', EMPTYFILE)
GO
ALTER DATABASE [tempdb] REMOVE FILE [tempdb1]
GO

When I double checked the messages window and I also found below.

DBCC SHRINKFILE: Page 3:39 could not be moved because it is a work table page.

Above is the main cause why SQL was not able to empty the file and remove it from TempDB database.

WORKAROUND/SOLUTION

Based on various other blogs on the internet, below T-SQL queries were tried and they DID NOT help.

DBCC DROPCLEANBUFFERS;
GO
DBCC FREESYSTEMCACHE ('ALL');
GO
DBCC FREEPROCCACHE;
GO
DBCC FREESESSIONCACHE;
GO

Note: Even restart of SQL Service was not helping us here and we were getting the same message for page 3:39

I remember writing a blog about the work table related error message for the tempdb database.

SQL SERVER – DBCC SHRINKFILE: Page 1:26423878 Could not be Moved Because it is a Work Table Page – TempDB Files

In short, starting SQL with a minimal configuration using “f” startup parameter allowed us to get rid of extra files. Have you faced such a situation? What solution worked for you?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Msg 2555: Cannot Move All Contents of File to Other Places to Complete the Emptyfile Operation

One of my existing clients has sent me an interesting question. In this blog, I would share one interesting situation where my client has created an availability group and it was visible in SSMS, but no role was visible in failover cluster manager.

Here is the picture which you explain what we were seeing.

Generally, when an availability group is created, an AG resource is created in a Windows cluster.

WORKAROUND/SOLUTION

The question which was asked by my client was perfectly valid. It is possible to run into the situation if we do below steps.

1. Enable feature of AlwaysOn availability group without creating windows cluster.
   
2. Create Availability Group and choose NONE as cluster type as there is no option called “NONE” as cluster type in below screen.
   
3. Now, enabled windows clustering feature and create windows cluster.

At this point, you can open cluster manager and would see no availability group role because availability group was created as Cluster Type None.

So, if you ever see such situation, check the Dashboard of Always On availability group and have a look at cluster type which is highlighted as below.

This feature is called as a Read Scale and you can read more about this at below link Use read-scale with Always On availability groups

Hope you found this information useful. Please comment and let me know.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Availability Group Seen in SSMS but missing in Cluster Manager. What’s Wrong?

I am not sure what changes I made in one of my Virtual Machine, but I was running into an issue. In this blog, we would discuss how to fix error “The ‘SqlStudio Profile Package’ package did not load correctly” while launching Visual Studio.

I checked into the file mentioned but didn’t find anything interesting to tell me the solution.

WORKAROUND/SOLUTION

I search on the internet and found various solutions. In this blog, I would share the consolidated list of solutions.

1. Based on one of the StackOverFlow link, delete below file.

%UserProfile%\AppData\Local\Microsoft\VisualStudio\11.0\ComponentModelCache

2. Rename below folders

C:\Users\<UserName>\AppData\Roaming\Microsoft\VisualStudio\11.0

C:\Users\<UserName>\AppData\Local\Microsoft\VisualStudio\11.0

3. Download and install SSDT from below this link.
4. Go to ‘Add/Remove Programs’ in Control Panel. Select “Microsoft SQL Server 2012 Data-Tier App Framework”, right click and choose “Repair”. If you find more versions of it, do the repair for all of them.

For me, option # 1 worked. If you face the same problem, please comment and let me know which option works for you. If you find any other solution, please comment and share with others.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on Visual Studio – FIX: The ‘SqlStudio Profile Package’ Package Did Not Load Correctly

As they say – There are more ways than one to skin a cat. In this blog we will find another method to fix an error which you might see during SQL upgrades – Cannot drop the assembly ‘ISSERVER’, because it does not exist or you do not have permission.

Here are the messages in SQL Server ERRORLOG when the upgrade was unable to start SQL Service. I have added bullets to have the readability of messages in ERRORLOG.

• Taking SSISDB to single user mode
• Setting database option SINGLE_USER to ON for database ‘SSISDB’.
• ———————————
• Starting assembly upgrade
• ———————————
• Altering assembly [ISSERVER]
• Attempting to load the library ‘xpstar.dll’ into memory. This is an informational message only. No user action is required.
• Using ‘xpstar.dll’ version ‘2014.120.5556’ to execute extended stored procedure ‘xp_regread’. This is an informational message only; no user action is required.
• Error: 50000, Severity: 16, State: 127.
• Cannot drop the assembly ‘ISSERVER’, because it does not exist or you do not have permission.
• Error: 50000, Severity: 16, State: 127.
• Cannot drop the assembly ‘ISSERVER’, because it does not exist or you do not have permission.
• Creating function internal.is_valid_name
• Error: 6528, Severity: 16, State: 1.
• Assembly ‘ISSERVER’ was not found in the SQL catalog of database ‘SSISDB’.
• Error: 912, Severity: 21, State: 2.

I have written earlier blog having one of the solutions to fix the error. SQL SERVER – Assembly ‘ISSERVER’ Was Not Found in the SQL Catalog of Database ‘SSISDB’

In athe bove blog post, a situation was easy where SSIS catalog was not in use. However, for my current client asked not to drop catalogs as he was not sure who is using it.

WORKAROUND/SOLUTION

If we have another server with the same version and having SSISDB installed, then we can perform below steps to recreate the assembly. We need to make sure that the version and patch level of SQL Server is matching.

1. Start the problematic SQL with trace flag 902.
2. Connect to SQL instance with working SSISDB database
3. Expand SSISDB, Programmability, Assemblies
4. Highlight ISSERVER, Right-click and choose Script Assembly as CREATE to New Query Editor Window
   
5. Copy the CREATE ASSEMBLY statement and paste it into a query window on the problematic server
6. Execute the CREATE ASSEMBLY statement

Note that the text, which you would copy, is going to be very long as it’s the complete .NET code in binary format.

7. Once an assembly is created stop SQL and the start it without trace flag 902.

Is there any other way to create assembly ISSERVER? If I find it, I would share in another blog. If you find it, please comment and share with the SQL community.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Cannot Drop the Assembly ‘ISSERVER’, Because it Does not Exist or You do not Have Permission

One of my clients contacted me via my On Demand (50 Minutes) offering and they were seeing below error message in ERRORLOG – New parallel operation cannot be started due to too many parallel operations executing at this time.

Here is the complete message which was seen in the SQL Server ERRORLOG file.

New parallel operation cannot be started due to too many parallel operations executing at this time. Use the “max worker threads” configuration option to increase the number of allowable threads, or reduce the number of parallel operations running on the system.

I asked for complete ERRORLOG and Event Logs to look it from all possible angles.

• Error: 18210, Severity: 16, State: 1.
• BackupIoRequest::ReportIoError: write failure on backup device ‘{BD27F651-5DB3-4CB4-9615-9FDEC4D8EECE}331’. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).

At the same time, I found below in event log

• SQLVDI: Loc=TriggerAbort. Desc=invoked. ErrorCode=(0). Process=1724. Thread=2300. Server. Instance=MSSQLSERVER. VD=Global\{FFC0C1C0-25D9-4A90-82B3-3ABBCFEA9476}247_SQLVDIMemoryName_0.
• BACKUP failed to complete the command BACKUP DATABASE ReportingService_Alerting. Check the backup application log for detailed messages.
• SQLVDI: Loc=TriggerAbort. Desc=invoked. ErrorCode=(0). Process=1724. Thread=9844. Server. Instance=MSSQLSERVER. VD=Global\{FFC0C1C0-25D9-4A90-82B3-3ABBCFEA9476}45_SQLVDIMemoryName_0.
• SQLVDI: Loc=TriggerAbort. Desc=invoked. ErrorCode=(0). Process=1724. Thread=8176. Server. Instance=MSSQLSERVER. VD=Global\{FFC0C1C0-25D9-4A90-82B3-3ABBCFEA9476}264_SQLVDIMemoryName_0.

From above it was clear that they are not taking native backups of SQL Server database using a maintenance plan of T-SQL. The backups are being taken using SQLVDI via 3^rd party software.

WORKAROUND/SOLUTION

Looking at the messages, it sounded like the error appears at the same time when a backup is running. Further investigation showed that they had more than 1000 databases and they were getting backed up via 3^rd party tool at the same time. I suggested them to

1. Talk to back up team and find a way to stagger the backups. This would not cause too many parallel threads for backups at the same time.
2. Increase the number of worker thread to avoid message in ERRORLOG.

Once they reduce many concurrent backups, error stopped.

Have you seen such messages earlier? What did you do to solve? Please share via comments with others in the SQL community.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – New Parallel Operation Cannot be Started Due to Too Many Parallel Operations Executing at this Time

A few days back, I published the blog on the same error message and one of the blog reader responded and ask how can I generate the script if I don’t have another SQL Server? In this blog, we would learn how to fix error Cannot drop the assembly ‘ISSERVER’, because it does not exist or you do not have permission.

Here is the earlier blog for your reference which talks about one solution to fix the error.

SQL SERVER – Cannot Drop the Assembly ‘ISSERVER’, Because it Does not Exist or You do not Have Permission

SQL ERRORLOG shows this:

Setting database option SINGLE_USER to ON for database ‘SSISDB’.
———————————-
Starting assembly upgrade
———————————-
Altering assembly [ISSERVER]
Error: 50000, Severity: 16, State: 127.
Cannot drop the assembly ‘ISSERVER’, because it does not exist or you do not have permission.
Error: 50000, Severity: 16, State: 127.
Cannot drop the assembly ‘ISSERVER’, because it does not exist or you do not have permission.
Creating function internal.is_valid_name
Error: 6528, Severity: 16, State: 1.
Assembly ‘ISSERVER’ was not found in the SQL catalog of database ‘SSISDB’.

I mentioned in an earlier blog to script the assembly from some other server and create it on a problematic server. One of my blog readers said –

I don’t have any other instance of SQL Server with the same version. Can I not use some other method?”

This was indeed a very valid ask. I spent some time on the internet, captured profiler while deploying the SSISDB database and found below in the profiler.

DECLARE @path NVARCHAR(255)
DECLARE @fullPath NVARCHAR(255)
EXEC master..xp_regread 'HKEY_LOCAL_MACHINE', @keypath, N'', @path OUTPUT
SET @fullPath = REPLACE( @path, '''', '''''' ) + N'Binn\Microsoft.SqlServer.IntegrationServices.Server.dll'
--PRINT(@fullPath);
-- Use OpenRowset to read the binary bits into @asm_bin
print ('DECLARE @asm_bin varbinary(max);
SELECT @asm_bin = BulkColumn FROM OPENROWSET (BULK N''' + @fullPath + ''',SINGLE_BLOB) AS dll
CREATE ASSEMBLY ISSERVER FROM @asm_bin WITH PERMISSION_SET = UNSAFE');

After running this, I got below statement as output

DECLARE @asm_bin varbinary(max);  
SELECT @asm_bin = BulkColumn FROM OPENROWSET (BULK N'd:\Program Files\Microsoft SQL Server\120\DTS\Binn\Microsoft.SqlServer.IntegrationServices.Server.dll',SINGLE_BLOB) AS dll
CREATE ASSEMBLY ISSERVER FROM  @asm_bin  WITH PERMISSION_SET = UNSAFE

The output might vary based on your SQL Server version.

Remember that you might be running into this when SQL is not getting started so you need to start SQL with trace flag 902 to bypass upgrade scripts, run CREATE ASSEMBLY and then restart SQL without trace flag. If you want to know about steps to enable trace flag, please read my earlier blog. SQL SERVER – What is Trace Flag – An Introduction

Please comment and let me know if this blog has helped you in solving upgrade issue.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Upgrade Error – Cannot Drop the Assembly ‘ISSERVER’, Because it Does not Exist or You Do Not Have Permission

This was indeed one of an interesting error which I have never seen earlier and it is about DAC Connection.

Here is the complete error message which my client reported.

Msg 3637, Level 16, State 3, Line 1
A parallel operation cannot be started from a DAC connection.
Msg 3013, Level 16, State 1, Line 1
BACKUP DATABASE is terminating abnormally.

By looking at the error message, I asked him how he is connecting to SQL Server and where exactly the error is appearing.

They informed that they have scheduled a backup job from Windows. Since it is an SQL Express edition, they created a batch file to take backup of the database. The batch file was running as a scheduled task via windows task scheduler. The error appears when a backup is initiated.

I asked them to show the batch file which is taking backup and it didn’t take much time to find what is going wrong while taking backup.

I was able to reproduce the error in my local lab environment as well.

WORKAROUND/SOLUTION

To reproduce the error, I connected to SQL Server via SQL Server Management Studio and gave the server name as “Admin:ServerName”. This caused the connection to go as Dedicated Admin Connection or DAC. This is not a normal connection to SQL Server via 1433 port. There are some limitations of DAC connection and unable to take a backup is one of them.

In my client’s situation, the .bat file had below.

SQLCMD -S <ServerName> -E -A -I”<path to backup script>

When I looked at the documentation, it says  [-A dedicated admin connection] and that explained everything. Here are my test results, with -A and without -A and you can see the difference.

In short, I asked the customer to change the script to remove “A” parameter to avoid DAC connection and since then the backups are running like a charm.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: 3637 – A Parallel Operation Cannot be Started From a DAC Connection

Sometimes while reproducing a situation results in another new error and I get an idea for a new blog. In this blog we would talk about an error Login-based server access validation failed with an infrastructure error. Login lacks connect endpoint permission.

The earlier blog was about connecting to SQL Server using Dedicated Admin Connection (DAC) in SQL Server. When I tried connecting to SQL using SQLCMD to DAC, I received below error.

Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login failed for user ‘daclogin’..

Here is the screenshot.

Above is one of the most common error which doesn’t tell the cause of the issue. Whenever you get such error, you should always look at SQL Server ERRORLOG SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location

In ERRORLOG, I saw the following message:

Error: 18456, Severity: 14, State: 149.
Login failed for user ‘daclogin’. Reason: Login-based server access validation failed with an infrastructure error. Login lacks connect endpoint permission. [CLIENT: 127.0.0.1]

WORKAROUND/SOLUTION

The key part of the error message was “Login lacks connect endpoint permission.”. I checked further and realized that this account “dacadmin” which I created was part of “public” role.

As soon as I gave him sysadmin, it was able to log in.

Have you encountered the same error in non-DAC connection also? What was the solution?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Error: 18456, State 149 – Login-based Server Access Validation Failed With an Infrastructure Error. Login Lacks Connect Endpoint Permission

This was the first experiment with gMSA account in my lab and I faced an interesting issue. In my lab environment, I have a complete domain server and member servers. Once I configured gMSA for SQL Server service and restarted the machine, SQL Service didn’t start automatically even though it was set for an automatic startup as shown below.

There was no ERRORLOG because SQL didn’t start. Event log showed me a bunch of Errors which I have listed below:

The interesting messages out of all are:

1. The specified domain either does not exist or could not be contacted.
2. The service did not start due to a logon failure.

WORKAROUND/SOLUTION

From the messages its clear that server was not able to contact the domain controller when it was getting started along with server startup. There are few things, which I am aware of, which would help.

1. Set SQL Server Service to “Automatic (Delayed Start)” as shown below.
   
2. Using registry editor, set the dependencies of SQL Server service on Netlogon and W32time service. Here are the steps:
   1. Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSSQLSERVER
   2. Look for “DependOnService” on the right pane.
   3. Edit the values and add W32Time and Netlogon as shown below. Note: KEYISO was already there.
      
   4. Close the settings and check via services to make sure dependency is set correctly.
      

After doing above, I never faced the same issue on this server.

Have you ever faced the same issue? Is there any other solution which you found? Please share via comments and I would write a blog with due credit.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – SQL Service Not Getting Started Automatically After Server Reboot While Using gMSA Account

Today’s blog post is directly inspired by the conversation I had during my Comprehensive Database Performance Health Check. During the consulting engagement, one of the developers reported below error in the database while trying to open a database diagram.

Cannot execute as the database principal because the principal ‘dbo’ does not exist, this type of principal cannot be impersonated, or you do not have permission. (Microsoft SQL Server, Error: 15517)

As per message, it is clear that something is not right with database principal dbo. I asked history about it and learned that this database was restored from some other server. I was able to run some queries and find the issue.

To verify if you are running into the same issue, you can run below query to find who is mapped to “dbo” in the database. My sample database name is AdventureWorks, so please change accordingly.

USE AdventureWorks
GO
SELECT SUSER_SNAME(sid), * from sys.database_principals

If the first column shows as NULL then below fix would work.

WORKAROUND/SOLUTION

This was just a different variation of error which I have explained in my earlier blog. SQL SERVER – ERROR: FIX – Database diagram support objects cannot be installed

We went ahead and changed the owner of the database in the UI (Properties > Files tab)

Here is the equivalent command.

USE [AdventureWorks]
GO
ALTER AUTHORIZATION ON DATABASE::[AdventureWorks] TO [sa]
GO

After doing this, the issue was resolved, and they were able to use this feature?

How many of you use this old feature? Truly speaking, I have not seen many using it in production.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: Database Diagram Error 15517 – Cannot Execute as the Database Principal Because the Principal ‘dbo’ Does Not Exist

After finishing my lab and testing, I went to my client and they showed me error while setting up cloud witness. They showed me a different error then earlier. In this blog, we would learn about fixing error Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host while configuring cloud witness.

I have written an earlier blog with a different error:

SQL SERVER – Unable to Set Cloud Witness. Error: The Client and Server Cannot Communicate, Because They do Not Possess a Common Algorithm

Here is the exact error message which was seen:

The error message clearly tells that there is some issue with connectivity between node and cloud witness.

WORKAROUND/SOLUTION

I always use Telnet or Test-NetConnection to test connectivity and here also it was helpful. I found that this server was not allowing internet outbound connections. As soon as I opened port 443 to storage, the issue was resolved.

For more details about an opening port in Azure VM, refer my earlier blog

SQL SERVER – What is the Meaning of PREEMPTIVE_HTTP_EVENT_WAIT? How to Fix it?

Even though the behavior is different, but the solution remains the same as above mentioned.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Unable to Set Cloud Witness. Error: An Existing Connection Was Forcibly Closed by The Remote Host

This was my first experiment with this feature called Stretch Database. I have started the wizard and encountered at the very end. Here is the error message.

And here is the text of error message.

Operation to create resource group stretchgroup-australiacentral failed. Details : {“error”:{“code”:”LocationNotAvailableForResourceGroup”,”message”:”The provided location ‘australiacentral’ is not available for resource group. List of available regions is ‘centralus,eastasia,southeastasia,eastus,eastus2,westus,westus2, northcentralus,southcentralus,westcentralus,northeurope,westeurope, japaneast,japanwest,brazilsouth, australiasoutheast,australiaeast,westindia,southindia,centralindia, canadacentral,canadaeast,uksouth,ukwest,koreacentral,koreasouth, francecentral,southafricanorth’.”}}

It also gave an option to read the logs and I found the same error there also.

1. [Informational] TaskUpdates: Message:Task : ‘Provision Azure Sql Server stretchserver-stretchdbdemo-20190317-031938571’ — Status : ‘Started’ — Details : ‘Task ‘Provision Azure Sql Server stretchserver-stretchdbdemo-20190317-031938571′ started ….’.
2. [Informational] TaskUpdates: Message:Task : ‘Provision Azure Sql Server stretchserver-stretchdbdemo-20190317-031938571’ — Status : ‘Running’ — Details : ‘Task failed due to following error: Microsoft.SqlServer.Management.StretchDatabase.Model.Tasks. CreateResourceGroupFailedException: Operation to create resource group stretchgroup-australiacentral failed. Details : {“error”:{“code”:”LocationNotAvailableForResourceGroup”,”message”:”The provided location ‘australiacentral’ is not available for resource group. List of available regions is ‘centralus,eastasia,southeastasia,eastus,eastus2,westus,westus2, northcentralus,southcentralus,westcentralus,northeurope,westeurope, japaneast,japanwest,brazilsouth,australiasoutheast,australiaeast, westindia,southindia,centralindia,canadacentral,canadaeast,uksouth, ukwest,koreacentral,koreasouth,francecentral,southafricanorth’.”}}

at Microsoft.SqlServer.Management.StretchDatabase.Model.Tasks. ProvisionSqlAzureServerTask.CreateNewResourceGroup(ResourceManagement resourceManagementChannel, ServiceOperationStatus& status)
at Microsoft.SqlServer.Management.StretchDatabase.Model.Tasks. ProvisionSqlAzureServerTask.Perform(IExecutionPolicy taskExecutionPolicy)
   at Microsoft.SqlServer.Management.StretchDatabase. Model.Common.Task.Perform(IExecutionPolicy policy, CancellationToken token, ScenarioTaskHandler taskDelegate), retrying …’.

WORKAROUND/SOLUTION

I must say the error is intuitive and tells that the location is not available. But I was wondering where did I choose the location? So, I launched the wizard again and found the place (highlighted below)

The problem here is that the location came as the default (alphabetically first) and I didn’t pay attention to it.

As soon as I selected “South India” I was able to proceed and stretch the table to Azure.

 Have you tested this feature? Do you have any interesting learning?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Stretch Database – ERROR: The Provided Location is Not Available for Resource Group

One of my clients contacted me to give advice about an error. In this blog, we would discuss error message – A read of the file at offset succeeded after failing 1 time(s)

Here is the complete error message they were seeing while all of the scheduled jobs in the SQL Server Agent were failing due to the IO errors.

Message. A read of the file ‘K:\MSSQL14.MSSQLSERVER\MSSQL\Data\HR_NCIDX.ndf’ at offset 0x000002a84dd000 succeeded after failing 1 time(s) with error: 21(failed to retrieve text for this error. Reason: 15105). Additional messages in the SQL Server error log and system event log may provide more detail. This error condition threatens database integrity and must be corrected. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.

Whenever I see such a message, I always ask to make sure that hardware is healthy. In the error message, we also see the error: 21. What does that mean?

Most of the cases you should be able to use NET HELPMSG command to convert the number to the text. I have written this trick in an earlier blog.

How to Convert Hex Windows Error Codes to the Meaningful Error Message – 0x80040002 and 0x80040005 and others? – Interview Question of the Week #182

So, error 21 means “The device is not ready”.

WORKAROUND/SOLUTION

Based on the error message, it is clear that in this situation there are definitely issue with hardware hosting K drive (in the error message we have the path of the file). This is also mentioned in Microsoft documentation – This message indicates that the read operation had to be reissued at least one time and indicates a major problem with the disk hardware.

I recommended my client to move ALL the file to new physical drive (on a new hardware LUN). The quick way was to do below.

1. Add a new Drive (let’s call it as X)
2. Create the same folder structure as on the K drive.
3. Stop SQL Service.
4. Move files from K drive to X drive.
5. Swap Drive letters (K > Y, X > K, Y > X)
6. Start the SQL Service. (you might get permission errors so set permissions accordingly)

I also asked them to engage hardware vendor and check the health of hardware. This is what they have replied.

Hi Pinal,

It was indeed a storage issue. After we have moved all our data new LUNs, all the jobs are running fine now. Thanks for quick turnaround time and your immediate help!

Such emails are so lovely to read.

Have you seen such error in ERRORLOGs? If yes, please take action right now before its too late.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Error: 825 – A Read of the File at Offset Succeeded After Failing 1 Time(s)

I wrote a blog earlier about my first experience (and error encountered) with “Stretch Database” feature. In this blog I would talk about the error – Free Trial subscriptions can provision Basic, Standard S0 through S3 databases, up to 100 eDTU Basic or Standard elastic pools and DW100 through DW400 data warehouses.

One of my blog readers read my earlier blog SQL SERVER – Stretch Database – ERROR: The Provided Location is Not Available for Resource Group.

He followed the wizard, but it failed with an error. He sent me an email and I asked for a complete log which was generated and here is what we see.

• [Informational] TaskUpdates: Message:Task : ‘Configure Stretch on the Database StretchTestDB’ — Status : ‘Running’ — Details : ‘Task failed due to following error: Microsoft.SqlServer.Management.Smo.FailedOperationException: Alter failed for Database ‘StretchTestDB’. —> Microsoft.SqlServer.Management.Common.ExecutionFailureException: An exception occurred while executing a Transact-SQL statement or batch. —> System.Data.SqlClient.SqlException: ‘Free Trial subscriptions can provision Basic, Standard S0 through S3 databases, up to 100 eDTU Basic or Standard elastic pools and DW100 through DW400 data warehouses’
• ALTER DATABASE statement failed.

WORKAROUND/SOLUTION

The error message gives and hints that the reader was using “Free” subscription which doesn’t have the capability to consume Database Stretch Unit (DSU). Here is what I see in my management studio.

Below is the screenshot from the portal.

As per error message below service tiers are possible.

• Basic, Standard S0 through S3 databases.
• Up to 100 eDTU Basic or Standard elastic pools
• DW100 through DW400 data warehouses

And none of them talk about DSU and hence the error. So, I replied to my reader that he needs a paid subscription to use this feature. I am surprised that Microsoft has not documented it, or at least I am not able to search it. If you find it, please share it via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Stretch Database – Free Trial Subscriptions Can Provision Basic, Standard S0 through S3 Databases, up to 100 eDTU Basic or Standard Elastic Pools and DW100 Through DW400 Data Warehouses

While trying to upgrade my SQL instance to the latest patch of SQL 2017, I encountered an issue. This has happened earlier with a few of my clients earlier but this time the cause was different. In this blog we would learn about cause and the fix of upgrade error:

The query has been canceled because the estimated cost of this query (37) exceeds the configured threshold of 30. Contact the system administrator.

Here are few earlier blogs about troubleshooting same issue.

• SQL SERVER – Script level upgrade for database ‘master’ failed because upgrade step msdb110_upgrade.sql encountered error 926, state 1, severity 25
• SQL SERVER – Script Level Upgrade for Database ‘master’ Failed Because Upgrade Step ‘sqlagent100_msdb_upgrade.sql’ – Error: 5041: MODIFY FILE Failed
• SQL SERVER – Script level upgrade for database ‘master’ failed because upgrade step ‘sqlagent100_msdb_upgrade.sql’
• SQL SERVER – Script level upgrade for database ‘master’ failed – There is already an object named ‘DatabaseMailUserRole’ in the database
• SQL SERVER – Script level upgrade for database master failed – Error: 4860, Severity: 16, State: 1 – Cannot bulk load. SqlTraceCollect.dtsx does not exist
• SQL SERVER – Error 15559 – Error 912 – Script Level Upgrade for Database ‘master’ Failed

This time it was a new error, so I am blogging it. As usual, I asked for ERRORLOG and was looking for the cause of failure.

Error: 8649, Severity: 17, State: 1.
The query has been canceled because the estimated cost of this query (37) exceeds the configured threshold of 30. Contact the system administrator.
Error: 912, Severity: 21, State: 2.
Script level upgrade for database ‘master’ failed because upgrade step ‘msdb110_upgrade.sql’ encountered error 8649, state 1, severity 17. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the ‘master’ database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.
Error: 3417, Severity: 21, State: 3.
Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.
SQL Server shutdown has been initiated
SQL Trace was stopped due to server shutdown. Trace ID = ‘1’. This is an informational message only; no user action is required.

WORKAROUND/SOLUTION

Whenever we have such upgrade script failure issue and SQL is not getting started, we need to use trace flag 902 to start SQL which would bypass script upgrade mode. This would allow us the find the cause and fix it. So, here are the steps I have done.

1. As I mentioned earlier, first we started SQL with trace flag 902. I started SQL using trace flag 902 as below via command prompt.

NET START MSSQLSERVER /T902

For named instance, we need to use below (replace instance name based on your environment)

NET START MSSQL$INSTANCENAME /T902

Refer: SQL SERVER – 2005 – Start Stop Restart SQL Server from Command Prompt

2. As soon as SQL was started, I was able to connect because the script didn’t run.

• In Object Explorer, right-click a server and select Properties.
• Click the Connections page.
• Clear the Use query governor to prevent long-running queries check box.

Here is the equivalent T-SQL:

EXEC sys.sp_configure N'query governor cost limit', N'0'
GO
RECONFIGURE WITH OVERRIDE
GO

3. After modifying the configuration, I stopped SQL Service and started normally (without trace flag) using SQL Server Configuration Manager.

And the issue was resolved. Have you faced any such interesting issue during SQL upgrades?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Script Level Upgrade for Database ‘master’ Failed Because Upgrade Step ‘msdb110_upgrade.sql’ Encountered Error 8649, State 1, Severity 17

There are many deployments where I have assisted my clients in creating Always On Availability Groups. Sometimes they come to me with special requirements, based on their infrastructure, and I always learn from them. In this situation, my client was trying to create a listener in a workgroup, and it was failing with error: Unable to determine if the computer ‘ListenerName’ exists in the domain ‘WORKGROUP’

THE SITUATION

My client has created two nodes Always On availability group in the cluster. This whole setup was in Azure infrastructure. Since they were testing, to save the cost they have deployed only 2 nodes availability group. Since they can’t afford 3^rd machine, they planned to deploy a domain-less cluster. They followed blogs on the internet, and it was deployed fine. They were also able to deploy availability group and databases synchronization was healthy.

The last step was to create a listener for this availability group. They were following this article

https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sql/virtual-machines-windows-portal-sql-availability-group-tutorial#configure-listener

One of the steps was to create Client Access Point for the listener and it was failing for them. Here is the screenshot of the error message.

WORKAROUND/SOLUTION

Instead of Cluster manager to create a listener, we need to use SSMS or T-SQL to create the listener. In the SSMS UI, we need to make sure that we are choosing “Static IP” and providing the IP Address as shown below.

Here is the T-SQL to achieve the same.

USE [master]
GO
ALTER AVAILABILITY GROUP [FIN-USW-AG]
ADD LISTENER N'FIN-USW-LIST' (
WITH IP
((N'10.0.1.22', N'255.255.255.0')
), PORT=1433);
GO

After creating a listener, they were able to follow the rest of the article and deploy Always On Availability Group in Workgroup in Azure Virtual Machines along with listener.

Have you seen such error earlier?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Error in Validation: Listener in Workgroup – Unable to determine if the computer exists in the domain ‘WORKGROUP’

Recently I was helping a client where SQL Server upgrade was failing with an error. If you have seen my blogs, there have been many blogs where I explain steps to fix upgrade-related issues. In this blog, we would learn about fixing another cause of script level upgrade for database ‘master’ failed.

Similar to earlier blogs of upgrade failures, here are the messages seen in the ERRORLOG. If you are new to SQL and don’t know about SQL Server ERRORLOG then please refer below blog. SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location

Here are the messages:

Error: 15151, Severity: 16, State: 1.
Cannot find the user ‘##MS_SSISServerCleanupJobUser##’, because it does not exist or you do not have permission.
Error: 912, Severity: 21, State: 2.
Script level upgrade for database ‘master’ failed because upgrade step ‘SSIS_hotfix_install.sql’ encountered error 15151, state 1, severity 16. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the ‘master’ database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.
Error: 3417, Severity: 21, State: 3.
Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.

From the first message, it is clear that user “##MS_SSISServerCleanupJobUser##” is missing from SSISDB database. Due to this, there is some action in upgrade script of SQL Server upgrade is failing and hence the error. As you can assume, the solution would be to create the user in the database. But keep in mind, SQL is not getting started and hence it is important to know the trick to start SQL without this error first. Read further to understand the trick.

WORKAROUND/SOLUTION

Whenever we have such upgrade script failure issue and SQL is not getting started, we need to use trace flag 902 to start SQL which would bypass script upgrade mode. This would allow us the find the cause and fix it. So, here are the steps I have done to fix missing user “##MS_SSISServerCleanupJobUser##”

1. As I mentioned earlier, first we started SQL with trace flag 902. I started SQL using trace flag 902 as below via command prompt.

NET START MSSQLSERVER /T902

For named instance, we need to use below (replace instance name based on your environment)

NET START MSSQL$INSTANCENAME /T902

Refer: SQL SERVER – 2005 – Start Stop Restart SQL Server from Command Prompt

2. As soon as SQL was started, I was able to connect because the upgrade script didn’t run. Here is the T-SQL which I ran to create the user which was missing.

USE [SSISDB] GO
CREATE USER [##MS_SSISServerCleanupJobUser##] FOR LOGIN [##MS_SSISServerCleanupJobLogin##] WITH DEFAULT_SCHEMA=[dbo] GO

3. After creating the login, I stopped SQL Service using SQL Server Configuration Manager. We can also do it via command prompt using below command. Below is for the default instance.

NET STOP MSSQLSERVER

If you are dealing with named instance, then below is the command ((replace InstanceName based on your environment)

NET START MSSQL$INSTANCENAME

4. Then start SQL normally (without trace flag) using SQL Server Configuration Manager.

And the issue was resolved. Have you faced any such interesting issue during SQL upgrade?

Reference : Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Script Level Upgrade for Database ‘master’ Failed Because Upgrade Step ‘SSIS_hotfix_install.sql’ Encountered Error 15151

I am happy to write this blog post as I am really happy that finally, SQL Server Engineering team has decided to fix the error which we all have been struggling for many years. I will say Since the year 2000, I have been working with SQL Server and it has always bothered me that every time when I try to insert data into my table which is larger than the data type of the column, I got an annoying error String or binary data would be truncated. I recently demonstrated this to my Comprehensive Database Performance Health Check.

To be honest, I am very much confident that every single person who has worked with SQL Server will agree that the error related to String or Binary data is the most annoying because it does tell us that error has occurred but will not tell us which column has the problem. This becomes extremely difficult to debug when you face this error doing a mass import of multiple rows with a wide table with multiple columns. I have seen users doing a different workaround to avoid this error as this just breaks your workflow so bad that the entire automation process becomes a manual process.

Let us run a quick test and here is the script for the same.

CREATE TABLE Table1
(Col1 VARCHAR(10), Col2 VARCHAR(10))
GO
INSERT INTO Table1 (Col1, Col2)
VALUES('ElevenChars','Cols')
GO

Let us see how this error occurs in SQL Server 2017.

Msg 8152, Level 16, State 30, Line 4
String or binary data would be truncated.
The statement has been terminated.

Now let see the same error in SQL Server 2019.

Msg 2628, Level 16, State 1, Line 4
String or binary data would be truncated in table
‘tempdb.dbo.Table1’, column ‘Col1’. Truncated value: ‘ElevenChar’.
The statement has been terminated.

You will see that error which has been annoying so far now finally started to make sense. One more reason to upgrade to the latest version in SQL Server.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Identify the column(s) responsible for “String or binary data would be truncated.”

I was working with one of my clients and they wanted to clean up old backup blobs from Azure. I have already blogged and shared my script. In this blog we would learn how to fix error: (403) Forbidden – This request is not authorized to perform this operation. SQL SERVER – PowerShell Script – Remove Old SQL Database Backup Files From Azure Storage

This was their on-premise SQL Server where the backups were taken directly to Azure Blob rather than local disk. I have modified the script as per their environment and used Automation Account and Run Book to schedule it.  They informed me that this was not working as expected. No 7 days old files were getting deleted!

I look at the output of runbook and was able to reproduce the error by a running script in PowerShell ISE.

Here is the text of the message

Get-AzureStorageBlob : The remote server returned an error: (403) Forbidden. HTTP Status Code: 403 – HTTP Error Message: This request is
not authorized to perform this operation.
At line:1 char:9
+ $blobs= Get-AzureStorageBlob -Container $ContainerName -Context $cont …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Get-AzureStorageBlob], StorageException
+ FullyQualifiedErrorId : StorageException,Microsoft.WindowsAzure.Commands.Storage.Blob.Cmdlet.GetAzureStorageBlobCommand

I used Azure Storage Explorer to connect to storage via Access Key and that also failed.

Interestingly, when I used Storage Explorer on the SQL Server machine, it was able to open.

WORKAROUND/SOLUTION

While there might be other reasons for the same error, my situation was unique. So, check this first and if it doesn’t solve the issue,

Since we were able to connect from SQL Server machine but not from others, I knew it would be someplace where they have blocked the access. I remembered such setting in the storage account, so I checked “Firewall and Virtual Networks” and found that they have allowed IP for SQL Server machine.

Once I modified the setting as per their environment, the run book job executed fine and other backups were deleted.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on Get-AzureStorageBlob: The Remote Server Returned an Error: (403) Forbidden. HTTP Status Code: 403 – HTTP Error Message: This Request is Not Authorized to Perform This Operation