Help: SQL Server

Sharing my knowlege about SQL Server Troubleshooting Skills

  • Blog Stats

    • 2,039,818 hits
  • Select GETDATE()

    April 2023
    M T W T F S S
     12
    3456789
    10111213141516
    17181920212223
    24252627282930

Help: SQL Server Always On Availability Group Listener not working in Azure SQL VMs (Using Internal Load Balancer – ILB). How to troubleshoot?

Posted by blakhani on April 17, 2018


As many of you might know that currently I have been supporting SQL Server in Azure Virtual Machines. Most of the concepts remains same but one of the common issue which was reported by many customer is about Always On listener connectivity. In this blog we would talk about the possible causes of SQL Server Always On Availability Group Listener not working in Azure SQL VMs.

The most common issue is that the listener got created successfully but it connects only from the server which is primary replica. Note that this issue will not occur in on-premise because there is no ILB needed.

Possible Cause # 1

Many of our customers have come to Microsoft via Azure support ticket, after following below article. (This is the most common issue)

Configure a load balancer for an Always On availability group in Azure

Is there something wrong with the article? No, there is nothing wrong there. Its mostly a oversight when customer miss running below script.

$ClusterNetworkName = "<MyClusterNetworkName>" # the cluster network name (Use Get-ClusterNetwork on Windows Server 2012 of higher to find the name)
$IPResourceName = "<IPResourceName>" # the IP Address resource name
$ILBIP = "<n.n.n.n>" # the IP Address of the Internal Load Balancer (ILB). This is the static IP address for the load balancer you configured in the Azure portal.
[int]$ProbePort = <nnnnn>

Import-Module FailoverClusters

Get-ClusterResource $IPResourceName | Set-ClusterParameter -Multiple @{"Address"="$ILBIP";"ProbePort"=$ProbePort;"SubnetMask"="255.255.255.255";"Network"="$ClusterNetworkName";"EnableDhcp"=0}

The parameters are explained in the article which I mentioned in the beginning. This is the script which maps probe port and IP which you have defined in the portal. Here is the PowerShell script which I use to figure out above (Thanks to my team member Ghufran)

 
#--  Probe Check Script:
#--------------------------------------------------------------
Clear-Host
Get-ClusterResource |`
Where-Object {$_.ResourceType.Name -like "IP Address"} |`
Get-ClusterParameter |`
Where-Object {($_.Name -like "Network") -or ($_.Name -like "Address") -or ($_.Name -like "ProbePort") -or ($_.Name -like "SubnetMask")}

Above script shows us the output which can be used to fill in first PowerShell script.

Possible Cause # 2

Second possible cause if that Load Balancing rules are not configured correctly. This information is also mentioned in the article.

image

Most of the time two values are set incorrectly “Session persistence” and/or “Floating IP (direct server return)”. If we choose any value other than what mentioned in above image, you would get connectivity issue from passive/other nodes.

Possible Cause # 3

In few customer cases I have seen that they run PowerShell for “Cluster IP Address” also. There is no real need to

Possible Cause # 4

In few cases, we have also seen that probe port was not opened in firewall. You should enable 1433 (SQL Server listening Port), 5022 (Always On Endpoint port), 59999 (Probe Port). If you are using any non-default port then take care of them in firewall.

Possible Cause # 5

If you are using NSG in Azure then make sure ports mentioned in #4 is are open.

This is the list I have compiled based on support cases reported. I will keep adding more items as and when they are reported.

  • Cheers,
  • Balmukund Lakhani
  • Twitter @blakhani
  • Author: SQL Server 2012 AlwaysOnPaperback, Kindle
  • Advertisement

    Posted in Always On, AlwaysOn, Azure ILB, listener, Microsoft Azure | Tagged: , , , , , | 4 Comments »

    FIX: Error – We experienced issues in connecting to Azure – While using SSMS to configure Always Encrypted feature using Azure Key Vault

    Posted by blakhani on July 21, 2017


    I have come across a situation where I was trying to configure Always Encrypted feature using SSMS. I was trying to use Azure Key Vault to configure it but getting an strange error. Here are few link which you can use to read more about the subject.

    https://docs.microsoft.com/en-us/sql/relational-databases/security/encryption/always-encrypted-wizard
    https://blogs.msdn.microsoft.com/sqlsecurity/2015/11/10/using-the-azure-key-vault-key-store-provider-for-always-encrypted/

    image

    If we click on “Details” then we would see below:

    image

    Here is the text of the error message. Note that you might see “Backup” or “Restore” instead of Recover below.

    TITLE: We experienced issues in connecting to Azure
    —————————–
    We experienced issues in connecting to Azure
    ——————————
    ADDITIONAL INFORMATION:
    Recover (Microsoft.SqlServer.Management.ServiceManagement)
    ——————————
    BUTTONS:
    OK
    ——————————

    If we click on “Show Technical Details” icon, we see below stack.

    Program Location:

       at Microsoft.SqlServer.Management.ServiceManagement.ResourceManagement.AzureKeyVaultKeyPermissionEnumConverter.ReadJson                                  
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable                                                                    
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList                                                                              
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList                                                                                
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal                                                                       
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue                                                                          
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject                                                                            
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject                                                                              
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal                                                                       
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue                                                                          
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject                                                                            
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject                                                                              
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal                                                                       
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList                                                                              
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList                                                                                
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal                                                                       
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue                                                                          
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject                                                                            
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject                                                                              
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal                                                                       
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue                                                                          
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject                                                                            
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject                                                                              
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal                                                                       
        at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize                                                                               
        at Newtonsoft.Json.JsonSerializer.DeserializeInternal                                                                                                   
        at Newtonsoft.Json.JsonConvert.DeserializeObject                                                                                                        
        at Newtonsoft.Json.JsonConvert.DeserializeObject[T]                                                                                                     
        at Microsoft.SqlServer.Management.ServiceManagement.ResourceManagement.AzureKeyVaultMethods.GetAzureKeyVault                                            
        at Microsoft.SqlServer.Management.ServiceManagement.ResourceManagement.ResourceManagement.GetAzureKeyVault                                              
        at Microsoft.SqlServer.Management.AlwaysEncryptedCommonControl.ColumnMasterKeyStoreCommonControl.AzureKeyVaultComboBox_SelectedValueChanged             
        at System.Windows.Forms.ListControl.OnSelectedValueChanged                                                                                              
         at System.Windows.Forms.ComboBox.OnSelectedValueChanged                                                                                                 
        at System.Windows.Forms.ComboBox.OnSelectedIndexChanged                                                                                                 
        at System.Windows.Forms.ComboBox.set_SelectedIndex                                                                                                      
        at Microsoft.SqlServer.Management.AlwaysEncryptedCommonControl.ColumnMasterKeyStoreCommonControl.SetAdalCredential                                      
        at Microsoft.SqlServer.Management.Azure.UI.CommonUserControl.OnSessionLoggedInEventSet                                                                  
        at Microsoft.SqlServer.Management.Azure.UI.CommonUserControl.AuthenticateUser                         

    If you are a geek then you can understand that SSMS is trying to get values in the Combo boxes. If there is a working subscription, then we can see that there are two combo boxes there, Subscriptions name(s) and Key Vault name(s).

    RESOLUTION

    • Login to portal using the same account which you are using in SSMS.
    • Go to “Key vaults”
    • Look for existing Key vaults defined.
    • Go to “Access policies” blade.
    • Choose the user.

    image

      As per documentation” To generate a column master key in the Azure Key Vault, the user must have the WrapKey, UnwrapKey, Verify, and Sign permissions to the key vault. Users might also need the Get, List, Create, Delete, Update, Import, Backup, and Restore permissions:

      • Once we click on user, I selected below permissions.
        • Under Key Permission (12 selected)
          Key Management Operations: Get, List, Update, Create, Import, Delete, Backup, Restore
          Cryptographic Operations: Unwrap Key, Wrap Key, Verify, Sign
        • Under Secret permissions(0 Selected)
          "Secret Management Operations" : None

      image

      Make sure UI looks like below

      image

        The error message in SSMS might say “Backup” or “Restore” or “Recover” based on what we select under “Secret permissions” There should be NONE selected from there.

        After setting permissions as above make sure you use “save” button to make changes. Hitting OK on previous screen doesn’t make the changes.

        image

        You have to check other Vaults also to make sure those settings are not there.

        I was able to move forward in the wizard and configure Always Encrypted using Azure Key Vault.

        Hope this helps.

      • Cheers,
      • Balmukund Lakhani
      • Twitter @blakhani
      • Author: SQL Server 2012 AlwaysOnPaperback, Kindle
      • Posted in Always Encrypted, Azure Key Vault, SQL Server Management Studio, SSMS | Leave a Comment »

        Linux Learning : How to get IP Address using command line? What is equivalent of IPCONFIG in Linux?

        Posted by blakhani on August 23, 2016


        Recently, I created a Linux Ubuntu VM in my Hyper-V environment. To connect, I was using PuTTY tool and was getting “Access Denied” error. On the other hand, I was able to connect using Hyper-V console. Here is what I mean

        I knew that I am definitely trying to connect to some other Ubuntu machine in my corporate network. All I need to figure out was the IP Address of my VM and connect using IP Address instead of the name. I took help from in-house expert Pradeep (b | t) and he showed me the magical command.

        In Windows, we are used to use IPCONFIG to know the IP address of a Windows machine. In the same way, IFCONFIG is the command in Linux. Since I was able to connect using Hyper-V console, I ran the magical command.

        It showed the IP Address (10.171.71.87) and then I came back to PuTTy and provided IP to connect and Voilà, I was able to get it!

        This might be something known to many (like Pradeep) but still unknown to few (like me) so sharing via blog.

        Cheers,
        Balmukund

        Posted in Linux Learning | Tagged: , , | 3 Comments »