Express Route: Error Setting up VPN Gateway

This week I hit a new oddity while working with a customer.  Yes, an Azure customer!

He had setup Express Route (ER) dozens of times before, but this time it just didn’t work!  He went thru the VPN gateway setup and when tried to save/create the gateway he got this:  Set-AzureVnetConfig : BadRequest: The virtual network xxxxxx has a gateway with connection type ‘Dedicated’, but you are not allowed to use the Express Route feature. Only users who are registered for Express Route can use ‘Dedicated’ gateways.  And without a working gateway, there is nothing to connect the ER to your Azure environment.

After a chat with some internal experts, I learned a little known fact: the ability use ER is actually a setting on your subscription!  While most subscriptions allow this by default, occasionally this is NOT the case.  When that happens, you just need to put in a request for Microsoft to enable that option and you’re good to go!

*previously posted at https://blogs.msdn.microsoft.com/nicole_welch/2015/08/azure-express-route-error-setting-up-vpn-gateway/

Bitlocker and Domain Controller Logical Disks

Recently I had a customer hit an issue that was hard to resolve…..until we stopped looking at the data and reconsidered our design.

What We Had

  1. Virtual DC (Hyper-V guest running in Azure, but the location and virtualization doesn’t matter!)
  2. C$ OS, E$ Sysvol/NTDS
  3. Bitlocker enabled for all drives

What Happened

The customer installed a relatively small and innocent piece of software, rebooted, and then we entered the BSOD loop — hard to see since it was on a guest in Azure.  After working through the night on the Azure aspect and out of ideas, we asked an AD guru to take a look.  Within minutes he had figured it out!

So what was going on?

  1. When the VM boots up, it tries to unencrypts the OS drive first. This OS disk key is stored in AD (accessible on another DC) or in 3rd party tool, like CloudLink
  2. Once the OS is unencrypted, Bitlocker frantically tries to unlock any data drives…..
  3. Meanwhile AD services are starting up (among the first services), but they can’t get to the AD database (it’s sitting on that locked E$…..)
  4. AD determines it can’t get to it’s database, crashes, throws the BSOD (with a nice pause so you can frantically try to write down the error message), and then reboots the server.
  5. The cycle repeats….

Basically our DC is so secure, even we can’t use it!  Luckily a DC is easily rebuilt and we all know better than to only have a single DC in the domain, right?

Lesson Learned

If you want to Bitlocker your DCs, put all those critical DC bits on the C$!

A big thanks to John Bay, our AD guru!

 

*previously posted at https://blogs.msdn.microsoft.com/nicole_welch/2016/01/bitlocker-and-domain-controller-logical-disks/

Learning Azure, aka “The Cloud”

I spend my days supporting Azure and SCOM here at Microsoft.  When I mention to my SCOM customers that I also do Azure, one of the first questions I get is “how do you learn Azure?”  I thought I’d take a little time to share my experiences and recommendations.  This isn’t a formal plan but rather my thoughts based on my own journey.

When I first heard about “the cloud” it was at MMS 2009 in Las Vegas.  My first thought was “this will never work for my company!” but I was wrong 🙂  Azure offers a pricing structure that most organizations can’t resist (only pay for what you use, lower operational costs, etc.) combined with short deployment timelines, impressive security compliance (https://www.microsoft.com/en-us/TrustCenter/Security/default.aspx), and reliability.  Needless to say I don’t know of a single organization that isn’t considering, if not already using, a public cloud.

First off, don’t put pressure on yourself to “learn Azure” or any cloud platform for that matter.  The cloud is a vast, ever-changing concept more than a “product” to be mastered.  Public clouds cover the entire stack (networking and server hardware thru applications) and very few of us would claim that broad mastery of our on-prem environments.    Add to that the fact that Azure is constantly evolving (a good thing!) and you have a moving target.  As to that point, with Azure (and as you’ve probably heard Windows 10) Microsoft is moving away from huge service packs to smaller, more frequent updates.  This means no waiting two years for a bug fix, product enhancement, or new feature set.  Awesome, right?  It also mean things you couldn’t do last month may now be possible as roughly monthly new features rollout complete with updates to the Azure PowerShell module.  If you are really going into the cloud, you’ll want to follow some key blogs (like https://azure.microsoft.com/en-us/blog/topics/announcements/) to ensure you always know “what’s new.”  That brings us back to the question of what your goal should be.  Personally I think your goal should be to become familiar with Azure, how on-prem solutions would look in the cloud, and where to go for answers when you get stumped.

So you want to start learning, but where to start?  With an Azure subscription.  Think of a subscription as setting up a line of credit…it establishes a payment method and is used to assign administrator access.  For many of us, we can use the monthly Azure credit that comes with our existing MSDN subscription.  If you don’t have an MSDN subscription, you can get a free Azure trial here: https://azure.microsoft.com/en-us/free/.  You get a set dollar amount to spend each month; when that runs out your environment is frozen until the next billing cycle begins.

Once you have a subscription, it’s time to dig in and try it out.  There are tons of how-to blogs so I’m not going to focus on that aspect rather focus on how to make the learning process as painless as possible.  Since I have a background in Windows server infrastructure, my first step was to setup a DC and new forest in the cloud (*note there are many Linux images available in Azure too!).  This grew into a SCOM 2012 R2 lab which I now use daily.

The key is to build something you will use regularly, isn’t a single-point-of-failure (in case you mess it all up or run out of a credits before the month-end), and ties into what you do regularly.  If you are in charge of the monthly patch cycle, maybe you test in the cloud first (it only takes minutes to build a IaaS VM).  Maybe you are a web guy…I’d setup an IaaS web server and then also do a PaaS website so you can get familiar with the differences, pros/cons of each.  Depending on who is paying for this subscription, you can even use it for personal content (family website, photo storage).  The objective is to use it fairly regularly so you become comfortable with Azure.

Tips

  • Try to use as many of the Azure features as possible.  I.e. use Azure Backup so you understand how that works, etc.
  • Make the most of your free money remember if you’re not using it, turn it off (VMs, services).  In Azure you pay for resources consumed and an online VM consumes resources 24/7.  Turn it off at night and on weekends to save money (there are some good blogs on how to use Azure automation to do this for you).
  • Pay attention to PaaS offerings (http://blogs.msdn.com/b/hanuk/archive/2013/12/03/which-windows-azure-cloud-architecture-paas-or-iaas.aspx)..  Most people start with IaaS (me included) since that’s our comfort zone…the server we can kick.  IaaS roughly allows you to recreate your on-prem environment in the cloud, but that leaves you responsible for the OS.  If possible use PaaS offerings to take one more layer of support off your shoulders.  You want to stop worrying about patches, disk space, OS performance, etc. right?  PaaS and SaaS allow you to stop supporting the server infrastructure and focus on the applications and services.  While using PaaS isn’t possible in all cases (sometimes you need control over OS and application settings that are not exposed in PaaS) but when it’s a viable option use it!

Honestly, coming from large servers environments Azure required me to grow my skill set.  Remember when I said the cloud covered the entire stack (OSI model)?  Most of use have spent our careers focused on a single layer, but Azure presents many layers via a single console to administrators.  No longer does the network team setup my subnets, the storage team my SAN disks, and the server team install my OS.  Instead when you install your first VM, you need to do all of this!  Knowing what to set in Azure (i.e. static IPs that on-prem would have been set via a network device) versus the OS was a challenge for me at first.  Most companies will have a few staffers who understand and design/setup the cloud aspect while everyone else just connects to VMs/applications “as usual,” so don’t let this scare you.  And for those in smaller organizations, you may already have the broader knowledge needed!

One last note….  Right now Azure is transitioning from the classic portal (manage.windowsazure.com) to the “new” portal (portal.azure.com).  You can think of this as the evolution of the cloud from v1 to v2.  V2 (the new portal) has a different backend that allows new features (role-based access!), faster deployment, and scales much larger.  Regardless of which portal is used, your resources can talk to each other (https://azure.microsoft.com/en-us/documentation/articles/virtual-networks-arm-asm-s2s/).  I’d recommend playing in both environments as the classic portal will be around quite a while and the new portal is still rolling out features.

I hope this helps you figure out where and how to start your journey.  Good luck exploring this brave new world!

*previously posted at https://blogs.msdn.microsoft.com/nicole_welch/2016/02/learning-azure-aka-the-cloud/

Moving Azure Images from the Commercial to the Azure Government Cloud (MAG)

*previously posted at https://blogs.msdn.microsoft.com/nicole_welch/2016/04/moving-azure-provider-images-from-the-commercial-to-azure-government-cloud-mag/

Some times we see customers move Microsoft provided images from the Azure commercial cloud to the Azure Government (MAG) cloud.  While this is technically supported, there are several things to consider.  When using the Microsoft provided images, there are configuration settings that are specific to the cloud environment.  When you move the VM (by moving the VHD), you risk having the wrong settings for your new cloud location.

Below is a list of settings that need to be changed.  This is NOT comprehensive and will be updated as needed.  Keeping mind the various endpoint that are different as well (https://azure.microsoft.com/en-us/documentation/articles/azure-government-developer-guide/)

Commercial Value MAG Value
KMS kms.core.windows.net:1688 kms.core.usgovcloudapi.net:1688

Azure IaaS VMs and the Three (now Four!) R’s

When you have a problematic IaaS VM (won’t start, won’t stop, can’t RDP even though it worked just yesterday….) and you’ve exhausted your usual tricks, turn to the Three Four R’s (the “did you reboot?” of Azure).

  1. Restart – Most users will think of this, just be sure you restart (and I mean a stop and start, not the restart button) from Azure (portal or PS) and not from the VM.  The Azure restart will give the fabric a chance to look for issues and self-heal.  Note: If you have a classic VMs (old portal) this also applies to your cloud service.  I’ve seen VMs acting up due to issues with their cloud service.  You can try restarting the cloud service….but keep in mind all hosted VMs by the cloud service will be restarted as well!
  2. Resize – Resizing (esp. if you increase the VM size to the largest possible), it will recreate certain elements of the VM <-> Fabric connection and could even move you to a new cluster node on the backend.   Resize, test to confirm it’s working, and then size back to your original size.  You will get charged at the higher VM size rate, but if it’s only for 15min that’s a minimal cost.  *Note: if you’re in ARM (VMs deployed via the new portal, not a classic VM) you can directly redeploy to a new cluster node using Azure powershell: https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-redeploy-to-new-node/ (step 4 below)
  3. Recreate – This one sounds scary, but if you make a note of your current configuration FIRST the recreation is quick (generally <20min) and relatively painless.  If you need to move your VM to a new cloud service, VNet, etc. or are having those “it’s just acting up” issues this is a good troubleshooting step to try out.   You basically are removing the Azure components and then recreating the Azure components (modifying if needed) — all while leaving your disks untouched.  See https://www.petri.com/recreate-virtual-machine-in-microsoft-azure  for step-by-step instructions.  *Note: The link is specific to ASM (classic portal) but the premise works for both classic and ARM VMs.
  4. Redeploy – Available in the new portal only for ARM VMs (not classic).  Effectively the same as a resize, only guaranteed that you change nodes.  https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-redeploy-to-new-node/

If these all fail, and you already confirmed there are no outages that could impact you (https://azure.microsoft.com/en-us/status/), it may be time to engage Microsoft.

See also https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-allocation-failure/

*previously posted at https://blogs.msdn.microsoft.com/nicole_welch/2016/05/azure-iaas-vms-and-the-three-rs/

Azure VMs need Internet Access

When customers move into the cloud, they tend to mimic their setup on-prem.  Not a bad thing, but when it comes to blocking internet access for servers this can create some unusual problems.

If you are using network security groups (NSGs), user defined routing (UDR), or forced-tunneling be sure to put in an exception for your Azure data center IP ranges, as lack of connectivity will impact many services including these:

  1. VM Extensions see https://blogs.msdn.microsoft.com/mast/2016/04/27/vm-stuck-in-updating-when-nsg-rule-restricts-outbound-internet-connectivity/
  2. Azure Backup see https://azure.microsoft.com/en-us/documentation/articles/backup-azure-vms-prepare/#network-connectivity
  3. Monitoring Agent/Extension see https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-proxy-firewall#configure-settings-with-the-microsoft-monitoring-agent
  4. KMShttps://docs.microsoft.com/en-us/azure/virtual-machines/troubleshooting/custom-routes-enable-kms-activation

Update 16 Aug 2018 – The use of service endpoints will limit the damage of blocking internet access.  Ensure all services you use/require are covered by service endpoints before blocking internet access.  https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-service-endpoints-overview

*previously posted at https://blogs.msdn.microsoft.com/nicole_welch/2016/08/azure-vms-need-internet-access/

Azure Classic Portal | How to Add a Data Disk Using a Different Storage Account than the OS Disk

When you add a new disk to an existing VM, you can only modify the  disk name and size which means you are stuck using the current/default storage account (see https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-classic-attach-disk/)

Using PowerShell, you can specify a different storage location for your new data disk though (see https://msdn.microsoft.com/library/azure/jj152837.aspx)

o Make sure you use -MediaLocation parameter as per the documentation “[i]f no location is specified, the data disk will be stored in the VHDs container within the default storage account for the current subscription.”

o The disk label you enter in the command shows in the storage view, but under the VM dashboard the disk title will look like this: [CloudService]-[VMName]-[LUN #]-[Date/time stamp].  If you expand the VHD column (copying it into notepad may be easier) you will be able to see the full VHD name and that WILL match what you specified in the media location parameter.

Example: Get-AzureVM -ServiceName “NLW-MAGBox” -Name “NLW-MAGBox” | Add-AzureDataDisk -CreateNew -DiskSizeInGB 128 -DiskLabel “NLWTest” -LUN 0 -MediaLocation “https://storage2.blob.core.usgovcloudapi.net/mycontainer/MyNewDisk.vhd” | Update-AzureVM

 

*previously posted at https://blogs.msdn.microsoft.com/nicole_welch/2016/09/azure-classic-portal-how-to-add-a-data-disk-using-a-different-storage-account-than-the-os-disk/

Connecting to MySQL for Azure Site Recovery (ASR)

When using ASR to replication VMware or physical machines into Azure two roles are required – the configuration and process servers (often combined on a single server) – to help coordinate and facilitate the data replication (https://docs.microsoft.com/en-us/azure/site-recovery/site-recovery-vmware-to-azure#run-site-recovery-unified-setup).  On the configuration server, configuration data is stored in a MySQL database.  *at this time this is a requirement to use MySQL, other databases types are not supported.

There are several scenarios when you may need to verify or modify data stored in this database.  Below are samples for your reference.

Note – database modifications will impact ASR and should be done with care

Login to MySQL and Connect to the ASR Database

from a command prompt:

mysql –u root –p  (you will be prompted to enter the password specified during installation)

show databases;  (will list all databases for your reference)

use svsdb1; (selects the ASR database so future queries will run against it)

image

 

To list all machines registered with the configuration server (CS)

from https://social.technet.microsoft.com/wiki/contents/articles/32026.how-do-we-cleanup-duplicatestale-entries-in-asr-vmware-to-azure-scenario.aspx

select id as hostid, name, ipaddress, ostype as operatingsystem, from_unixtime(lasthostupdatetime) as heartbeat from hosts where name!=’InMageProfiler’\G;

image

 

To Cleanup Duplicate/Stale Entries

see https://social.technet.microsoft.com/wiki/contents/articles/32026.how-do-we-cleanup-duplicatestale-entries-in-asr-vmware-to-azure-scenario.aspx

 

To Update the IP of a Machine

update hosts set ipaddress='[new address]’ where ipaddress='[old address]’;

example, update hosts set ipaddress=’192.168.0.4′ where ipaddress=’11.0.0.10′;

 

*previously posted at https://blogs.msdn.microsoft.com/nicole_welch/2017/02/connecting-to-mysql-for-azure-site-recovery-asr/

Azure IP Ranges

*Updated June 15, 2018*

For a myriad of reasons it’s nice to know what IPs you can expect to see coming to/from your Azure space.  Below is a quick cheat sheet.

Microsoft Azure Datacenter IP Ranges

updated 20 Aug 2018, thanks to Michael Ketchum of Microsoft for the additional information

The XML file now breaks down the IP ranges as follows:

  • “<SERVICE>” = Includes all IP’s for that service across all regions in the applicable cloud
  •  “<SERVICE>.<REGION>” = Includes all IP’s for that service in a specific region
  •  “AzureCloud.<REGION>” = Includes all IP’s/and services for that region
  •  “AzureCloud” = Includes all IP’s/and services for that cloud, such as Gov, commercial, etc.

The “Secret Azure IPs” you MUST Include – 168.63.129.16 and 169.254.169.254

Office 365 URLs and IP address ranges

Office 365 US Government: Endpoints for US Federal and US Defense Clouds (preview)

 

*previously posted at https://blogs.msdn.microsoft.com/nicole_welch/2017/02/azure-ip-ranges/