AzToso.com

Managed Identities vs Service Principals - when to use what ?

2023-01-12T01:00:00+01:00

On this page, you can find an overview of the identity types available in Azure and a recommendation on when to use what. At the end, there is a section with an example implementation of the best practices.

Key takeaway: Always preffer Managed Identities over service principals.

Identity types

Managed Identities

Managed Identities eliminate the need for users to manage credentials by providing an identity for the Azure resource in Azure AD and using it to obtain Azure Active Directory (Azure AD) tokens.

Top 3 benefits of using Managed Identities:

Eliminate the need for users to manage credentials
Nonexportable certificate-based credentials, valid for 90 days, rolled after 45 days
Uses long-lived tokens (24 hours) with a proactive token refresh every 12 hours (it can handle a maximum Azure AD downtime between 12-24 hours)

A Managed Identity is an Azure Resource Manager (ARM) object you create in a resource group. During the creation, you specify the location for the Managed Identity, which is just for storing the metadata. Managed Identities are special service principals created in Azure Active Directory (AAD). The Managed Identities will continue to work if there is a region outage. You can use Managed Identities only with Azure Resources to access other Azure Resources (VM access to KeyVault, for example), or any other OAuth2 protected endpoint.

You can create two types of Managed Identities:

System-assigned Managed Identities

The Managed Identity is tied to the lifecycle of the resources, which means that if you delete the resource, the Managed Identity will also be deleted. For that reason, it is not visible in the portal as a separate resource.

Use System-assigned Managed Identities when:

you want to be 100% sure that only that and no other resource can access a particular target
you don’t need to pre-provision the access before you create the Azure resource with enabled System-assigned Managed Identity

User-assigned Managed Identities

The Managed Identity is created as a separate resource in Azure.

Use User-assigned Managed Identities when:

you want to assign the same identity to multiple Azure resources
you need to pre-provision the access before you create the Azure resource
the principal you use for the deployment of your application doesn’t have an owner or user access administrator rights (which is a best practice)

As a best practice, use a separate deployment stack to create User-assigned Managed Identities in a dedicated resource group. Also, ensure you fine-grain the managed identity permissions and don’t use a single Manage Identity for everything you deploy. A good practice is to have a single identity per workload. The minimal RBAC role to be able to assign them to an Azure resource is Managed Identity Operator.

Service principals

Although Managed Identity is a special type of service principal, we usually refer to a service principal as a local representation, or application instance, of a global application object in a single tenant or directory. A service principal object is created when an application is given permission to access resources in a tenant (upon registration or consent). A service principal is created automatically when you register an application using the Azure portal. You can also create service principal objects in a tenant using Azure PowerShell, Azure CLI, Microsoft Graph, and other tools. A good overview of the differences between application ↔ service principal is given on the following page.

In some scenarios you must use service principals because Managed Identities are not supported. Keep in mind that you need to regulary rotate the secrets. One good practice is to create a new secret with each deployment as a step in your release pipeline. Check the example GitHub action bellow that can help you achiving this.

When to use what?

GitHub Actions

At the end of 2021, GitHub added OpenID Connect (OIDC) support to GitHub Actions. OIDC allows you to use Federated Credentials for your service principal, without requiring a secret and its subsequent rotation. Detailed steps to set this up are in the GitHub and Azure documentation. With the federated identity in place, you no longer need to worry about secret rotations.

Azure DevOps

From Azure DevOps, you can connect to Azure using Azure Resource Manager service connections. There are several types of connections:

Service principal (automatic): Azure DevOps automatically create a service principal with a valid secret for two years. To rotate the secret, you must edit the service connection and click Verify.
Service principal (manual): you manually create the service principal and assign it to the service connection. To rotate the secret, you need to generate a new secret in Azure AD and update the service connection.
Managed identity: you need to run your own Azure DevOps agents in Azure VMs with assigned managed identities. For this service connection, you don’t need to rotate the secrets.
Publish profile: it uses a publish profile to deploy to a WebApp
“Implicit”: this creates a service principal behind the scenes when choosing service connection types that connect to an Azure subscription. Examples are:
- Docker Registry (when choosing the Azure Container Registry subtype)
- Kubernetes (when choosing the Azure Kubernetes Service subtype)

As a general best practice, deploy a self-hosting agents in Azure and use Managed Identities.

Virtual Machines/Virtual Machine Scale Sets

If you need to access another Azure service (including databases) from within the Azure VM, you can use Managed Identities almost in all scenarios. The same applies if you need to access any OAuth2 endpoint (for example, an application or service).

Under the hood, you will use the Azure Instance Metadata Service to get the OAuth2 token for the assigned Managed Identity and present that token to the endpoint you call. Check the Microsoft documentation on how that works and how you can grab the token using different programming languages.

Azure PaaS services

Almost all of the Azure PaaS Services support Managed Identities. As a general best practice, always use them when accessing any other OAuth2 endpoint, including other Azure services.

For example, in Logic Apps, use the Managed Identity to access a storage account or a Log Analytics workspace. If that is not possible with the built-in connector, check the Rest APIs because they usually support the OAuth2 authorization workflow.

Promitor

Promitor is an Azure Monitor scraper that makes the metrics available for Prometheus. When configuring, make sure you use a Managed Identity to connect to Azure Monitor.

Grafana

Grafana also supports a Managed Identity when connecting to an Azure Monitor data source. To make use, you need to enable the feature in the configuration.

If you use Azure Active Directory as an identity provider for Grafana, you must use a service principal and add its secret to the Grafana configuration file. This means that you must rotate that secret. As mentioned above, you create a new secret with each deployment as a step in your release pipeline and redeploy Grafana every 3 months (for example). Check the example GitHub action bellow that can help you achiving this.

Azure Kubernetes Service (AKS)

Cluster & Kubelet Identity

AKS needs its identity to create additional resources like load balancers and managed disks or fetch container images from an Azure Container Registry. Always use a Managed Identity for both cluster and kubelet identities. If you have a running cluster that uses service principals, you can convert it to Managed Identity by following this procedure.

If you often redeploy the clusters (blue/green deployments) to an existing VNET, using User Assigned Managed Identities for both cluster and kublet makes more sense because you can use the same identity for all your deployments without the need to pre-provision the permissions with each deployment:

az aks create -g MyResourceGroup -n MyManagedCluster --assign-identity <control-plane-identity-resource-id> --assign-kubelet-identity <kubelet-identity-resource-id>

Pod identities

As a best practice, you should not use fixed credentials within pods or container images, as they are at risk of exposure or abuse. Instead, assign a managed identity to pods to automatically request access to resources using a central Azure AD identity solution. AAD Pod Identity enables Kubernetes applications to access cloud resources securely with Azure Active Directory. Using Kubernetes primitives, administrators configure identities and bindings to match pods. Then without any code modifications, your containerized applications can leverage any resource in the cloud that supports AAD as an identity provider.

Always use user-managed identities (type 0) when you create AzureIdentity definitions. Make sure those identities follow the lifecycle of the applications or the AKS cluster. Using service principals is strongly not recommended.

Azure AD Workflow identities

AAD workflow identity integrates with the Kubernetes native capabilities to federate with external identity providers. It is the next big thing for handling identities in AKS. In a nutshell, it enables you to exchange the Kubernetes service account token for an OAuth2 token representing a Managed Identity. More details about how it works are available in the documentation.

Kafka (Event Hubs)

Use a Managed Identity to connect to an OAuth2 protected listener in Kafka. (This repo)[https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth/java/managedidentity] provides an example of how you can build consumer and producer applications in java that connects to an Azure Event Hub using the Kafka protocol.

Best practices in action

Access HashiCorp Vault using Managed Identities

Enable Azure Auth on your HashiCorp namespace and allow access from the managed identity by assigning the policy. It is highly recommended to use User Assigned Managed Identity because they can reuse and assign the same identity to multiple resources:

# Login TO THE VAULT
vault login -method=oidc # make sure you login as a namespace admin
 
# Enable Azure Auth
vault write auth/azure/config \
    tenant_id=<YOUR_AAD_TENANT_ID> \
    resource=https://management.azure.com
 
# Enable a KV-V2 secret engine to store an example secret at a custom path
vault secrets enable -path=example-secrets kv-v2
 
# Add an example secre into the vault
vault kv put example-secrets/example-password password=password1234
 
# Create an example policy
vault policy write example-policy \
    path "example-secrets/data/example-password" {
    capabilities = ["read"]
}
 
# Create a role for the Managed Identity. IDENTITY_PRINCIPAL_ID = the object (principal) id of the Managed Identity
 vault write auth/azure/role/example-role \
    policies="example-policy" \
    bound_service_principal_ids="${IDENTITY_PRINCIPAL_ID}"

Get secrets from Azure Key Vault with Ansible using Managed Identities

Using Managed Identity, you can use the azure_keyvault_secret lookup to get a secret from an Azure Key Vault. A sample implementation is provided by the CIT-DSP team here.

GitHub Actions sample use cases

Upload files to a storage account with federated identity (no secret/storage access key)

Ensure that the service principal you use in the azure/login action has at least the Storage Blob Data Contributor Azure RBAC role on the storage container.

name: Upload file to a storage account
on: [push]
 
permissions:
      id-token: write
      contents: read
       
jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
    - name: 'check the code'
      uses: actions/checkout@v2
    - name: 'Az CLI login'
      uses: azure/login@v1
      with:
          client-id: $
          tenant-id: $         
          allow-no-subscriptions: true
   
    - name: 'Run Azure CLI commands'
      run: |        
         az storage blob upload-batch --auth-mode login -d 'https://tttosokrtest.blob.core.windows.net/upload' -s .

Rotate service principal’s secrets with federated identity

Check the GitHub action I build for this purpose.

How to get the System Assigned Managed Identity Object Id from within the VM?

2022-10-13T02:00:00+02:00

While working with Terraform and Azure DevOps self-hosted agent, I faced a challenge getting the Object(Principal) Id of a System Assigned Managed Identity to the VM running the agent. I needed that Object Id to assign access to a Key Vault. There is a way to do so via Azure CLI:

az resource list -n $(curl -s -H Metadata:true --noproxy '*' 'http://169.254.169.254/metadata/instance?api-version=2021-02-01' | jq -r .compute.name) -g $(curl -s -H Metadata:true --noproxy '*' 'http://169.254.169.254/metadata/instance?api-version=2021-02-01' | jq -r .compute.resourceGroupName) --query '{principal_id:[0].identity.principalId,tenant_id:[0].identity.tenantId}' --out json

but that approach requires the installation of Azure CLI and configuring Reader access on the VM itself.

A more elegant approach is to extract the Object Id from the access token received by calling the Azure Instance Metadata Service (IMDS) endpoint:

curl "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/" --header "Metadata: true" | jq -r .
access_token | jq -R 'split(".") | .[1] | @base64d | fromjson | .oid' -r

In Terraform, you can use the external provider to execute the script:

data "external" "account_info" {
  program = ["bash", "-c", "curl -s 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/' --header 'Metadata: true' | jq -r .access_token | jq -R 'split(\".\") | .[1] | @base64d | fromjson | {oid: .oid, appid: .appid,tid: .tid}'"]
}

and then access the Object Id (and the Tenant Id) with:

tomap(data.external.account_info.result).oid

For example:

resource "azurerm_key_vault_access_policy" "current-user" {
  key_vault_id = azurerm_key_vault.kv.id

  tenant_id = tomap(data.external.account_info.result).tid
  object_id = tomap(data.external.account_info.result).oid

  key_permissions = [
    "Create",
    "Delete",
    "Get",
    "Purge",
    "Recover",
    "Update",
    "List",
    "Decrypt",
    "Sign"
  ]
}

Deep dive into Azure Container Apps - Overview and Networking

2022-08-22T02:00:00+02:00

Deep dive into Azure Container Apps - Operating, security, reliability and pricing

Overview

Azure Container Apps is a serverless offering you can use to host your containers. It is a good fit for containerized apps and hosting microservices. Integrated services like KEDA, Envoy proxy, and Dapr provide you with out-of-the-box auto-scaling, ingress, traffic splitting, and simplified microservice connectivity.

Container Apps service is built on top of Kubernetes. Container Apps are an Azure Resource Manager deployment object, meaning you can’t just use your existing Kubernetes object descriptions and migrate them to Container Apps. You need to rewrite your deployment stack using Bicep or ARM templates. Terraform is not yet supported.

For each Container App, you need to specify the resources you want to allocate. The current range starts from 0.25 vCPU and 0.5Gi memory up to 2 vCPUs and 4.0Gi memory. If your container needs more resources to operate, you must brake your business logic down into multiple containers (or refer to them as microservices). Container Apps is not a good fit for you if that is not possible.

This document provides deep dive into Container Apps by clarifying not well-documented features, provides recommendations for deployments and gives examples of how to perform some of the actions. For a complete reference, check the official product page.

When are Container Apps a good fit?

If you want to avoid the complexity of operating a Kubernetes cluster, Container Apps are a good fit. It means that you need to trade off the control you have in Kubernetes for the simplicity the Container Apps service brings. For example, with Kubernetes, you are free to add platform features like automatic certificate rotations, external DNS registration, etc. In Container Apps, you need to wait for Microsoft to introduce such features in the platform.

If you want to host a website, App Service is a much better choice for that. If the website is a microservice application composed of multiple containers, Container Apps is a better fit.

Azure Container Instances is the preferred option for running single isolated containers that perform background processing and don’t need to scale out.

Dapr

Dapr (Distributed Application Runtime) is a portable, serverless, event-driven runtime that makes it easy for developers to build resilient, stateless, and stateful microservices that run on the cloud and edge and embraces the diversity of languages and developer frameworks. Dapr building blocks expose common distributed application capabilities, such as state management, service-to-service invocation, and pub/sub messaging

If you enable Dapr in Container Apps, it will inject a sidecar container which you can use for the microservices connectivity. You enable Dapr per Container App, meaning that in a Container Apps Environment, you can have apps with and without Dapr support. After the sidecar injection, you can talk to the Dapr container via HTTP or gRPC. Calls between Dapr sidecars are always made with gRPC, which delivers high performance. Dapr is cloud-agnostic and provides language-specific SDKs for easy development. The benefit of Dapr is that it speeds up application development by eliminating the need for a plumbing code while simultaneously, built applications are platform (cloud) agnostic.

Networking

Ingress controller

An Envoy proxy is used on the backend to provide ingress connectivity to the Container Apps. As part of the Container Apps deployment, you can configure the accessibility level (public, VNET, or Container Apps Environment), traffic split rules (when using multiple revisions), custom DNS name, and certificate to associate with the ingress. Other configuration options for the ingress are not available. Because the ingress is part of the platform, you don’t need to take care of updating it.

By default, ports 80 and 443 are open with HTTP to HTTPS redirection. As a general security guideline, ALWAYS enable only HTTPS traffic, even when you allow connection only from Container Apps Environment.

The fully qualified domain name (FQDN) associated with the ingress depends on the accessibility level:

Ingress accessibility	FQDN
Public	APP_NAME.UNIQUE_IDENTIFIER.REGION_NAME.azurecontainerapps.io
VNET only	APP_NAME.UNIQUE_IDENTIFIER.REGION_NAME.azurecontainerapps.io
Container Apps Environment only	APP_NAME.internal.UNIQUE_IDENTIFIER.REGION_NAME.azurecontainerapps.io

Note: mTLS is not (yet) supported by the ingress controller. Use OAuth2 to secure your application.

Custom DNS

Azure Container Apps allows you to bind one or more custom domains to a container app. For each custom domain, you also need a Server Name Indication (SNI) domain certificate associated with the ingress.

Steps to follow:

Upload a private key certificate (.pfx or .pem) to the Container Apps Environment.
In Container Apps, link the certificate with the custom DNS domain name

There is no possibility of storing the certificate in a Key Vault. By developing an automation script, you can rotate the certificate on a scheduled basis.

Communication between Container Apps

If you enable ingress, your Container App will be exposed using a dedicated DNS name. You can use that name to reach the Container App from other Container Apps. It is worth mentioning that the communication between Container Apps deployed in the same environment never leaves the environment:

As you can see from the picture, the environment DNS suffix is available inside containers via the environment variable CONTAINER_APP_ENV_DNS_SUFFIX. You can append that value to the name of the Container App you want to establish a connection to.

If you enable Dapr, you can levarage the built-in service discovery (the command is executed from another container app, using the console):

VNET integration

By default, when you create Container Apps Environment, it uses a managed VNET, that is a VNET that is part of the service, and you can’t see it or perform any configurational changes. If you want to:

use private endpoints (Private Link) to connect to your service (for example, database )
establish a private connection to a service that offers a VNET integration (such as Flexible Server for PostgreSQL)
communicate to a service that is available only on your internal network

then you need to use a custom VNET when creating the Container Apps Environment.

With custom VNET, you need a minimum of /23 subnet dedicated to Container Apps Environment. When deployed, the service reserves 60 IPs per VMSS configuration, meaning that if you enable Availability Zone, it will reserve 3 x 60 = 180 IPs.

Custom VNET with an internal environment

When using an internal environment (without a public exposure), the underneath Kubernetes cluster is exposed using a Private Load Balancer, which consumes a single IP address from the subnet where it is deployed. In the managed resource group (starts with mc_) there is also a Public Load Balancer deployed, which is used only for outbound connections, providing Internet connectivity to underlying Kubernetes nodes and deployed revisions.

To resolve the DNS names of your Container Apps from the VNET, you need to:

Create a Private DNS Zone using the Container Apps Environment default domain name. The default domain is in the format: VARIABLE_NAME.LOCATION.azurecontainerapps.io. For example icyplant-362dc571.westeurope.azurecontainerapps.io
Create a “star” DNS record inside the zone, pointing to the Internal Load Balancer IP address.
Link the Private DNS Zone with your VNET

It is impossible to use a custom private DNS zone for the Container Apps, because the service can’t verify the ownership. You have two options:

use a public DNS zone to create a private IP record
create a link to container apps environment Private DNS zone to all VNETs that need access to your service

If you choose to enable the HTTP ingress, you can:

limit the traffic to the VNET - everyone who has network connectivity to your VNET will potentially have access
limit to the Container Apps Environment - the app will be reachable only within the environment by other Container Apps

Custom VNET with an external environment

With an external environment, the configuration is more straightforward because there is no need for a private DNS zone integration into the VNET.

If you choose to enable the HTTP ingress, you can:

limit to the Container Apps Environment - the app will be reachable only within the environment by other Container Apps
accept traffic from everywhere - everyone who has network connectivity to your VNET will potentially have access

Suppose you want to apply an allow list to the incoming connections. In that case, you must create a Network Security Group (NSG) and associate it to the subnet where the Container Apps are deployed. The rule should look like this:

Source: List of source IP addresses
Source port ranges: * (usually allow all source ports)
Destination: The public IP address of the environment.
Destination port ranges: 443

Note: Never modify the NSG (aks-agentpool-{id}-nsg) that is deployed in the mc_ managed resource group. The Container Apps service controls that NSG, and it can be overwritten.

Deep dive into Azure Container Apps - Operating, security, reliability and pricing

2022-08-22T02:00:00+02:00

Deep dive into Azure Container Apps - Overview and Networking

Operating Container Apps

Revisions

In short, revisions are the versions of the Container App. A new revision is created when a revision-scope change is applied to the Container App, such as a change in the configuration or an image version. A Container App can have single or multiple active revisions at the same time. Multiple revisions enable traffic splitting, where a predefined percentage of the traffic can hit particular revisions, which is helpful in A/B testing and Blue/Green deployments scenarios.

By default, the revision name is composed by applying an alphanumerical random suffix to the Container App Name, like CONTAINER_APP_NAME–RANDOM_SUFFIX. It is recommended to replace the random with a custom suffix by using the commit id of your deployment repo (populate the value dynamically during the deployment).The commit id is helping in the case when you need to apply revision-scope changes without changing the image version. The release tag is not a good candidate because it usually contains dots that are not a good accepted in the revision suffix.

The revision labels can help you directly reach the desired revision if you plan to use traffic splitting. This can be useful if you want to perform some smoke test before gradually switching the traffic to the latest version (revision) or perform a full switch when using blue/green deployments. With that in mind, having the “latest” revision label applied to the last revision makes sense. You can move the label between the revisions as desired.

traffic: [         
    {           
        revisionName: '${containerAppName}--${previousRevisionSuffix}'
        weight: 100
    }
    {
        revisionName: '${containerAppName}--${newRevisionSuffix}'
        label: 'latest'
        weight: 0
    }
]

Note: When using the multiple revision mode, revisions remain active until you disable them. If your revision doesn’t scale to zero, you will pay for each active revision. As a best practice, always disable the revisions you don’t need.

Auto-scaling

Container Apps support three scale triggers: HTTP traffic, Event-driven, and Utilization (CPU or Memory usage). The utilization trigger doesn’t support scale to zero. If you plan to use the KEDA event-based triggers, it is recommended to set the revision mode to single (activeRevisionMode: ‘single’). The default settings (it is not clear if those are possible to change) of the auto scaler are:

pooling interval (scale-out): 30 seconds
cool down interval (scale-in): 300 seconds

Health probes

Service supports liveness, readiness and startup probes which are actually based on the standard Kubernetes health probes. You have an option to use HTTP or TCP for the probes. Make sure that your Container Apps always use health probes, which will lead to an increased overall availability of your applocations.

Storage

Note: Volume mounting is in preview

The following three storage types are currently supported by Container Apps: container file system, temporary storage, and Azure Files. If you need to persist data from your application, Azure Files is the only choice. At the moment of writing, the only way (at this moment) to mount storage into the running containers is by providing a YAML file with Azure CLI. This approach is not recommended for production because of the inability to provide a declarative specification for the deployment using tools like Bicep, ARM templates, or Terraform. Using a private endpoint (Private Link) to access the storage account (Azure Files) is strongly recommended.

Observability

Container Apps integrates out-of-the-box with the platform’s native monitoring service - Azure Monitor. Everything you write to stdout and stderr from the running containers is collected and stored in the table ContainerAppConsoleLogs_CL in Log Analytics Workspace. You can use Kusto to query the records and create an Azure Monitor log alert rules to receive alerts when desired. For debugging purposes, you can utilize the Log Streaming functionality to view the stdout and stderr messages in real time and use the Container Console to connect to a container and perform a debug from inside. The retention period depends on the configured retention period of the Log Analytics Workspace - from 30 up to 730 days:

Metrics are also stored in Azure Monitor. In the Metrics Explorer, you can view the metrics, apply filters and splitting and create multiple charts in a single view to correlate the metrics between different Container Apps and/or services (for example, Container App + Database).

Based on the metric value, you can also create alerts. The default retention period is 93 days, and it is not extendable at this moment (not until the Diagnostic Settings are available for Container Apps).

Security

Managed Identities

Both types of managed identities are supported: System and User assigned. Because the identity can be used to get the image from the container registry, you must always use a user-assigned identity. Ensure that the user-assigned identity is pre-provisioned and has the AcrPull permission over the Azure Container Registry before deploying the Container Apps. From within your application (container), you can call the internal REST endpoint to get the access token that you can use to access other OAuth2 protected services like Key Vault, Storage Account, etc. Always use the client libraries available for multiple programming languages when interacting with the REST endpoint.

Managing Secrets

Container Apps offer integrated secret management that you can use to store your secrets. You can use the secrets for storing connection strings for KEDA-based scale triggers or pass them as environment variables to the containers you deploy. Never store the secret value in the repository. Store the secrets as GitHub secrets, HashiCorp Vault, or Azure Key Vault. During the deployment phase, read the value from the vault and pass it as a secure parameter value.

Because Container Apps does not support Dapr Secrets Management API it is recommended to use the built-in secret management. Otherwise, you will need to add service-specific code to handle the secrets, which will add complexity without bringing any value.

Built-in auth mechanism

Note: Auth stands for authentication + authorization

Similar to Web Apps and Function Apps, Container Apps offer a built-in mechanism for authentication and authorization for published applications. Rather than building your auth stack in the application, it is recommended to use the built-in one.

When configuring an Azure Active Directory (Microsoft) as an identity provider, the Azure AD application registration can be created automatically, or you can provide an existing (pre-created) application registration. A recommendation is to pre-created application registration using the following script.

The last command in the script will enable the built-in auth mechanism for the Container Apps. If you prefer to do this via the portal, all the information you need is available via variables:

After the successful addition of an identity provider, the client’s (application) secret is stored in Container Apps’ secret under the name microsoft-provider-authentication-secret. If you rotate the secret, make sure you update the value before you create a new revision.

Note: Although the client secret is an optional parameter when using an existing application, always configure it because otherwise, you will end up using the implicit grant flow which these days is considered unsecured.

When enabled, the built-in auth deploys a sidecar container to each replica in the revision. The sidecar container handles all the aspects of authentication and authorization, meaning there is no need to change your code to support auth. If from your code you need to know who is accessing the application, you can access that information from the following headers:

X-MS-CLIENT-PRINCIPAL-NAME
X-MS-CLIENT-PRINCIPAL-ID

By default, any principal (user or service) can request tokens from Azure AD and access the application by presenting that token. If the behavior is not desired, you can:

Reliability

High-availability

If you deploy in a custom VNET, you have an option to enable Availability Zones support to get a data center redundancy. In the backend, the Container Apps service creates VMSS in each Availability Zones for scheduling the applications. The Availability Zone distribution of the containers is hidden from the users. Although you have an option to disable it, as a general recommendation, always use Availability Zone deployments because they improve your overall application availability.

From an application perspective, make sure you use at least two application replicas to provide high availability. Also, the event-based scaling for your application can improve availability. Why? Because if some of the containers are becoming unresponsive due to some reason, the auto-scaler will kick in and spin up new replicas to handle the load.

Geo-redundancy

An Azure region is a set of datacenters deployed within a latency-defined perimeter and connected through a dedicated regional low-latency network. Azure consists of 60+ regions around the world. Some regions are further divided into availability zones and unique physical locations, enabling redundancy at a regional level.

Under rare circumstances, it is possible that facilities in an entire region can become inaccessible, for example, due to network failures or natural disasters. To protect against such a scenario, you can apply one of the following strategies:

Redeploy on disaster: In this approach, the infrastructure and application are redeployed from scratch at the time of disaster. Copy of the data is usually available in the disaster region in backups or asynchronous replications. This strategy is appropriate for non-critical applications that don’t require a guaranteed recovery time. To be successful with this strategy, your infrastructure as code scripts need to be region independent. You need to test your deployments in the disaster recovery region, for example, deploying your test environment once every month to verify that all the resource SKUs you use are also available in the DR region.
Warm Spare (Active/Passive): A secondary hosted service is created in an alternate region, and roles are deployed to guarantee minimal capacity; however, the roles don’t receive production traffic. This approach is useful for applications not designed to distribute traffic across regions.
Hot Spare (Active/Active): The application is designed to receive a production load in multiple regions. The cloud services in each region might be configured for higher capacity than required for disaster recovery purposes. Alternatively, the cloud services might scale-out as necessary during a disaster and failover. This approach requires substantial investment in application design, but it has significant benefits. These include low and guaranteed recovery time, continuous testing of all recovery locations, and efficient capacity usage.

For Warm and Hot spare strategies for external environments, you can deploy Front Door or Traffic Manager to act as a global load balancer for your Container Apps applications:

For internal environments, it is possible, although not recommended due to the complexity, to achieve active/passive geo-redundancy by using private DNS zones.

Service Level Agreement (SLA)

Container Apps has a 99.95% SLA which translates to almost 22 minutes of acceptable downtime per month. The offered SLA is in line with the uptime SLA for AKS, which is also 99.95%. What is strange with the Container App’s SLA is that it is independent of the usage of Availability Zones. So, with or without Availability Zones support, you will get the same SLA.

Pricing

Container Apps costs are calculated based on three meters:

vCPU seconds - number of seconds you allocate a vCPU
Gibibyte seconds - number of seconds you allocate a memory
Requests - number of processed requests (with enabled ingress only)

Implementing a proper scaling policy will reduce costs in all environments, especially on dev and test. If the Container Apps scale to zero, you don’t pay anything because they are not allocating resources or serving requests. For Container Apps replicas in idle state (resource usage below active billing threshold), you will pay ~90% less for the vCPU allocation. You may have Container Apps (replicas) in an idle state to avoid cold starts for your application.

Billing at seconds granularity enables cheaper instant spikes handling. Also, there is a monthly free grant (on a subscription level) which is automatically applied.

Deep dive into Defender for Containers

2022-04-05T02:00:00+02:00

Overview

Microsoft Defender for Containers is the new plan that merges the capabilities of the two existing Microsoft Defender for Cloud plans, Microsoft Defender for Kubernetes and Microsoft Defender for container registries, and adds a new set of features:

Multi-cloud support: AKS and any Cloud Native Computing Foundation (CNCF) certified Kubernetes clusters (through Azure Arc)
Kubernetes-native deployment: automatic deployment using DaemonSet
Advanced Threat Detection: deterministic, AI, and anomaly-based detection
Vulnerability assessment: continuous scan for running images

Provisioning

First, the plan (the product) needs to be provisioned on the subscription via the portal or using the REST API:

az rest --method PUT \
    --uri "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Security/pricings/Containers?api-version=2018-06-01" \
    --body '{"properties": {"pricingTier": "standard"}}'

or using CLI:

az security pricing create --name Containers --tier Standard --subscription $SUBSCRIPTION_ID

After enabling the plan, the components can be provisioned automatically or manually on the clusters. The provisioning is related to two of the components in Defender for Containers:

Defender security profile (the DaemonSet): Security profile deployed to each worker node, collects security-related data and sends it to Defender for analysis. It is required for runtime protections and security capabilities provided by Defender for Containers.
Azure policy for Kubernetes add-on: Extends Gatekeeper v3, required to apply at-scale auditing, enforcements, and safeguards on clusters in a centralized, consistent manner. The policy add-on is not an exclusive feature of Defender for Containers, but it is an integral part of the overall AKS security solution.

Automatic

The automatic provisioning is enabled using Azure Policies:

Configure Azure Kubernetes Service clusters to enable Defender profile - uses a System assigned managed identity with Contributor and Log Analytics Contributor roles to remediate the cluster.
Deploy Azure Policy Add-on to Azure Kubernetes Service clusters - uses a System assigned managed identity with Azure Kubernetes Service Contributor Role role to remediate the cluster.

Because it is based on Azure Policies, the automatic provisioning will work only for clusters that have enabled Azure RBAC (something not mentioned in the documentation). The two policies can be applied on the Management Group level, coupled with a custom policy that enables Azure RBAC on deployed clusters for an automated Defender for Container provisioning across the tenant.

It can happen that both policies will try to update the cluster simultaneously. In that situation, one of the two components will not be deployed into the cluster with the following error displayed under Deployments in the cluster’s resource group: Operation is not allowed: Another operation (systemtemp - Updating) is in progress, please wait for it to finish before starting a new operation. See https://aka.ms/aks-pending-operation for more details.

A policy remediation task is needed to protect against end-users deleting the components deployments and fixing the above issue.

Manual

The manual deployment can be done through the portal (by fixing a recommendation in Defender) or using the REST API:

az rest --method PUT \
    --uri "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCE_GROUP/providers/Microsoft.ContainerService/managedClusters/$CLUSTER_NAME?api-version=2022-01-01" \
    --body '{"location": "'$LOCATION'","properties": {"securityProfile": {"azureDefender": {"enabled": true,"logAnalyticsWorkspaceResourceId": "'$logAnalyticsWorkspaceResourceId'"}}}}'

Features

Scanning images for vulnerability

Defender for Containers scans images for vulnerabilities stored in an ACR. It pulls the image from the registry and runs it in an isolated sandbox with the Qualys scanner. Image scan is triggered for every image that is pushed, pulled, or imported into the registry. In addition, the recently (in the last 30 days) pulled images are scanned weekly. If the defender security profile (the DaemonSet) is deployed in a cluster, it will initiate a weekly scan of the running images pulled from ACR.

The vulnerabilities findings are listed under Recommendations and are visible a few seconds after the scan finishes.

Available in private preview, there is a policy named Kubernetes clusters should gate deployment of vulnerable images which can deny the deployment of images with vulnerabilities identified by the CI/CD scanning or ACR scanning. To make use of it, the Azure Policy add-on must be installed on the cluster.

There is a possibility to disable specific findings based on multiple parameters, including severity and CVSS (Common Vulnerability Scoring System) score. That will prevent unnecessary noise and enable us to focus only on the findings that need action.

CI/CD integration

For Github workflows, there is a Azure/container-scan action that scans the images for vulnerabilities using Trivy and performs static code analysis using Dockle. The results from the scans can be pushed to the Defender for Containers using an Application Insight. The complete guideline is available here.

As it is documented, the integration can improve visibility. It only makes sense if it is combined with the policy Kubernetes clusters should gate deployment of vulnerable images. Another option is to parse the output and not push to the ACR images with specific vulnerabilities or best practices violations.

Runtime security

Defender for Containers provides real-time threat protection and generates alerts for suspicious activities. Threat protection at the cluster level is provided by the Defender profile (the DaemonSet) and analysis of the Kubernetes audit logs.

The node-level threat detection, DaemonSet that runs on all of the nodes in the cluster, inspects activity on the Kubernetes worker-node to detect suspicious activity that runs by the containers on the nodes. The solution is developed for Kubernetes and provides container-specific alerts and vulnerabilities detections as they are discovered. It also tracks the MITRE ATT&CK® matrix for Containers, which is a collection of tactics and techniques of attacks related to container environments.

The Defender profile deploys the following components into the cluster:

Although initial resource requests are low, proper testing is needed because the total max limits for the DaemonSets per node are: 270m CPU and 328Mi memory.

When Defender for Container is enabled on a cluster, it will monitor the Kubernetes API operations to find suspicious and malicious activities in the Kubernetes control plane. Under the hood, Defender collects the API logs over the Azure backbone, analyzes those, and presents the findings. The process of collecting and analyzing is transparent and there is no cost associated with storing the API logs.

Security Alerts

Microsoft Defender for Containers provides security alerts categorized in three severity levels: High, Medium, and Low. Fired alerts are visible in the Security alerts sections in Defender for Cloud. Email notifications are also configurable, based on the severity level, to the chosen email recipients (by default to the owners of the subscription). Microsoft has an add-on that uses Microsoft Graph Security API to ingest the security alerts into Splunk. The security alerts and recommendations are also exportable to an Event Hub from where they can be consumed by Splunk using the Splunk add-on for Microsoft Cloud Services.

Pricing

Microsoft Defender for Container is list priced $7 per vCore associated with a Kubernetes cluster. Please note that vCore differs from vCPU for the SKUs that support hyper-threading (D series, for example), and the ratio is 2:1 (vCPU:vCore). The average number of vCores in the last 30 days is taken into account for the monthly calculations. The price also includes 20 free ACR vulnerabilities scans per vCore. Any additional scan is list priced at $0.29 per image digest.

A workbook for cost estimation across multiple subscriptions is available on GitHub. The workbook supports a maximum of 200 subscriptions. To generate the cost estimate for bigger tenants, use the following Resource Graph Query:

resources
| where type =~ "microsoft.containerservice/managedclusters"
| extend machineType = tostring(properties.agentPoolProfiles[0].vmSize), nodesCount = toint(properties.agentPoolProfiles[0]['count']), powerState = tostring(properties.agentPoolProfiles[0].powerState.code), minMachineCount = toint(properties.agentPoolProfiles[0].minCount), maxMachineCount = toint(properties.agentPoolProfiles[0].maxCount)
| extend vCoresPerMachine = toint(extract_all(@"(\d+)", machineType)[0])
| extend coresTotal = vCoresPerMachine * nodesCount
| extend minMonthlyCost = vCoresPerMachine * minMachineCount * 7, maxMonthlyCost = vCoresPerMachine * maxMachineCount * 7, estimatedMonthlyCost = coresTotal * 7
| summarize sum(estimatedMonthlyCost)

Conclusion

Defender for Containers can significantly improve the security of the container environments. The solution can be automatically provisioned to the newly created and already existing clusters through the (Azure) platform. People with Security admin and Security reader roles on the subscriptions can view their security recommendations and alerts and have an active role in the security observation of the deployed resources. A single support contact point covers both the product and the underlying infrastructure in case of an issue.

Pushed, pulled, and imported images to ACR will be scanned for vulnerabilities with Qualys. On top of it, if used, the CI/CD integration will add scan results based on Trivy and security best practices. With the runtime integration, the images running in Kubernetes will be scanned every week for any newly discovered vulnerabilities. At this moment, there is no support for scanning running images pulled from public repositories. Microsoft has that feature in the backlog.

The runtime security doesn’t provide an inventory list of pods running in the cluster. For environments with many clusters, having that central list of running images accross the tenant is very useful. Today, to get that inventory list yet another agent need to be deployed into the cluster - the Log Analytics agent.

The Azure Policy-addon for Kubernetes is required in order to get the maximum out of Defender for Containers, like denying deployment of vurnelable images. The policy add-on will apply all of the policies applicable to the resource, coming from the management group, subscription and resource group. This can lead to an unexpected behaivour, because Kubernetes related policies are part of different policy initiatives that maybe are already applied at the subscription level.

OAuth 2.0 with Managed Identities

2022-03-23T01:00:00+01:00

Using Managed Identities to access an OAuth 2.0 protected application is a best practice for an application to application communication or, as referred to in the OAuth 2.0 terminology - Client Credentials Grant Flow. This article will show you how to configure your application in Azure AD and use Managed Identity to access the application.

The Microsoft identity platform (Azure Active Directory) offers authentication and authorization services using standards-compliant implementations of OAuth 2.0 and OpenID Connect (OIDC) 1.0. You can use Azure Active Directory to build an authentication and authorization layer for your application. There are multiple OAuth 2.0 flows implementation, mainly depending on the type of the client application. The Client Credentials Flow is the best fit when it comes to service-to-service communication.

OAuth 2.0 Client Credentials Grant Flow permits a web service (serving the role of a confidential client) to use its credentials to authenticate when calling another web service instead of impersonating a user. In this scenario, the client is typically a middle-tier web service, a daemon service, or a website:

The client application authenticates to the Azure AD token issuance endpoint and requests an access token.
The Azure AD token issuance endpoint issues the access token.
The access token is used to authenticate to the secured resource.
Data from the secured resource is returned to the client application.

Authentication is performed using the OpenId Connect protocol on top of OAuth2. The client uses a service principal with a secret or certificate to authenticate to Azure AD. A recommended approach for the clients is to use Managed Identities:

Eliminate the need for users having to manage credentials
Non exportable certificate-based credentials, valid for 90 days, rolled after 45 days
Uses long-lived tokens (24 hours) with a proactive token refresh every 12 hours (it can handle a maximum Azure AD downtime between 12-24 hours)

To register your application in Azure AD, you need to:

Create a manifest.json file that describes the roles your service support. Based on the provided role in the token, you can perform Role Base Access Control in your service:

 cat > manifest.json << ENDOFFILE
 [{
     "allowedMemberTypes": [
     "Application"
     ],
     "description": "Consumers can use the service",
     "displayName": "Consumer",
     "isEnabled": "true",
     "value": "consumer"
 }]
 ENDOFFILE

 # Register an application in Azure AD
 appId=$(az ad app create --display-name aztoso-com-app \
     --identifier-uris https://myapp.aztoso.com --app-roles @manifest.json --query appId)
 # Install the application in the local Azure AD - by creating a service principal
 az ad sp create --id $appId

Assign a role to the User Managed Identity client will use to connect to your service:

 #Object ID of your application registered in Azure AD
 appId="<YOUR_APPPLICATION_ID_HERE>"
 # ObjectId of the Managed Identity used by the client
 managedIdentityObjectId="<OBJECT_ID_OF_MANAGED_IDENTITY>"
 # Role you want to assign
 role="consumer"
    
 # Get the application role id
 appRoleId=$(az ad app show --id $appId --query appRoles[?appRoles.value==$role].id -o tsv)
 # Application's service principal id
 spObjectId=$(az ad sp show --id $appId --query objectId -o tsv)
 # Create the body for the Graph API call
 cat > body.json << ENDOFFILE
 {
 "principalId": "${managedIdentityObjectId}",
 "resourceId": "${spObjectId}",
 "appRoleId": "${appRoleId}"
 }
 ENDOFFILE
 # Assign the role to the managed identity
 az rest --method POST --uri "https://graph.microsoft.com/v1.0/servicePrincipals/$managedIdentityObjectId/appRoleAssignedTo" \
     --headers 'Content-Type=application/json' --body @body.json

After assigning the necessary role to the Managed Identity, the client can get the token from the Azure Instance Metadata Service and use the token to access your service:

curl 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmyapp.aztoso.com' \
    -H Metadata:true -s | jq -r .access_token

The issued token contains claims that you can use to authorize clients to your service:

aud - Identifies the intended recipient of the token. Your service should validate this value, and reject the token if the value does not match.
iss - Identifies the security token service (STS) that constructs and returns the token, and the Azure AD tenant in which the user was authenticated. Your service should use the GUID portion of the claim to restrict the set of tenants that can sign in to the app.
appid - The application ID of the client using the token. Your service should use appid or oid to identify the client that connects to your service.
oid - The immutable identifier for an object in the Microsoft identity system - the Object (principal) ID. Your service should use appid or oid to identify the client that connects to your service.
roles - The set of permissions exposed by your application that the requesting application has been given permission to call. If implementing RBAC, your service should validate the roles of the calling client.

Private Link and Flexible Server DNS resolution using custom VNET resolvers

2022-02-10T01:00:00+01:00

Overview

In enterprise scenarios, when you need to resolve on-premise DNS records or have cross-subscription DNS resolution of the private DNS zones, you need to have some kind of hybrid DNS configuration. One such setup is having the DNS resolvers in a central subscription with all Private DNS zones linked to the VNET those resolvers reside in.

Private (Link) Endpoints

If you plan to create many private endpoints across your subscriptions, the above approach will not scale. A better solution is to have a single private DNS zone per service linked to the resolvers VNET and add the records there during the endpoint creation. To make it work, after you create the private endpoint for the service in your VNET you need to create a DNS Zone Group:

az network private-endpoint dns-zone-group create 
    --endpoint-name # the name of the enpoint you created
    --name # name of the dns-zone-group (free to use whatever you like)
    --private-dns-zone # resource id of the private DNS zone
    --resource-group # resource group where you created the private endpoint
    --zone-name # name of the private DNS zone
    [--subscription] # if endpoint is created in another subscription
# example
az network private-endpoint dns-zone-group create 
    --endpoint-name privatelink-storage \
    --resource-group rg-privatelink-test \
    --name test-dns-zone-group \
    --private-dns-zone "/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/rg-test-privatelink/providers/Microsoft.Network/privateDnsZones/privatelink.blob.core.windows.net" \
    --zone-name privatelink.blob.core.windows.net

To be able to create that DNS Zone Group, you need the following permissions on the private DNS zone:

"Microsoft.Network/privateDnsZones/A/read",
"Microsoft.Network/privateDnsZones/A/write",
"Microsoft.Network/privateDnsZones/read",
"Microsoft.Network/privateDnsZones/join/action",

To create the Private DNS Zone Groups through the portal, you also need the Microsoft.Resources/subscriptions/resourceGroups/read permission on the resource group hosting the Private DNS Zone.

Flexible Server

Flexible Server uses VNET integration to provide private access to the databases. Under the hood, when you create the service, you delegate a dedicated subnet to Microsoft to deploy and operate the VMs used for hosting the database server for you. As part of the VNET integration, the deployment creates a Private DNS zone with the name equal to the server’s name, making it impossible to use the same solution as with the private endpoints. The only possibility to overcome this challenge is to host custom DNS resolvers in the (spoke) subscriptions:

DNS resolution from AKS

Suppose you only need to connect to the Flexible Server from AKS. In that case, instead of deploying custom DNS resolvers in the VNET, you can customize CoreDNS to forward all the queries for Flexible Server to the Azure DNS Server. For PostgreSQL, the custom configuration will look like this:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  flexible.server: |
   postgres.database.azure.com:53 {
      errors
      cache 30
      forward . 168.63.129.16
      reload
    }
EOF

After you apply the coredns-custom configmap, force CoreDNS to reload the ConfigMap (as per the documentation, the command bellow will not cause any downtime):

kubectl delete pod --namespace kube-system -l k8s-app=kube-dns

Building an AKS baseline architecture - Part 4 - AAD Pod Identity

2022-01-04T01:00:00+01:00

Posts from this series:

Building an AKS baseline architecture - Part 1 - Cluster creation

Building an AKS baseline architecture - Part 2 - Governance

Building an AKS baseline architecture - Part 3 - GitOps with Flux2

Building an AKS baseline architecture - Part 4 - AAD Pod Identity

Overview

When we develop solutions, they usually include multiple Azure services that need to communicate between them. Moving the compute part of the solution to AKS leaves the same problem - who are we going to authenticate the communication between those Azure services? With the traditional approach, using service principals or connection keys, we have the problem of safely storing and rotating the credentials. Microsoft solved this problem by introducing Managed Identities. We can use Managed Identities for the AKS Control plane and Kubelet, but can we use them for pods inside AKS? The answer is YES - with the open-source component named AAD Pod Identity.

AAD Pod Identity enables Kubernetes applications to access cloud resources securely with Azure Active Directory. In addition to the security benefit, Managed Identities uses long-lived tokens and can handle Azure AD hiccups with a maximum duration between 12 and 24 hours.

Note: In preview, a managed add-on is available that you can enable for your AKS cluster, which will provide you with the same functionality. However, this add-on will never transition to General Availability because Microsoft is working on a V2 version.

How it works?

The solution consists of two core components:

Managed Identity Controller (MIC) - manage the AzureAssignedIdentity. Assigns the specified Users Managed Identity to the nodepool (VMSS) on a pod scheduling event.
Node Managed Identity (NMI) - intercept the requests to the Azure Instance Metadata Service and get the correct token based on the AzureIdentityBindings

and four Custom Resource Definitions (CRDs):

AzureIdentity - describes the Azure Identity resource. It supports: User Assigned Identity and Service Principal (with password or certificate)
AzureIdentityBindings - enables binding of the AzureIdentity to a pod using the specified selector
AzureAssignedIdentity - managed by the Managed Identity Controller, it describes the state of the identity binding
AzurePodIdentityException - enables pods to access the Azure Instance Metadata Service directly, without the request being intercepted by NMI

The following diagram describe the complete workflow of how AAD Pod Identity works:

Role Based Access Control Assignments

The AKS cluster needs to have the necessary permissions to assign and unassign the user-managed identities on the underlying Virtual Machine Scale Set. You need to have the role of Owner or User Access Administrator to perform RBAC assignments in Azure.

To create the assignments using Azure CLI, use the instructions below. Consider automating the process and do the assignments using a pipeline:

RESOURCE_GROUP= # resource group where the AKS cluster is deployed
CLUSTER_NAME=  # the name of the AKS cluster
SUBSCRIPTION_ID= #  the id of the subscription
IDENTITIES_RESOURCE_GROUP= # resource group name where the identities you plan to assign to pods are stored. If you plan to bind their lifetime to the cluster, you can keep them in the NODE_RESOURCE_GROUP. Otherwise, it is a best practice to use a dedicated resource group

# Obtain the ID of the managed identity assigned on the node pools
CLUSTER_IDENTITY_ID=$(az aks show -g $RESOURCE_GROUP -n $CLUSTER_NAME --query identityProfile.kubeletidentity.clientId -o tsv)

# Get the node resource group
NODE_RESOURCE_GROUP=$(az aks show -g $RESOURCE_GROUP -n $CLUSTER_NAME --query nodeResourceGroup -o tsv --only-show-errors)

# Assign the necessary role to the AKS cluster identity, to be able to modify the VMSS settings
az role assignment create --role "Virtual Machine Contributor" --assignee $CLUSTER_IDENTITY_ID --scope /subscriptions/$SUBSCRIPTION_ID/resourcegroups/$NODE_RESOURCE_GROUP

# Option 1 - Assign the necessary role to the AKS cluster identity, to be able to attach the managed identities in the IDENTITIES_RESOURCE_GROUP to the VMSS.
az role assignment create --role "Managed Identity Operator" --assignee $CLUSTER_IDENTITY_ID --scope /subscriptions/$SUBSCRIPTION_ID/resourcegroups/$IDENTITIES_RESOURCE_GROUP

# Option 2 - Assing the necessary role to the AKS cluster identity only to particular user managed identities:
az role assignment create --role "Managed Identity Operator" --assignee $CLUSTER_IDENTITY_ID --scope /subscriptions/$SUBSCRIPTION_ID/resourcegroups/$IDENTITIES_RESOURCE_GROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/$IDENTITY_NAME

Security considerations with Kubenet

Running aad-pod-identity in a cluster with Kubenet network plugin is not a recommended configuration because of the security implication. Kubenet is susceptible to ARP spoofing, making it possible for other pods to impersonate a pod with access to identity. Using CAP_NET_RAW capability, an attacker pod could impersonate a pod with valid access and then request a token.

To mitigate the vulnerability, you need to add a securityContext in your pod defintion that drops the NET_RAW capability:

securityContext:
  capabilities:
    drop:
    - NET_RAW

To enforce this mitigation at the cluster level, use Azure Policy add-on for AKS and enroll the Kubernetes cluster containers should only use allowed capabilities policy. Add NET_RAW into Required drop capabilities in the policy parameter values.

Deploy with Flux2

# PULL THE HELM CHART LOCALY
helm repo add aad-pod-identity https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts
helm pull aad-pod-identity/aad-pod-identity

# PUSH THE HELM CHART TO ACR
az acr helm push -n $ACR_NAME aad-pod-identity-*.tgz

# PUSH THE IMAGE TO ACR
AADPODIDENTITY_CHART_VERSION=$(helm show chart aad-pod-identity/aad-pod-identity | grep version |  awk -F" " '{print $2}')
az acr import --source mcr.microsoft.com/oss/azure/aad-pod-identity/nmi:v$(helm show chart aad-pod-identity/aad-pod-identity | grep appVersion |  awk -F" " '{print $2}') \
-n $ACR_NAME --force
az acr import --source mcr.microsoft.com/oss/azure/aad-pod-identity/mic:v$(helm show chart aad-pod-identity/aad-pod-identity | grep appVersion |  awk -F" " '{print $2}') \
-n $ACR_NAME --force

# CREATE HELMRELEASE MANIFEST FOR AAD-POD-IDENTITY
mkdir -p $GITHUB_REPO/infrastructure/base/aad-pod-identity
cat > $GITHUB_REPO/infrastructure/base/aad-pod-identity/release.yaml << ENDOFFILE
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: aad-pod-identity
spec:
  releaseName: aad-pod-identity
  chart:
    spec:
      chart: aad-pod-identity
      sourceRef:
        kind: HelmRepository
        name: acr-private
        namespace: flux-system
      version: "${AADPODIDENTITY_CHART_VERSION}"
  interval: 1h0m0s
  install:
    remediation:
      retries: 3
  # Default values
  # https://github.com/Azure/aad-pod-identity/blob/master/charts/aad-pod-identity/values.yaml
  values:    
    image:
      repository: ${ACR_NAME}.azurecr.io/oss/azure/aad-pod-identity
ENDOFFILE

# KUSTOMIZATION MANIFEST FOR AAD-POD-IDENTITY
cat > $GITHUB_REPO/infrastructure/base/aad-pod-identity/kustomization.yaml << ENDOFFILE
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: baseline
resources:  
  - release.yaml
ENDOFFILE

# COMMIT AND PUSH THE CHANGES
cd $GITHUB_REPO
git add *
git commit -m "aad-pod-identity installation"
git push
cd ..

Building an AKS baseline architecture - Part 3 - GitOps with Flux2

2021-10-16T02:00:00+02:00

Posts from this series:

Building an AKS baseline architecture - Part 1 - Cluster creation

Building an AKS baseline architecture - Part 2 - Governance

Building an AKS baseline architecture - Part 3 - GitOps with Flux2

Building an AKS baseline architecture - Part 4 - AAD Pod Identity

What is GitOps?

“GitOps is a way of implementing Continuous Deployment for cloud native applications. It focuses on a developer-centric experience when operating infrastructure, by using tools developers are already familiar with, including Git and Continuous Deployment tools. The core idea of GitOps is having a Git repository that always contains declarative descriptions of the infrastructure currently desired in the production environment and an automated process to make the production environment match the described state in the repository. If you want to deploy a new application or update an existing one, you only need to update the repository - the automated process handles everything else. It’s like having cruise control for managing your applications in production.”- GitOps.tech

There is a great GitOps for AKS document available in the Azure Architecture Center that can help you learn more about the need and the benefits of using the GitOps approach for AKS. In this article, I will focus on the implementation part of GitOps - how to make it work with Flux.

Flux

Flux is a tool for keeping Kubernetes clusters in sync with sources of configuration (like Git repositories), and automating updates to configuration when there is new code to deploy.

There are multiple approaches for organizing the git repos when enrolling Flux. In the baseline architecture, we will implement the monorepo approach - a single repository for multiple environments. Environments will have the same base infrastructure components, but the versions of custom applications will be different:

├── apps
│   ├── production 
│   └── staging
├── infrastructure
│   ├── sources
│   ├── base 
└── clusters
    ├── production
    └── staging

The design is not optimal for enterprise deployments, but it is a good starting point for learning.

As a best practice, you should accommodate the trank-based development approach: create minor updates in short-living branches that you will merge to main by opening pull requests. For the simplicity of the demo, we will perform all commits directly to our main branch.

Install Flux

export GITHUB_USER=
export GITHUB_TOKEN= # PERSONAL ACCESS TOKEN FOR GITHUB. SCOPE: Full control of private repositories
export GITHUB_REPO=aks-baseline-clusters # REPO FOR FLUX
export GITHUB_PATH=clusters/production # PATH TO USE INSIDE THE REPO 

curl -s https://fluxcd.io/install.sh | sudo bash # INSTALL FLUX LOCALY

# ENABLE COMPLETITIONS in ~/.bash_profile
. <(flux completion bash)

# IMPORT THE FLUX IMAGES IN OUR AZURE CONTAINER REGISTRY (https://aztoso.com/aks/baseline-part-2/#azure-container-registry-acr)
az acr import --source $(flux install --export | grep ghcr.io/fluxcd/helm-controller | awk -F" " '{print $2}') -n $ACR_NAME --force
az acr import --source $(flux install --export | grep ghcr.io/fluxcd/kustomize-controller | awk -F" " '{print $2}') -n $ACR_NAME --force
az acr import --source $(flux install --export | grep ghcr.io/fluxcd/notification-controller | awk -F" " '{print $2}') -n $ACR_NAME --force
az acr import --source $(flux install --export | grep ghcr.io/fluxcd/source-controller | awk -F" " '{print $2}') -n $ACR_NAME --force

# BOOTSTRAP FLUX
flux bootstrap github --owner=$GITHUB_USER --repository=$GITHUB_REPO \
--path=$GITHUB_PATH --personal --registry $ACR_NAME.azurecr.io/fluxcd

# CLONE THE REPO LOCALY
git clone https://$GITHUB_USER:$GITHUB_TOKEN@github.com/$GITHUB_USER/$GITHUB_REPO.git  ../$GITHUB_REPO

We will use Azure Container Registry to host our Helm Charts. Because OCI is not yet supported by Flux, we can’t use Managed Identities for authenticating to the ACR. The only option is to use service principal (make sure you regulary rotate the secret):

# CREATE A SERVICE PRINCIPAL WITH PERMISSION TO PULL FROM THE ACR
fluxHelmAcr=$(az ad sp create-for-rbac -n sp-aks-baseline-$ACR_NAME-acrpull --role "AcrPull" --scope $(az acr show -g $RESOURCE_GROUP_ACR -n $ACR_NAME --query id -o tsv --only-show-errors)
FLUX_HELM_ACR_APPID=$(echo -n $fluxHelmAcr | jq '.appId' -r | tr -d '\n'| base64)
FLUX_HELM_ACR_SECRET=$(echo -n $fluxHelmAcr | jq '.password' -r | tr -d '\n'| base64)

# KUBERNETES SECRET THAT STORES THE SERVICE PRINCIPAL DETAILS
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: flux-helm-acr-credentials
  namespace: flux-system
type: Opaque
data:
  username: ${FLUX_HELM_ACR_APPID}
  password: ${FLUX_HELM_ACR_SECRET}
EOF

# THE KUSTOMIZATION DEFINITION FOR THE CLUSTER:
mkdir -p $GITHUB_REPO/clusters/production
cat > $GITHUB_REPO/clusters/production/infrastructure.yaml << ENDOFFILE
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  interval: 10m0s
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure
  prune: true
  validation: client
ENDOFFILE

cd $GITHUB_REPO
git add *
git commit -m "infrastructure production kustomization"
git push
cd ..

# CREATE HELMREPOSITORY MANIFEST FOR THE ACR
mkdir -p $GITHUB_REPO/infrastructure/sources
cat > $GITHUB_REPO/infrastructure/sources/acr.yaml << ENDOFFILE
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: acr-private
  namespace: flux-system
spec:
  url: https://${ACR_NAME}.azurecr.io/helm/v1/repo
  secretRef:
    name: flux-helm-acr-credentials
  interval: 1m
ENDOFFILE

# COMMIT AND PUSH THE CHANGES
cd $GITHUB_REPO
git add *
git commit -m "acr helm repository"
git push
cd ..

To verify that Flux created the Helm repository definition on the cluster, execute the following command:

kubectl get helmrepositories.source.toolkit.fluxcd.io -n flux-system

NAME          URL                                                READY   STATUS                                                       AGE
acr-private   https://xw5OUsrKQmU40Im4.azurecr.io/helm/v1/repo   True    Fetched revision: c45818b304c4b776703ab0fc56efbe492690830e   3h9m

Sample Kured Helm deployment using Flux

Kured (Kubernetes Reboot Daemon) is a DaemonSet that triggers a node reboot if a file exists at the predefined path. The underlying OS in AKS, Ubuntu 18.04, performs a security or kernel patch check every night and automatically installs the patches. If the patch requires a restart, AKS will create a/var/run/reboot-required file. In simple words, Kured makes sure that all critical updates are installed on the cluster.

Note: Kured needs to be configured with the following tolerations to run on the dedicated system nodes:

    tolerations:
    - key: CriticalAddonsOnly
      operator: Exists

Deploy Kured using HelmRelease and Kustomization:

# PULL THE HELM CHART LOCALY
helm repo add kured https://weaveworks.github.io/kured
helm pull kured/kured

# PUSH THE HELM CHART TO ACR
az acr helm push -n $ACR_NAME kured-*.tgz

# PUSH THE IMAGE TO ACR
KURED_CHART_VERSION=$(helm show chart kured/kured | grep version |  awk -F" " '{print $2}')
az acr import --source docker.io/weaveworks/kured:$(helm show chart kured/kured | grep appVersion |  awk -F" " '{print $2}') \
-n $ACR_NAME --force

# NAMESPACE TO USE FOR BASELINE PODS
cat > $GITHUB_REPO/infrastructure/base/namespace/namespace.yaml << ENDOFFILE
apiVersion: v1
kind: Namespace
metadata:
  name: baseline
ENDOFFILE

# KUSTOMIZATION MANIFEST FOR NAMESPACE
cat > $GITHUB_REPO/infrastructure/base/namespace/kustomization.yaml << ENDOFFILE
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: baseline
resources:
  - namespace.yaml
ENDOFFILE

# CREATE HELMRELEASE MANIFEST FOR KURED
mkdir -p $GITHUB_REPO/infrastructure/base/kured
cat > $GITHUB_REPO/infrastructure/base/kured/release.yaml << ENDOFFILE
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: kured
spec:
  releaseName: kured
  chart:
    spec:
      chart: kured
      sourceRef:
        kind: HelmRepository
        name: acr-private
        namespace: flux-system
      version: "${KURED_CHART_VERSION}"
  interval: 1h0m0s
  install:
    remediation:
      retries: 3
  # Default values
  # https://github.com/weaveworks/kured/blob/main/charts/kured/values.yaml
  values:
    tolerations:
    - key: CriticalAddonsOnly
      operator: Exists
    image:
      repository: ${ACR_NAME}.azurecr.io/weaveworks/kured
ENDOFFILE

# KUSTOMIZATION MANIFEST FOR KURED
cat > $GITHUB_REPO/infrastructure/base/kured/kustomization.yaml << ENDOFFILE
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: baseline
resources:
  - release.yaml
ENDOFFILE

# COMMIT AND PUSH THE CHANGES
cd $GITHUB_REPO
git add *
git commit -m "kured installation"
git push
cd ..

Useful flux commands

flux logs

The logs command displays formatted logs from various Flux components. Check the usage:

flux logs --help

Executing the command without any flags will display all of the flux related logs:

flux logs

...
2021-10-17T17:32:11.598Z info GitRepository/flux-system.flux-system - Reconciliation finished in 1.197141019s, next run in 1m0s
2021-10-17T17:32:40.881Z info HelmRepository/acr-private.flux-system - Reconciliation finished in 113.821332ms, next run in 1m0s
2021-10-17T17:33:12.798Z info GitRepository/flux-system.flux-system - Reconciliation finished in 1.200518767s, next run in 1m0s
2021-10-17T17:33:41.017Z info HelmRepository/acr-private.flux-system - Reconciliation finished in 134.590725ms, next run in 1m0s

flux reconcile

The reconcile sub-commands trigger a reconciliation of sources and resources. Check the usage:

flux reconcile --help

Using the configuration from above, if you want to reconcile the complete deployment, execute:

flux reconcile kustomization infrastructure

► annotating Kustomization infrastructure in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ applied revision main/2a85d7ec6a63976aaac090197ff0882540a076c0

Azure Function keys - what are those and how to access them

2021-10-09T02:00:00+02:00

Usage of keys in Azure Functions

Azure Functions offer three levels of authorization:

anonymous: no key is required
function: a function-specific or a host key is required
admin: the master key is required

Function access keys are the simple way to protect an Azure Function from unauthorized access. You can provide the access key in the code query parameter or in the x-functions-key HTTP header when accessing a function. There are two access scopes for function-level keys:

function: these keys apply only to the specific functions under which they are defined
host: keys with a host scope can be used to access all functions within the function app

The master key also provides access to all functions within the function app and administrative access to the runtime REST APIs. As a best practice, never use the master key in client applications to access an Azure Function.

One particular type of key, named system key, is used by extensions installed in the Azure Function, such as Event Grid and Durable Functions.

How to get the keys value?

In the examples below, the following applies:

rg-functions-test - resource group where the function app is deployed
aztosotest - the name of the function app
Hello - the name of the function

Azure Portal

After opening the Function App in the portal, you can view the host and system keys by selecting App keys from the left sidebar:

To view the function-specific keys, open the function and chose Function Keys from the left sidebar:

ARM template

Note: if the function is deployed in a different resource group, you need to specify the resource group name as a first parameter in the resourceId function

Host(default) key:

listkeys(concat(resourceId('Microsoft.Web/sites', 'aztosotest'), '/host/default/'),'2021-02-01').functionKeys.default

Master key:

listkeys(concat(resourceId('Microsoft.Web/sites', 'aztosotest'), '/host/default/'),'2021-02-01').masterKey

System key (named durabletask_extension):

listkeys(concat(resourceId('Microsoft.Web/sites', 'aztosotest'), '/host/default/'),'2021-02-01').systemkeys.durabletask_extension

Function-specific key:

listKeys(resourceId('Microsoft.Web/sites/functions', 'aztosotest', 'Hello'),'2021-02-01').default

Azure CLI

To get the host, master, and system key, use the following command:

az functionapp keys list -g rg-functions-test -n aztosotest

{
  "functionKeys": {
    "default": "68ahyfSAT2PF5HfIuGMyc8Bv6NErsFxnk6lT0l2fYacYCnVameOAmw=="
  },
  "masterKey": "N4O1a4ng/gLemQEiDFF9HUpNznT/pFGNSnTj4jQXVOhvZAa8MivuOg==",
  "systemKeys": {
    "durabletask_extension": "DtieTxAOV5x76Zfv6NWBqbbW3/u4YiHVhycZfPgn2VIgkcPAGP8kaA=="
  }
}

The following command will list the function-specific keys:

az functionapp function keys list -g rg-functions-test -n aztosotest --function-name Hello

{
  "default": "ukRsq3yoeayz5ELYgTGK3fbU3yfbHXavy5m9Y6ci0ZSykqFYUxyd3Q==",
  "id": null,
  "kind": null,
  "name": null,
  "properties": null,
  "systemData": null,
  "type": null
}