AWS Database Migration Service (Part 2 of 2)

In the previous article we went over the high-level approach around using AWS DMS and then created the replication instance on which your migration processing will run and then created the source and target endpoints that manage the connection to the source and target databases. The last step is to create the database migration task.

Creating your Database Migration Task

So far, we have defined the resource set that is going to do the work as well as the places where data will be coming from and where it will be going. There is one area that we have not yet defined, and that is the database migration task. This task defines the work that will be done. As part of this task, you can specify which tables to use, define any special processing, configure logging, etc. Let’s take a look at creating one of these tasks.

First, go into the Database migration tasks screen in the AWS DMS console and then click the Create task button. This will bring up the creation screen, with the first section being Task configuration. This section allows you to:

·         Provide a Task identifier or name for the task

·         Select the Replication instance to use

·         Select the Source database endpoint

·         Select the target database endpoint

·         Select the Migration type

The Migration type is where you tell DMS the kind of work that you want this task to perform. There are three different options that you can select. The first is to Migrate existing data. Using this as a migration type means that you’re looking to do a one-time copy of the data and would be ideal for doing that one-time migration. The next option is to Migrate existing data and replicate ongoing changes. The name pretty much describes what is going on with this approach, and is most appropriate when you need to run both the source and target systems in parallel but want them to stay as updated as possible. This approach is especially common in Data Lake scenarios where data is being moved from a transactional system to an analytics or reporting system. The last migration type option is to Replicate data changes only where you replicate any changes in data but do not perform that one-time migration.

The next major section to complete when creating a migration task is the Task settings. Task settings control the behavior of your task and can be configured through a Wizard or through a JSON editor. We will use the wizard mode so that we can more easily talk about the major settings.

The first item to configure is the Target table preparation mode, or how DMS should be preparing the tables at the target endpoint. There are three options, Do nothing, Drop tables on target, and Truncate. When you select the “do nothing” option then target tables will not be affected. Any tables that do not exist will be created. When you select to drop the tables, then DMS will drop and recreate all affected tables. Truncating means that all tables and metadata remain, but all of the data is removed.

The next item to configure is Include LOB columns in replication. LOB are large objects and you have the option as to whether or not you want to include those object columns in the target data. You have three options, the first of which is Don’t include LOB columns, and the second of which is Full LOB mode; both of which are rather straightforward. The third option is Limited LOB mode. In this mode, DMS will truncate each LOB to a defined size, the Maximum LOB size (kb) value.

You then can configure whether you want to Enable validation. Checking this box will cause DMS to compare the source and target data immediately after the full load is performed. This ensures your data is migrated correctly, but it takes additional time to perform, and thus increases cost. You next can Enable CloudWatch logs. There are also some advanced task settings, but we won’t go into those as part of this discussion.

The next section is Table mappings. This section is where you define the rules about what data is moved and how it is moved. At a high-level, you will create a Selection rule, which determines the data that you wish to replicate, and then you can create a Transformation rule that modifies the selected data before it is provided to the destination endpoint. The table mappings section also gives you the opportunity to use a Wizard approach or a JSON editor to enter all table mappings in JSON. We will walk through using the wizard.

The first step is to select the Add new selection rule button. This expands the selection rule section as shown in Figure 1.

Figure 1. Creating selection rules for a database migration task
Figure 1. Creating selection rules for a database migration task

Expanding the Schema drop-down will show that there is only one option – to Enter a schema. Selecting this option will add another textbox in which you can provide the Source name. This allows you to limit, by schema, the data that is being selected. You can enter % to select all schemas in the database or enter the schema name. You do the same for the Source table name, entering % if you want all the tables replicated. Once you have those defined, you then select the appropriate Action, to either Include or Exclude the items that fit your selection criteria. You can create as many rules as desired, however, you must always have at least one rule with an include action.

Once you have the selection rule configured you can Add column filter. This allows you to limit the number and type of records. A column filter requires the Column name, one or more Conditions, and then one or more comparative values. You have the following options for the conditions:

·         Less than or equal to

·         Greater than or equal to

·         Equal to

·         Not equal to

·         Equal to or between to values

·         Note between two values

·         Null

·         Not null

You can create any number of column filters per each selection rule.

Once you have completed your selection rule you can then add one or more Transformation rules. These rules allow you to change or transform schema, table, or column names of some or all the items that you have selected. Since we are simply copying the database across, we do not need to add any of these, especially since any changes will likely break our code!

Your next option is to determine whether you want to Enable premigration assessment run. This will warn you of any potential migration issues. Checking the box will expand the UI and present you with a set of Assessments to run as shown in Figure 2.

Figure 2. Enabling premigration assessment run on a scheduled task
Figure 2. Enabling premigration assessment run on a scheduled task

Once you have all of your selection and transformation rules created, you can select to Start migration task either Automatically on Create, the default, or Manually later. Lastly, add any tags that you desire and click the Create task button.

This will bring you back to the database migration tasks list screen where you will see your task being created. Once created you can either start the task manually or allow it to run itself if so configured. You will be able to watch the table count move from Tables queued to Tables Loading to Tables loaded as they are processed. Returning to the AWS DMS Dashboard will show that there is 1 Load complete as shown in Figure 3.

Figure 3. Dashboard showing completed migration task
Figure 3. Dashboard showing completed migration task

For those cases where you simply want to migrate data sets with minimal changes other than perhaps renaming some columns, the Database Migration Service works like a dream. Relatively painless to setup and powerful enough to move data between servers, even servers that are of dissimilar types, such as where we just copied data from SQL Server to Amazon Aurora. However, there is a tool that will help you move more disparate data between different database engines. We will take a look at that tool next.

AWS Database Migration Service (Part 1 of 2)

The AWS Database Migration Service (AWS DMS) was designed to help quickly and securely migrate databases into AWS. The premise is that the source database remains available during the migration to help minimize application downtown. AWS DMS supports homogeneous migrations such as SQL Server to SQL Server or Oracle to Oracle as well as some heterogeneous migrations between different platforms. You can also use the service to continuously replicate data from any supported source to any supported target, meaning you can use DMS for both one-time replications as well as ongoing replications. AWS DMS works with relational databases and NoSQL databases as well as other types of data stores. One thing to note, however, is that at least one end of your migration must be on an AWS service, you cannot use AWS DMS to migrate between two on-premises databases.

How Does it Work?

You can best think of DMS as replication software running on a server in the cloud. There are literally dozens of these kinds of tools, some cloud-based, some that you install locally to move data between on-premise systems. The DMS’ claim to fame is that you only pay for the work that you have it perform – there is no licensing fee for the service itself like with most of the other software solutions.

Figure 1 shows DMS at a high level. The green box in Figure 1 is the overall service and contains three major subcomponents. Two of these are endpoints used to connect to the source and target databases, and the third is the replication instance.

Figure 1. A high-level look at AWS Data Migration Service
Figure 1. A high-level look at AWS Data Migration Service

The replication instance is an Amazon EC2 instance that provides the resources necessary to carry out the database migration. Since it is a replication instance, you can get high availability and failover support if you select to use a multi-region-based process.

AWS DMS uses this replication instance to connect to your source database through the source endpoint. The instance then reads the source data and performs any data formatting necessary to make it compatible with the target database. The instance then loads that data into the target database. Much of this processing is done in memory, however large data sets may need to be buffered onto disk as part of the transfer. Logs and other replication-specific data are also written onto the replication instance.

Creating a Replication Instance

Enough about the way that it is put together, let’s jump directly into creating a migration service, and we will go over the various options as they come up in the process.

Note: Not all EC2 instance classes are available for use as a replication instance. As of the time of this writing, only T3 (general purpose), C5 (compute-optimized), and R5 (memory-optimized) Amazon EC2 instance classes can be used. You can use a t3.micro instance under the AWS Free Tier, however, there is a chance that you may be charged if the utilization of the instance over a rolling 24-hour period exceeds the baseline utilization. This will not be a problem in our example, but it may be with other approaches, especially if you use ongoing replication.

You can get to the AWS DMS console by searching for “DMS” or by going into the Migration & Transfer service group and selecting it there. Click the Create replication instance button once you get to the console landing page. This will take you to the creation page. Remember as you go through this that all we are doing here is creating the EC2 instance that DMS will use for processing, so all the questions will be around that.

The fields that you can enter in the Replication instance configuration section are:

·         Name – must be unique across all replication instances in the current region

·         Descriptive Amazon Resource Name (ARN) – This field is optional, but it allows you to use a friendly name for the ARN rather than the typical set of nonsense that AWS creates by default. This value cannot be changed after creation.

·         Description – Short description of the instance

·         Instance class – This is where you select the instance class on which your migration process will be running.

·         Engine version – This option allows the targeting of previous versions of DMS, or the software that runs within the instance class – though we have no idea why you would ever target an older version.

·         Allocated storage – The amount of storage space that you want in your instance. This is where items like log files will be stored and will also be used for disc caching if the instance’s memory is not sufficient to handle all of the processing.

·         VPC – Where the instance should be run.

·         Multi AZ – You can choose between Production workload which will set up multi-AZ or Dev or test workload which will create the instance in a single AZ.

·         Publicly accessible – This is necessary if you are looking to connect to databases outside of your VPC, or even outside of AWS.

There are three additional sections that you can configure. The first of these is Advanced security and network configuration where you can define the specific subnet group for your replication instance, the availability zone in which your replication instance should run, and VPC security groups that you want to be assigned to your replication instance, and the AWS Key Management Service key that you would like used.

The next section is Maintenance, where you can define the weekly maintenance window that AWS will use for maintaining the DMS engine software and operating system. You must have this configured, and AWS will set up a default window for you. The last section that you can configure is, of course, Tags.

Once you click the Create button you will see that your replication instance is being created as shown in Figure 2. This creation process will take several minutes.

Figure 2. Creating a DMS replication instance
Figure 2. Creating a DMS replication instance

Now that you have a replication instance, the next step is to create your endpoints.

Creating your Source and Target Endpoints

As briefly mentioned above, the endpoints manage the connection to your source and target databases. They are managed independently from the replication instance because there are many cases where there are multiple replications that talk to a single source or target, such as copying one set of data to one target and another set of data from the same source to a second target such as shown in Figure 3.

Figure 3. Multiple replications against a single source endpoint
Figure 3. Multiple replications against a single source endpoint

To create an endpoint, go into Endpoints and select Create endpoint. This will bring up the Create endpoint screen. Your first option is to define the Endpoint type, as shown in Figure 4.

Figure 4. Endpoint type options when creating a DMS endpoint
Figure 4. Endpoint type options when creating a DMS endpoint

Your first option when creating the endpoint is to determine whether the endpoint is going to be a source or target endpoint. You would think that this wouldn’t really matter because a database connection is a database connection whether you are reading or writing, but DMS has made decisions around which databases they will support reading from and which databases you can write to, and, as you can likely predict, they are not the same list. Table 1 lists the different databases supported for each endpoint type, as of the time of this writing.

DatabaseAs SourceAs Target
Oracle v10.2 and laterXX
SQL Server 2005 and laterXX
MySQL 5.5 and laterXX
MariaDB 10.0.24 and laterXX
PostgreSQL 9.4 and laterXX
SAP Adaptive Server Enterprise (ASE) 12.5 and aboveXX
IBM DB2 multiple versionsX 
Redis 6.x X
Azure SQL DatabaseX 
Google Cloud for MySQLX 
All RDS instance databasesX 
Amazon S3X 
Amazon DocumentDBX 
Amazon OpenSearch Service X
Amazon ElastiCache for Redis X
Amazon Kinesis Data Streams X
Amazon DynamoDB X
Amazon Neptune X
Apache Kafka X
Table 1. Databases available as sources and targets

The next option in the Endpoint type section is a checkbox to Select RDS DB instance. Checking this box will bring up a dropdown containing a list of RDS instances as shown in Figure 5.

Figure 5. Selecting an RDS database when creating an endpoint
Figure 5. Selecting an RDS database when creating an endpoint

 The next section is the Endpoint configuration. There are two primary sections to this section, the first section allows you to name the endpoint and select the type of database to which you are connecting and the second is Endpoint settings where you can define those additional settings needed to access a specific database. Selecting the Source\Target engine will expand the form, adding some additional fields.

The first of these fields is Access to endpoint database. There are two options available and the choice you make will change the rest of the form. These two options are AWS Secrets Manager, where you use stored secrets for the login credentials, or Provide access information manually where you manually configure the database connection.

Selecting to use AWS Secrets Manager will bring up additional fields as described below. These fields are used to fetch and access the appropriate secret.

·         Secret ID – the actual secret to be used when logging into the database

·         IAM role – the IAM role that grants Amazon DMS the appropriate permissions to use the necessary secret

·         Secure Socket Layer (SSL) mode – whether to use SSL when connecting to the database.

Selecting to Provide access information manually brings up the various fields necessary to connect to that identified engine. Figure 6 shows what this looks like when connecting to a SQL Server, and hopefully, all these values look familiar because we have used them multiple times in earlier articles.

Figure 6. Providing SQL Server information manually for an endpoint
Figure 6. Providing SQL Server information manually for an endpoint

The next section is the Endpoint settings section. The purpose of this section is to add any additional settings that may be necessary for this particular instance of the database to which it is connecting. There are two ways in which you can provide this information. The first is through a Wizard, while the second is through an Editor. When using the Wizard approach, clicking the Add new setting button will bring up a Setting \ Value row, with the Setting being a drop-down list of known settings as shown in Figure 7. These values will be different for each engine as well as whether you are using the endpoint as a source or a target.

Figure 7. Endpoint settings section when creating a SQL Server endpoint
Figure 7. Endpoint settings section when creating a SQL Server endpoint

Selecting to use the Editor approach will bring up a large text box where you can enter the endpoint settings in JSON format. This would likely be the best approach if you need to configure multiple DMS endpoints with the same additional settings.

Once you have Endpoint configuration section complete, the next section is KMS key where you select the appropriate key to be used when encrypting the data that you have input into the configuration. The next section is Tags. The last section entitled Test endpoint connection (optional) is shown in Figure 8 and is where you can test all the information that you have just filled out.

Figure 8. Testing an endpoint configuration
Figure 8. Testing an endpoint configuration

There are two values that you must identify before you can run the test, and that is the VPC and replication instance that you want to use, which is why we had you create the replication instance first! These are necessary because these are the resources that will be used to perform the work of connecting to the database. Once the values are selected, click the Run test button. After a surprising amount of time where you see indications that the test is running, you should get confirmation that your test was successful. This output is shown in Figure 9.

Figure 9. Successful test on an endpoint configuration
Figure 9. Successful test on an endpoint configuration

Obviously, you will need to configure at least one source endpoint and one target endpoint before you can run DMS end to end. However, you also need to make sure that you have each of them configured before you can configure the database migration task. We’ll finish that up in the next article!

Deploying New Container Using AWS App2Container

In our last article, we went through the containerization of a running application. The last step of this process is to deploy the container. The default approach is to deploy a container image to ECR and then create the CloudFormation templates to run that image in Amazon ECS using Fargate. If you would prefer to deploy to Amazon EKS instead, you will need to go to the deployment.json file in the output directory. This editable file contains the default settings for the application, ECR, ECS, and EKS. We will walk through each of the major areas in turn.

The first section is responsible for defining the application and is shown below.

"a2CTemplateVersion": "1.0",
"applicationId": "iis-tradeyourtools-6bc0a317",
"imageName": "iis-tradeyourtools-6bc0a317",
"exposedPorts": [
       {
              "localPort": 80,
              "protocol": "http"
       }
],
"environment": [],

The applicationId and the imageName are values we have seen before when going through App2Containers. The exposedPorts value should contain all of the IIS ports configured for the application. The one used in the example was not configured for HTTPS, but if it was there would be another entry for that value. The environment value allows you to enter any environment variables as key/value pairs that may be used by the application. Unfortunately, App2Container is not able to determine those because it does its analysis on running code rather than the code base. In our example, there are no environmental variables that are necessary.

Note – If you aren’t sure whether there are environment variables that your application may access, you can see which variables are available by going into the System -> Advanced system settings -> Environment variables. This will provide you with a list of available variables and you can evaluate those as to their relevance to your application.

The next section is quite small and contains the ECR configuration. The ECR repository that will be created is named with the imageName from above and then versioned with the value in the ecrRepoTag as shown below.

"ecrParameters": {
       "ecrRepoTag": "latest"
},

We are using the value latest as our version tag.

There are two remaining sections in the deployment.json file. The first is the ECS setup information with the second being the EKS setup information. We will first look at the ECS section. This entire section is listed below.

"ecsParameters": {
       "createEcsArtifacts": true,
       "ecsFamily": "iis-tradeyourtools-6bc0a317",
       "cpu": 2,
       "memory": 4096,
       "dockerSecurityOption": "",
       "enableCloudwatchLogging": false,
       "publicApp": true,
       "stackName": "a2c-iis-tradeyourtools-6bc0a317-ECS",
       "resourceTags": [
              {
                     "key": "example-key",
                     "value": "example-value"
              }
       ],
       "reuseResources": {
              "vpcId": "vpc-f4e4d48c",
              "reuseExistingA2cStack": {
                     "cfnStackName": "",
                     "microserviceUrlPath": ""
              },
              "sshKeyPairName": "",
              "acmCertificateArn": ""
       },
       "gMSAParameters": {
              "domainSecretsArn": "",
              "domainDNSName": "",
              "domainNetBIOSName": "",
              "createGMSA": false,
              "gMSAName": ""
       },
       "deployTarget": "FARGATE",
       "dependentApps": []
},

The most important value here is createEcsArtifacts, which if set to true means that deploying with App2Container will deploy the image into ECS. The next ones to look at are cpu and memory. These values are only used for Linux containers. In our case, these values do not matter because this is a Windows container. The next two values, dockerSecurityOption and enableCloudwatchLogging are only changed in special cases, so they will generally stay at their default values. The next value, publicApp, determines whether the application will be configured into a public subnet with a public endpoint. This is set to true because this is our hoped-for behavior. The next value, stackName, defines the name of the CloudFormation stack while the value after that, resourceTags, are the custom tags that should be added to the ECS task definition. There is a default set of key/values in the file, but those will not be used if kept in; only keys that are not defined as example-key will be added.

The next section, reuseResources, is where you can configure whether you wish to use any pre-existing resources, namely VPC – which is added to the vpcId value. When left blank, as shown below, App2Container will create a new VPC.

"reuseResources": {
     "vpcId": "",
     "reuseExistingA2cStack": {
            "cfnStackName": "",
            "microserviceUrlPath": ""
     },
     "sshKeyPairName": "",
     "acmCertificateArn": ""
}

Running the deployment with these settings will result in a brand new VPC being created. This means that, by default, you wouldn’t be able to connect in or out of the VPC without making changes to the VPC. If, however, you have an already existing VPC that you want to use, update the vpcId key with the ID of the appropriate VPC.

Note: App2Container requires that the included VPS has a routing table that is associated with at least two subnets and an internet gateway. The CloudFormation template for the ECS service requires this so that there is a route from your service to the internet from at least two different AZs for availability. Currently, there is no way for you to define these subnets. You will receive a Resource creation failures: PublicLoadBalancer: At least two subnets in two different Availability Zones must be specified message if your VPC is not set up properly.

You can also choose to reuse an existing stack created by App2Container. Doing this will ensure that the application is deployed into the already existing VPC and that the URL for the new application is added to the already created Application Load Balancer rather than being added to a new ALB.

The next value, sshKeyPairName, is the name of the EC2 key pair used for the instances on which your container runs. Using this rather defeats the point of using containers, so we left it blank as well. The last value, acmCertificateArn, is for the AWS Certificate Manager ARN that you want to use if you are enabling HTTPS on the created ALB. This parameter is required if you use an HTTPS endpoint for your ALB, and remember as we went over earlier this means that the request being forwarded into the application will be on port 80 and unencrypted because this would have been handled in the ALB.

The next set of configuration values are part of the gMSAParameters section. This becomes important to manage if your application relies upon group Managed Service Account (gMSA) Active Directory groups. This can only be used if deploying to EC2 and not Fargate (more on this later). These individual values are:

·         domainSecretsArn – The AWS Secrets Manager ARN containing the domain credentials required to join the ECS nodes to Active Directory.

·         domainDNSName – The DNS Name of the Active Directory the ECS nodes will join.

·         domainNetBIOSName – The NetBIOS name of the Active Directory to join.

·         createGMSA – A flag determining whether to create the gMSA Active Directory security group and account using the name supplied in the gMSAName field.

·         gMSAName – The name of the Active Directory account the container should use for access.

There are two fields remaining, deployTarget and dependentApps. For deployTarget there are two valid values for .NET applications running on Windows; fargate and ec2. You can only deploy to Fargate if your container is Windows 2019 or more recent. This would only be possible if your worker machine, the one you used for containerizing, was running Windows 2019+. Also, you cannot deploy to Fargate if you are using gMSA.

The value dependentApps is interesting, as it handles those applications that AWS defines as “complex Windows applications”. We won’t go into it in more details here, but you can go to https://docs.aws.amazon.com/app2container/latest/UserGuide/summary-complex-win-apps.html if you are interested in learning more about these types of applications.

The next section in the deployment.json file is eksParameters. You will see that much of these parameters are the same as what we went over when talking about the ECS parameters. The only differences are the createEksArtifacts parameter, which needs to be set to true if deploying to EKS, and in the gMSA section, the gMSAName parameter has inexplicably been changed to gMSAAccountName.

Once you have the deployment file set as desired, you next deploy the container:

PS C:\App2Container> app2container generate app-deployment --application-id APPID --deploy

This process takes several minutes, and you should get an output like Figure 1. The gold arrow points to the URL where you can go see your deployed application – go ahead and look at it to confirm that it has been successfully deployed and is running.

Figure 1. Output from generating an application deployment in App2Container

Logging in to the AWS console and going to Amazon ECR will show you the ECR repository that was created to store your image as shown in Figure 2.

Figure 2. Verifying the new container image is available in ECR

Once everything has been deployed and verified, you can poke around in ECS to see how it is all put together. Remember though, if you are looking to make modifications it is highly recommended that you use the CloudFormation templates, make the changes there, and then re-upload them as a new version. That way you will be able to easily redeploy as needed and not worry about losing any changes that you may have added. You can either alter the templates in the CloudFormation section of the console or you can find the templates in your App2Container working directory, update those, and then use those to update the stack.

Containerizing a Running Application with AWS App2Container

Now that we have gone through containerizing an already existing application where you have access to the source code, let’s look at containerizing a .NET application in a different way. This is for those applications you may have that are running and where you may not have access to the source code, or you don’t deploy it, or there are other reasons where you don’t want to change the source code as we just went over earlier. Instead, you want to containerize the application by just “picking it up off its server” and moving it into a container. Up until recently, that was not a simple thing to do. However, AWS created a tool to help you do just that. Let’s look at that now.

What is AWS App2Container?

AWS App2Container is a command-line tool that is designed to help migrate .NET web applications into a container format. You can learn more about and download this tool at https://aws.amazon.com/app2container/.  It also does Java, but hey, we’re all about .NET, so we won’t talk about that anymore! You can see the process in Figure 1, but at a high level, there are five major steps.

Figure 1. How AWS App2Container works

These steps are:

1.      Inventory – This step goes through the applications running on the server looking for running applications. At the time of writing, App2Container supports ASP.NET 3.5, and greater, applications running in IIS 7.5+ on Windows.

2.      Analyze – A chosen application is analyzed in detail to identify dependencies including known cooperating processes and network port dependencies. You can also manually add any dependencies that App2Container was unable to find.

3.      Containerize – In this step, all the application artifacts discovered during the “Analyze” phase are “dockerized.”

4.      Create – This step creates the various deployment artifacts (generally as CloudFormation templates) such as ECS task or Kubernetes pod definitions.

5.      Deploy – Store the image in Amazon ECR and deploy to ECS or EKS as desired.

There are three different modes in which you can use App2Container. The first is a mode where you perform the steps on two different machines. If using this approach, App2Container must be installed on both machines. The first machine, the Server, is the machine on which the application(s) that you want to containerize is running. You will run the first two steps on the server. The second machine, the Worker, is the machine that will perform the final three steps of the process based on artifacts that you copy from the server. The second mode is when you perform all the steps on the same machine, so it basically fills both the server and worker roles. The third mode is when you run all the commands on your worker machine, connecting to the server machine using the Windows Remote Management (WinRM) protocol. This approach has the benefit of not having to install App2Container on the server, but it also means that you must have WinRM installed and running. We will not be demonstrating this mode.

App2Container is a command-line tool that has some prerequisites that must be installed before the tool will run. These prerequisites are listed below.

·         AWS CLI – must be installed on both server and worker

·         PowerShell 5.0+ – must be installed on both server and worker

·         Administrator rights – You must be running as a Windows administrator

·         Appropriate permissions – You must have AWS credentials stored on the worker machine as was discussed in the earlier articles when installing the AWS CLI.

·         Docker tools – Docker version 17.07 or later must be installed on worker

·         Windows Server OS – Your worker system must run on Windows OS versions that support containers, namely Windows Server 2016 or 2019. If working in server\worker mode, the server system must be Windows 2008+.

·         Free Space – 20-30 GB of free space should be available on both server and worker

The currently supported types of applications are

·         Simple ASP.NET applications running on a single server

·         A Windows service running on a single server

·         Complex ASP.NET applications that depend on WCF, running on a single server or multiple servers

·         Complex ASP.NET applications that depend on Windows services or processes outside of IIS, running on a single server or multiple servers

·         Complex, multi-node IIS or Windows service applications, running on a single server or multiple servers

There are also two types of applications that are not supported:

·         ASP.NET applications that use files and registries outside of IIS web application directories

·         ASP.NET applications that depend on features of a Windows operating system version prior to Windows Server Core 2016

Now that we have described App2Container as well as the .NET applications on which it will and will not work, the next step is to show how to use the tool.

Using AWS App2Container to Containerize an Application

We will first describe the application that we are going to containerize. We have installed a .NET Framework 4.7.2 application onto a Windows EC2 instance that supports containers; the AMI we used is shown in Figure 2. Please note that since EC2 regularly revises its AMIs, you may see a different Id.

Figure 2. AMI used to host the website to containerize

The application is connected to an RDS SQL Server instance for database access using Entity Framework, and the connection string is stored in the web.config file.

The next step, now that we have a running application, is to download the AWS App2Container tool. You can access the tool by going to https://aws.amazon.com/app2container/ and clicking the Download AWS App2Container button at the top of the page. This will bring you to the Install App2Container page in the documentation which has a link to download a zip file containing the App2Container installation package. Download the file and extract it to a folder on the server. If you are doing the work using the server\worker mode, then download and extract the file on both servers. After you unzip the downloaded file, you should have 5 files, one of which is another zipped file.

Open PowerShell and navigate to the folder containing App2Container. You must then run the install script.

PS C:\App2Container> .\install.ps1

You will see the script running through several checks and then present some terms and conditions text that will require you to respond with a y to continue. You will then be able to see the tool complete its installation.

The next step is to initialize and configure App2Container. If using server/worker mode, then you will need to do this on each machine. You start the initializing with the following command.

PS C:\App2Container> app2container init

It will then prompt you for a Workspace directory path for artifacts value. This is where the files from the analysis and any containerization will be stored. Click enter to accept the default value or enter a new directory. It will then ask for an Optional AWS Profile. You can click enter if you have a default profile setup or you can enter the name of the profile to use if different.

Note: It is likely that a server running the application you want to containerize does not have the appropriate profile available. If not, you can set one up by running the aws configure command to set up your CLI installation that App2Container will use to create and upload the created container.

Next, the initialization will ask you for an Optional S3 bucket for application artifacts. Providing a value in this step will result in the tool output also being copied to the provided bucket. You can click enter to use the default of “no bucket” however, at the time of this writing you must have this value configured so that it can act as storage for moving the container image into ECR. We used an S3 bucket called “prodotnetonaws-app2container”. The next initialization step is whether you wish to Report usage metrics to AWS? (Y/N). No personal or confidential information is gathered, so we recommend that you click enter to accept the default of “Y”. The following initialization prompt asks if you want to Automatically upload logs and App2Container generated artifacts on crashes and internal errors? (Y/N). We want AWS to know as soon as possible if something went wrong so we selected “y”. The last initialization prompt is asking whether to Require images to be signed using Docker Content Trust (DCT)? (Y/N). We selected the default value, “n”. The initialization will then display the path in which the artifacts will be created and stored. Figure 3 shows our installation when completed.

Figure 3. Output from running the App2Container initialization

For those of you using the server/worker mode approach, take note of the application artifact directory displayed in the last line of the command output as this will contain the artifacts that you will need to move to the worker machine. Now that the application is initialized, the next step is to take the inventory of eligible applications running on the server. You do this by issuing the following command:

PS C:\App2Container> app2container inventory

The output from this command is a JSON object collection that has one entry for each application. The output on our EC2 server is shown below:

{
     "iis-demo-site-a7b69c34": {
          "siteName": "Demo Site",
          "bindings": "http/*:8080:",
          "applicationType": "IIS"
      },
      "iis-tradeyourtools-6bc0a317": {
          "siteName": "TradeYourTools",
          "bindings": "http/*:80:",
          "applicationType": "IIS"
      }
}

As you can see, there are two applications on our server, the “Trade Your Tools” app we described earlier as well as another website “Demo Site” that is running under IIS and is bound to port 8080. The initial key is the application ID that you will need moving forward.

Note: You can only containerize one application at a time. If you wish to containerize multiple applications from the same server you will need to repeat the following steps for each one of those applications.

The next step is to analyze the specific application that you are going to containerize. You do that with the following command, replacing the application ID (APPID) in the command with your own.

PS C:\App2Container> app2container analyze --application-id APPID

You will see a lot of flashing that shows the progress output as the tool analyzes the application, and when it is complete you will get output like that shown in Figure 4.

 Figure 4. Output from running the App2Container analyze command

The primary output from this analysis is the analysis.json file that is listed in the command output. Locating and opening that file will allow you to see the information that the tool gathered about the application, much of which is a capture of the IIS configuration for the site running your application. We won’t show the contents of the file here as it is several hundred lines long, however, much of the content of this file can be edited as you see necessary.

The next steps branch depending upon whether you are using a single server or using the server/worker mode.

When containerizing on a single server

Once you are done reviewing the artifacts created from the analysis, the next step is to containerize the application. You do this with the following command

PS C:\App2Container> app2container containerize --application-id APPID

The processing in this step may take some time to run, especially if, like us, you used a free-tier low-powered machine! Once completed, you will see output like Figure 5.

Figure 5. Output from containerizing an application in App2Container

At this point, you are ready to deploy your container and can skip to the next article, “Deploying…”, if you don’t care about containerizing using server/worker mode.

When containerizing using server/worker mode

Once you are done reviewing the artifacts created from the analysis, the next step is to extract the application. This will create the archive that will need to be moved to the worker machine for containerizing. Also, the tool will upload the archive to the S3 bucket provided during initialization. Since we didn’t provide a bucket, we must manually copy the file. The command to extract the application is:

PS C:\App2Container> app2container extract --application-id APPID

This command will process, and you should get a simple “Extraction successful” message.

Returning to the artifact directory that was displayed when initializing App2Container, you will see a new zip file named with your Application ID. Copy this file to the worker server.

Once you are on the worker server and App2Container has been initialized, the next step is to containerize the content from the archive. You do that with the following command

PS C:\App2Container> app2container containerize --input-archive PathToZip

The output from this step matches the output from running the containerization on a single server and can be seen in Figure 5 above.

The next article will show how to deploy this containerized application into AWS.

Containerizing a .NET Core-based Application for AWS

In our last post in this series, we talked about Containerizing a .NET 4.x Application for deployment onto AWS, and as you may have seen it was a somewhat convoluted affair. Containerizing a .NET Core type application is much easier, because a lot of the hoops that you must leap through to manage a Windows container will not be necessary. Instead, all AWS products, as well as IDEs, will support this out the gate.

Using Visual Studio

We have already gone through adding container support using Visual Studio, and that we are doing it now using a .NET Core-based application does not change that part of the process at all. What does change, however, is the ease of getting the newly containerized application into AWS. Once the Docker file has been added, the “Publish to AWS” options when right-clicking on the project name in the Solution Explorer is greatly expanded. Since our objective is to get this application deployed to Amazon ECR, make the choice to Push Container Images to Amazon Elastic Container Registry and click the Publish button. You will see the process walk through a few steps and it will end with a message stating that the image has been successfully deployed into ECR.

Using JetBrains Rider

The process of adding a container using JetBrains Rider is very similar to the process used in Visual Studio. Open your application in Rider, right-click the project, select Add, and then Docker Support as shown in Figure 1.

Figure 1. Adding Docker Support in JetBrains Rider

This will bring up a window where you select the Target OS, in this case, Linux.  Once you have this finished you will see a Dockerfile show up in your solution. Unfortunately, the AWS Toolkit for Rider does not currently support deploying the new container image to ECR. This means that any deployment to the cloud must be done with the AWS CLI or the AWS Tools for Powershell and would be the same as the upload process used when storing a Windows container in ECR that we went over in an earlier post.

As you can see, containerizing a .NET Core based application is much easier to do as well as easier to deploy into AWS.

Containerizing a .NET Framework 4.x Application for AWS

In this post we are going to demonstrate ways in which you can containerize your applications for deployment into the cloud, the next step in minimizing resource usage and likely saving money. This article is different from the previous entries in this series because those were a discussion of containers and running them within the AWS infrastructure while this post is much more practical and based upon getting to that point from an existing non-containerized application.

Using Visual Studio

Adding container support using Visual Studio is straightforward.

Adding Docker Support

Open an old ASP.NET Framework 4.7 application or create a new one. Once open, right-click on the project name, select Add, and then Docker Support as shown in Figure 1.

Figure 1. Adding Docker Support to an application.

Your Output view, when set to showing output from Container Tools, will show multiple steps being performed, and then it should finish successfully. When completed, you will see two new files added in the Solution Explorer, Dockerfile, and a subordinate .dockerignore file. You will also see that your default Debug setting has changed to Docker. You can see both changes in Figure 2.

Figure 2. Changes in Visual Studio after adding Docker support

You can test the support by clicking the Docker button. This will build the container, run it under your local Docker Desktop, and then open your default browser. This time, rather than going to a localhost URL you will instead go to an IP address, and if you compare the IP address in the URL to your local IP you will see that they are not the same. That is because this new IP address points to the container running on your system.

Before closing the browser and stopping the debug process, you will be able to confirm that the container is running by using the Containers view in Visual Studio as shown in Figure 3.

Figure 3. Using the Containers view in Visual Studio to see the running container

You can also use Docker Desktop to view running containers. Open Docker Desktop and select Containers / Apps. This will bring you to a list of the running containers and apps, one of which will be the container that you just started as shown in Figure 4.

Figure 4. Viewing a running container in Docker Desktop

Once these steps have been completed, you are ready to save your container in ECR, just as we covered earlier in this series.

Deploying your Windows Container to ECR

However, there are some complications with this, as the AWS Toolkit for Visual Studio does not support the container deployment options we saw earlier when looking at the toolkit when working with Windows containers. Instead, we are going to use the AWS PowerShell tools to build and publish your image to ECR. At a high level, the steps are:

·         Build your application in Release mode. This is the only way that Visual Studio puts the appropriate files in the right place, namely the obj\Docker\publish subdirectory of your project directory. You can see this value called out in the last line of your Dockerfile: COPY ${source:-obj/Docker/publish} .

·         Refresh your ECR authentication token. You need this later in the process so that you can login to ECR to push the image.

·         Build the Docker image.

·         Tag the image. Creates the image tag on the repository

·         Push the image to the server. Copy the image into ECR

Let’s walk through them now. The first step is to build your application in Release mode. However, before you can do that, you will need to stop your currently running container. You can do that through either Docker Desktop or the Containers view in Visual Studio. If you do not do this, your build will fail because you will not be able to override the necessary files. Once that is completed, your Release mode build should be able to run without problem.

Next, open PowerShell and navigate to your project directory. This directory needs to be the one that contains the Docker file. First thing we will do is to set the authentication context. We do that by first getting the command to execute, and then executing that command. That is why this process has two steps.

$loginCommand = Get-ECRLoginCommand -Region <repository region>

And then

Invoke-Expression $loginCommand.Command

This refreshed the authentication token into ECR. The remaining commands are based upon an existing ECR repository. You can access this information through the AWS Explorer by clicking on the repository name. This will bring up the details page as shown in Figure 5.

Figure 5. Viewing a running container in Docker Desktop

The value shown by the 1 is the repository name and by number 2 is the repository URI. You will need both of those values for the remaining steps. Build the image:

docker build -t <repository> .

The next step is to tag the image. In this example we are setting this version as the latest version by appending both the repository name and URI with “:latest”.

docker tag <repository>:latest <URI>:latest

The last step is to push the image to the server:

docker push <URI>:latest

You will see a lot of work going on as everything is pushed to the repository but eventually it will finish processing and you will be able to see your new image in the repository.

Note: Not all container services on AWS support Windows containers. Amazon ECS on AWS Fargate is one of the services that does as long as you make the appropriate choices as you configure your tasks. There are detailed directions to doing just that at https://aws.amazon.com/blogs/containers/running-windows-containers-with-amazon-ecs-on-aws-fargate/.

While Visual Studio offers a menu-driven approach to containerizing your application, you always have the option to containerize your application manually.

Containerizing Manually

Containerizing an application manually requires several steps. You’ll need to create your Docker file and then coordinate the build of the application so that it works with the Docker file you created. We’ll start with those steps first, and we’ll do it using JetBrains Rider. The first thing you’ll need to do is to add a Docker file to your sample application, called Dockerfile. This file needs to be in the root of your active project directory. Once you have this added to the project, right-click the file to open the Properties window and change the Build action to None and the Copy to output directory to Do not copy as shown in Figure 6.

Figure 6. Build properties for the new Docker file

This is important because it makes sure that the Docker file itself will not end up deployed into the container.

Now that we have the file, let’s start adding the instructions:

FROM mcr.microsoft.com/dotnet/framework/aspnet:4.8-windowsservercore-ltsc2019
ARG source
WORKDIR /inetpub/wwwroot

These commands are defining the source image with FROM, defining an argument, and then defining the directory and entry point where the code is going to be running on the container. The source image that we have defined includes support for ASP.NET and .NET version 4.8, mcr.microsoft.com/dotnet/framework/aspnet:4.8, and is being deployed onto Windows Server 2019, windowsservercore-ltsc2019. There is an image for Windows Server 2022, windowsservercore-ltsc2022, but this may not be usable for you if you are not running the most current version of Windows on your machine

The last part that we need to do is to configure the Docker file to include the compiled application. However, before we can do that, we need to build the application in such a way that we can access these deployed bits. This is done by publishing the application. In Rider, you publish the application by right-clicking on the project and selecting the Publish option. This will give you the option to publish to either a Local folder or Server. This brings up the configuration screen where you can select the directory in which to publish as shown in Figure 7.

Figure 7. Selecting a publish directory

It will be easiest if you select a directory underneath the project directory; we recommend within the bin directory so that the IDEs will tend to ignore it. Clicking the Run button will publish the app to the directory. The last step is to add one more command to the Dockerfile where you point the source command to the directory in which you published the application.

COPY ${source:-bin/release} .

Once you add this last line into the Dockerfile, you are ready to deploy the Windows container to ECR using the steps that we went through in the last section.

Now that we have walked through two different approaches for containerizing your older .NET Framework-based Windows application, the next step is to do the same with a .NET Core-based application. As you will see, this process is a lot easier because we will build the application onto a Linux-based container so you will see a lot of additional support in the IDEs. Let’s look at that next.

Amazon RDS Oracle for .NET Developers

The last database available in RDS that we will go over is the oldest commercial SQL-based database management system, Oracle. While originally strictly relational, Oracle is now considered a multi-model database management system, which means that it can support multiple data models, such as document, graph, relational, and key-value rather than simple supporting relational data like many of the systems we have been talking about up until now. It is also the database of choice for many different packaged software systems and is generally believed to have the largest RDBMS market share (based on revenue) – which means that it would not be surprising to be a .NET developer and yet be working with Oracle. And Amazon RDS makes it easy to do that in the cloud.

Oracle and .NET

Let’s first talk about using Oracle as a .NET developer. Since Oracle is a commercial database system, which is different from the rest of the systems we have talked about in this series, it has a lot of additional tools that are designed to help .NET developers interact with Oracle products. The first of these is the Oracle Developer Tools for Visual Studio.

Oracle Developer Tools for Visual Studio

There are a lot of .NET applications based upon Oracle, which means that it is to Oracle’s advantage to make that interaction as easy as possible. One of the ways that they did this was to create the Oracle Developer Tools for Visual Studio (ODT for VS). This tool runs within Visual Studio 2017 or 2019 (2022 was not supported at the time of this writing) and brings in features designed to provide insight and improve the developer experience. Examples of the features within this tool include:

·         Database browsing – Use Server Explorer to browse your Oracle database schemas and to launch the appropriate designers and wizards to create and alter schema objects.

·         Schema comparison – View differences between two different schemas and generate a script that can modify the target schema to match the source schema. You can do this by connecting to live databases or by using scripts within an Oracle Database project.

·         Entity Framework support – Use Visual Studio’s Entity Designer for Database First and Model First object-relational mapping. (“Code First” is also supported).

·         Automatic code generation– You can use various windows, designers, and wizards to drag and drop and automatically generate .NET code.

·         PL/SQL Editor and debugger– Allows you to take advantage of Visual Studio’s debugging features from within PL/SQL code, including seamlessly stepping from .NET code into your PL/SQL code and back out again.

You need to have a free Oracle account before you can download the tools from https://www.oracle.com/database/technologies/net-downloads.html. Please note that installing these tools will also install functionality to interact with Oracle Cloud, but those details are for a different article! Once the tools are downloaded and installed you will see a new section in your Tools menu as shown in Figure 1.

Figure 1. New features added to Tools menu by ODT for VS

You will also find four new project templates added to the Create a new project wizard:

·         Visual C# Oracle CLR project – creates a C#-based project for creating classes to use in Oracle database

·         Visual Basic Oracle CLR project – creates a Visual Basic project for creating classes to use in Oracle database

·         Oracle Database project – creates a project for maintaining a set of scripts that can be generated using Server Explorer menus. This project type does NOT support schema comparison.

·         Oracle Database project Version 2 – creates a project for maintaining a standardized set of SQL scripts that represent your Oracle database schema. This project type supports schema comparison.

There are additional features to these tools, so suffice to say that Oracle provides various ways to help .NET developers interact with their Oracle databases. Lots of ways. Many more than you will find for any of the other databases we have looked at in this series. And it should not surprise you to find that they also support connecting to Oracle databases from within your .NET application.

Oracle Data Provider for .NET (ODP.NET)

Where the ODT for VS is designed to help improve a developer’s productivity when interacting with Oracle databases, ODP.NET instead manages the interconnectivity between .NET applications and Oracle databases. ODP.NET does that by providing several NuGet packages, Oracle.ManagedDataAccess.Core and Oracle.EntityFrameworkCore, that support .NET 5 and more recent versions and several NuGet packages supporting .NET versions prior to 5.0, Oracle.ManagedDataAcess and Oracle.ManagedDataAccess.EntityFramework. Once you have the packages, the next thing that you need to do is to configure your application to use Oracle. You do this by using the UseOracle method when overriding the OnConfiguring method in the context class as shown below:

protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
    optionsBuilder.UseOracle("connection string here");
}

A connection string for Oracle has three required fields:

·         User Id – username for use with the connection

·         Password – password

·         Data Source – the Transparent Network Substrate (tns) name is the name of the entry in tnsnames.ora file for the database server. This file can be found in the $ORACLE_HOME/network/admin directory.

This makes it seem like this should be an easy task to manage a connection string. However, of course, there is a caveat – you must be willing to deploy a file that has to be in a very specific place on the server and contain a reference to the server to which you need to connect. If you are okay with that approach then this is a simple connection string – “user id=prodotnetonaws;password=password123;data source=OrcleDB”. However, since a lot of the flexibility inherent in the cloud will go away if you start making this a requirement (you are no longer simply deploying just your application), then you will have to build a much uglier connection string using a Connect Descriptor:

“user id=prodotnetonaws;password=password123;data source=”(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=servernamehere)(PORT=1521))(CONNECT_DATA=(SID=databasename)))”

This means that we will need to build our connection string with additional values:

  • Host – The address of the server to which the application will connect
  • SID – The database, on the host server, to which the application is connecting

Let’s now setup our Oracle database and see where you get those values from.

Setting up an Oracle Database on Amazon RDS

Now that we know how to setup our .NET application to access an Oracle database, let’s go look at setting up an Oracle instance. First, log into the console, go to RDS, select Create database. On the Create Database screen, select Standard create and then Oracle. This will bring up the remainder of that section as shown in Figure 2

Figure 2. Options after selecting Oracle when creating a new RDS Database

As you can see, your next option is to select the Database management type, for which there are two options, the default Amazon RDS and Amazon RDS Custom. The Amazon RDS Custom management type requires you to upload your own installation files and patches to Amazon S3. Selecting that management type will change the UI as shown in Figure 3.

Figure 3. Selecting Amazon RDS Customs management type

In Amazon RDS Custom, a custom engine version (CEV) is a binary volume snapshot of a database engine and specific AMI. You first upload installation files and patches to Amazon S3 from which you create CEVs. These CEVs are used as the resources for your database. While this gives you much more control over the resources used by your database as well as managing the extra options you may have purchased as add-ons, it is out of scope for this article, so select Amazon RDS instead!

The next configuration option is a checkbox to Use multitenant architecture. This is a very interesting Oracle feature that allows for the concept of a container database (CDB) that contains one or more pluggable databases (PDB). A PDB is a set of schemas, objects, and related structures that appear logically to a client application as a separate, fully functional database. RDS for Oracle currently supports only 1 PDB for each CDB.

The next configuration option is the database Edition, with Oracle Enterprise Edition and Oracle Standard Edition Two as the only available choices currently. When selecting the Enterprise edition, you will see that you must bring your own license, however, selecting the Standard edition will allow you to bring your own license or to choose a license-included version. Standard edition is significantly less expensive, so you should consider that approach unless you need the full enterprise functionality. We chose the standard edition, license-included, most-recent version.

Once you have gone through those, all the remaining sections are ones that you have seen before as they are the same as are available on MySQL, MariaDB, and PostgreSQL (there is no serverless instance approach like was available with Amazon Aurora). However, this will not enable us to be able to automatically connect with our .NET application.

If we look back at our Oracle connection string:

“user id=prodotnetonaws;password=password123;data source=”(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=servernamehere)(PORT=1521))(CONNECT_DATA=(SID=databasename)))”

There are two values that are needed, the servername and the databasename. We know that once the server has been created that there will be a servername, or host, but there is not yet a database with which to connect. Remember, this work you are doing right now is not to create the Oracle database, it is instead around getting the Oracle server set up and available. You can create an initial database by expanding the Additional Configuration section and filling out the Initial database name field in the Database options section as shown in Figure 4.

Figure 4. Creating an initial database during setup

Add in an initial database name and complete the set-up. Once you click the Create button then the process will start. However, since Oracle is a much more complicated server than any of the others, this initial creation and setup process will be considerably longer than it was with the other databases.

Once your database is available, clicking on the DB identifier will bring up the database details. This is where you will be able to see the endpoint of the server. Using that value plus the database name that you created during the setup process will finish the process for updating your application to use Oracle as its primary database.

Amazon RDS – Aurora for .NET Developers 

Amazon Aurora is a MySQL and PostgreSQL-compatible relational database designed for the cloud. AWS claims that with some workloads Aurora can deliver up to 5x the throughput of MySQL and up to 3x the throughput of PostgreSQL without requiring any application changes. Aurora can do this because its storage subsystem was specifically designed to run on AWS’ fast distributed storage; in other words, Aurora was designed with cloud resources in mind, while those other “non-cloud only” databases are simply running on cloud resources. This design approach allows for automatic storage growth as needed, up to a cluster volume maximum size of 128 tebibytes (TiB) and offers 99.99% availability by replicating six copies of your data across three Availability Zones and backing up your data continuously to Amazon S3. It transparently recovers from physical storage failures; instance failover typically takes less than 30 seconds.

Note: A tebibyte (TiB) is a unit of measure used to describe computing capacity. The prefix tebi comes from the power-of-2 (binary) system for measuring data capacity. That system is based on powers of two. A terabyte (the unit normally seen on disk drives and RAM) is a power-of-10 multiplier, a “simpler” way of looking at the value. Thus, one terabyte = 1012 bytes, or 1,000,000,000,000 bytes as opposed to one tebibyte, which equals 240 bytes, or 1,099,511,627,776 bytes

Also, because of this customized design, Aurora can automate and standardize database replication and clustering. The last uniquely Aurora feature is the ability to use push-button migration tools to convert any already-existing RDS for MySQL and RDS for PostgreSQL applications to use RDS for Aurora instead. The argument for this ease in migration, and for Amazon Aurora in general, is that even though Aurora may be 20% more expensive than MySQL, Amazon claims that Aurora is 5x faster than MySQL, has 3x the throughput of standard PostgreSQL, and is able to scale to much larger datasets.

Creating an Amazon Aurora database in RDS

Let’s next look at creating a new Aurora database. First, log into the console, go to RDS, select Create database. On the Create Database screen, select Standard create and then Aurora.  This should bring up some Aurora-specific sections as shown in Figure 1.

Figure 1. Selecting edition and capacity type when building an Aurora database

The first selection, Edition, asks you to determine whether you wish a MySQL or PostgreSQL compatible edition.

MySQL compatible edition

The default selection when creating an Aurora database is MySQL, as shown above in Figure 1. By making this choice, values will be optimized for MySQL and default filters will be so set for the options within the Available versions dropdown. The next area, Capacity type, provides two choices: Provisioned and Serverless. Selecting a provisioned capacity type will require you to select the number and instance classes that you will need to manage your workload as well as determine your preferred Availability & durability settings as shown in Figure 2.

Figure 2. Settings for creating a provisioned database

Selecting the serverless capacity type, on the other hand, simply requires you to select a minimum and maximum value for capacity units as shown in Figure 3. A capacity unit is comparable to a specific compute and memory configuration. Based on the minimum capacity unit setting, Aurora creates scaling rules for thresholds for CPU utilization, connections, and available memory. Aurora then reduces the resources for the DB cluster when its workload is below these thresholds, all the way down to the minimum capacity unit.

Figure 3. Capacity settings when creating a serverless database

You also have the ability to configure additional aspects around scaling using the Additional scaling configuration options. The first value is Autoscaling timeout and action. Aurora looks for a scaling point before changing capacity during the autoscaling process. A scaling point is a point in time when no transactions or long-running queries are in process. By default, if Aurora can’t find a scaling point within the specified timeout period, it will stop looking and keep the current capacity. You will need to choose the Force the capacity change option to make the change even without a scaling point. Choosing this option can affect any in-process transactions and queries. The last selection is whether you want the database to Scale the capacity to 0 ACUs when cluster is idle. The name of the option pretty much tells the story; when that item is selected then your database will basically shut off when not being used. It will then scale back up as requests are generated. There will be a performance impact on that first call, however, you will also not be charged any processing fees.

The rest of the configuration sections on this page are the same as they have been for the previous RDS database engines that we posted about earlier.

PostgreSQL compatible edition

Selecting to create a PostgreSQL-compatible Aurora database will give you very similar options as you would get when selecting MySQL. You have the option to select either a Provisioned or Serverless capacity type, however, when selecting the serverless capacity type you will see that the default values are higher. While the 1 ACU setting is not available, the ability to scale to 0 capacity units when the cluster is idle is still supported.

There is one additional option that is available when creating a provisioned system, Babelfish settings. Aurora’s approach towards building compatibility with the largest OSS relational database systems has proven to be successful for those using those systems. AWS took the first step into building compatibility with commercial software by releasing Babelfish for Aurora PostgreSQL. As briefly touched on earlier, Babelfish for Aurora PostgreSQL is a new capability that enables Aurora to understand commands from applications written for Microsoft SQL Server as shown in Figure 4. 

Figure 4. Accessing Amazon Aurora through Babelfish

With Babelfish, Aurora PostgreSQL now “understands” T-SQL and supports the SQL Server communications protocol, so your .NET apps that were originally written for SQL Server will work with Aurora – hopefully with minimal code changes. Babelfish is a built-in capability of Amazon Aurora and has no additional cost, although it does require that you be using a version greater than PostgreSQL 13.4, which at the time of this writing was not available on Serverless and is why this option is unable to be selected from that mode.

Amazon Aurora and .NET

As briefly touched on earlier, the primary outcome of your making a choice between PostgreSQL and MySQL is that the choice determines how you will interact with the database. This means that using the MySQL-compatible version of Aurora requires the use of the MySql.EntityFrameworkCore NuGet packages, while connecting to the PostgreSQL-compatible edition requires the Npgsql and Npgsql.EntityFrameworkCore.PostgreSQL packages, just like they were used earlier in those sections of this series. If you are considering using Babelfish with the PostgreSQL-compatible, then you would use the standard SQL Server NuGet packages as we worked with in the last few posts.

This means that moving from MySQL on-premises to MySQL-compatible Aurora Serverless would require no code changes to systems accessing the database; the only change you would have to manage would be the connection string so that you can ensure that you are talking to the database. Same for PostgreSQL and even SQL Server. This approach for compatibility has made it much easier to move from well-known database systems to Amazon’s cloud-native database, Aurora.

Amazon RDS – PostgreSQL for .NET Developers

PostgreSQL is a free, open-source database that emphasizes extensibility and SQL compliance and was first released in 1996. A true competitor to commercial databases such as SQL Server and Oracle, PostgreSQL supports both online transaction processing (OLTP) and online analytical processing (OLAP) and has one of the most advanced performance features available, multi-version concurrency control (MVCC). MVCC supports the simultaneous processing of multiple transactions with almost no deadlock, so transaction-heavy applications and systems will most likely benefit from using PostgreSQL over SQL Server, and there are companies that use PostgreSQL to manage petabytes of data.

Another feature that makes PostgreSQL attractive is that not only does it support your traditional relational database approach, but it also fully supports a JSON/JSONB key/value storage approach that makes it a valid alternative to your more traditional NoSQL databases; so, you can now use a single product to support the two most common data access approaches. Because of its enterprise-level of features and the amount of work it takes to manage and maintain those, even though it is also open source and free software like MySQL and MariaDB, it is slightly more expensive to run PostgreSQL on Amazon RDS than those other open-source products.

PostgreSQL and .NET

As with any database products that you will access from your .NET application, its level of support for .NET is important. Fortunately for us, there is a large community involved in helping ensure that PostgreSQL is relevant to .NET users.

Let’s look at what you need to do to get .NET and PostgreSQL working together. The first thing you need to do is to include the necessary NuGet packages, Npgsql and Npgsql.EntityFrameworkCore.PostgreSQL as shown in Figure 1.

Figure 1. NuGet packages required to connect to PostgreSQL

Once you have the packages, the next thing that you need to do is to configure your application to use PostgreSQL. You do this by using the UseNpgsql method when overriding the OnConfiguring method in the context class as shown below:

protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
    optionsBuilder.UseNpgsql("connection string here");
}

A connection string for PostgreSQL has six required fields:

  • server – server with which to connect
  • port – port number on which PostgreSQL is listening
  • user id – user name
  • password – password
  • database – database with which to connect
  • pooling – whether to use connection pooling (true or false)

When working in an ASP.NET Core application the connection string is added to the appsettings.json file as shown in Figure 2.

Figure 2. Adding a connection string to an ASP.NET Core application

Let’s now go create a PostgreSQL database.

Setting up a PostgreSQL Database on Amazon RDS

Now that we know how to set up our .NET application to access PostgreSQL, let’s go look at setting up a PostgreSQL instance. First, log into the console, go to RDS, select Create database. On the Create Database screen, select Standard create and then PostgreSQL. You then have a lot of different versions that you can select from, however, the NuGet packages that we used in our earlier example require a reasonably modern version of PostgreSQL, so unless you have any specific reason to use an older version you should always use the default, most updated version.

Once you have defined the version of PostgreSQL that you will use, your next option is to select the Template that you would like to use. Note that you only have two different templates to choose from:

·         Production – defaults are set to support high availability and fast, consistent performance.

·         Dev/Test – defaults are set in the middle of the range.

Note: Both MySQL and MariaDB had a third template, Free tier, that is not available when creating a PostgreSQL database. That does not mean that you must automatically pay, however, as the AWS Free Tier for Amazon RDS offer provides free use of Single-AZ Micro DB instances running PostgreSQL. It is important to consider that the free usage tier is capped at 750 instance hours per month across all your RDS databases.

Selecting the template sets defaults across the rest of the setup screen and we will call those values out as we go through those items.

Once you select a template, your next setup area is Availability and durability. There are three options to choose from:

·         Multi-AZ DB cluster – As of the time of writing, this option is in preview. Selecting this option creates a DB cluster with a primary DB instance and two readable standby instances, with each instance in a different Availability Zone (AZ). Provides high availability, data redundancy and increases capacity to serve read workloads.

·         Multi-AZ DB instance – This option creates a primary DB instance and a standby DB instance in a different AZ. Provides high availability and data redundancy, but the standby instance doesn’t support connections for read workloads. This is the default value if you chose the Production template.

·         Single DB instance– This option creates a single DB instance with no standby instances. This is the default value if you chose the Dev/Test template.

The next section, Settings, is where you provide the DB instance identifier, or database name, and your Master username and Master password. Your database identifier value must be unique across all the database instances you have in the current region, regardless of engine option. You also have the option of having AWS auto-generate a password for you.

The next section allows you to select the DB instance class. You have the same three filters that you had before of Standard classes, Memory optimized classes, and Burstable classes. Selecting one of the filters changes the values in the instance drop-down box, You need to select Burstable classes and then one of the instances with micro in the title, such as a db.t3.micro as shown in Figure 3.

Figure 3. Selecting a free-tier compatible DB instance

The next section in the setup is the Storage section, with the same options that you had available when going through the MySQL and MariaDB setups, though the default values may be different based upon the instance class that you selected. After the storage section are the Connectivity and Database authentication sections that we walked through earlier, so we will not go through them again now – they are standard across all RDS engine options. Selecting the Create database button will take you back to the RDS Databases screen where you will get a notification that the database is being created as well as a button that you can click to access the connection details. Make sure you get the password if you selected for AWS to create your administrative password. You will only be able to access the password this one time.

The pricing for PostgreSQL is slightly higher than MariaDB or MySQL when looking at compatible configurations, about 6% higher.

Selecting between PostgreSQL and MySQL/MariaDB

There are some significant differences between PostgreSQL and MySQL\MariaDB that can become meaningful when building your .NET application. Some of the more important differences are listed below. There are quite a few management and configuration differences, but those are not mentioned since RDS manages all of those for you!

·         Multi-Version Concurrency Control – PostgreSQL was the first DBMS to rollout multi-version concurrency control (MVCC), which means reading data never blocks writing data, and vice versa. If your database is heavily used for both reading and writing than this may be a significant influencer.

·         More types supported – PostgreSQL natively supports NoSQL as well as a rich set of data types including Numeric Types, Boolean, Network Address, Bit String Types, and Arrays. It also supports JSON, hstore (a list of comma-separated key/value pairs), and XML, and users can even add new types.

·         Sequence support – PostgreSQL supports multiple tables taking their ids from the same sequence while MySQL/MariaDB do not.

·         Index flexibility – PostgreSQL can use functions and conditional indexes, which makes PostgreSQL database tuning very flexible, such as not having a problem if primary key values aren’t inserted sequentially.

·         Spatial capability – PostgreSQL has much richer support for spatial data management, quantity measurement, and geometric topology analysis.

While PostgreSQL is considered one of the most advanced databases around, that doesn’t mean that it should automatically be your choice. Many of the advantages listed above can be considered advanced functionality that you may not need. If you simply need a place to store rarely changing data, then MySQL\MariaDB may still be a better choice. Why? Because it is less expensive and performs better than PostgreSQL when performing simple reads with simple join. As always, keep your use cases in mind when selecting your database.

Note: AWS contributes to an open-source project called Babelfish for PostgreSQL, which is designed to provide the capability for PostgreSQL to understand queries from applications written for Microsoft SQL Server. Babelfish understands the SQL Server wire-protocol and T-SQL. This understanding means that you can use SQL Server drivers for .NET to talk to PostgreSQL databases. As of this writing, this functionality is not yet available in the PostgreSQL version of RDS. It is, however, available for Aurora PostgreSQL. We will go over this in more detail later in the chapter. The project can be seen at https://www.babelfishpg.org.

MariaDB, MySQL, and PostgreSQL are all open-source databases that have existed for years and that you can use anywhere, including that old server under your desk. The next database we will talk about is only available in the cloud and within RDS, Amazon Aurora.

Amazon RDS – MariaDB for .NET Developers

MariaDB is a community-developed, commercially supported fork of MySQL that is intended to remain free and open-source software under the GNU General Public License (the same license that MySQL started under). As just mentioned, it was forked because of MySQL’s acquisition by Oracle where many of the initial MySQL developers were afraid that because of how MySQL competed against the Oracle database, progress would be slowed or stopped on MySQL. MariaDB’s API and protocol are compatible with those used by MySQL, plus some features to support native non-blocking operations and progress reporting. This means that all connectors, libraries, and applications which work with MySQL should also work on MariaDB. However, for recent MySQL features, MariaDB either has no equivalent yet, such as geography, or deliberately chose not to be 100% compatible. This list of incompatibilities will likely continue to grow with each version.

MariaDB and .NET

Using .NET with MariaDB is easy to configure because of how similar the APIs are for MariaDB and for MySQL. To be honest, they are so identical that the easiest way to consume MariaDB in a .NET application is to use the same MySQL NuGet package and connection approach as we went over in the last post. The MariaDB team does not really spend any time building connectors, and instead works to ensure that the connectors that are out there, such as those built by the MySQL team, are compatible.

Setting up a MariaDB Database on Amazon RDS

Now that we know how to setup our .NET application to access MariaDB, let’s go look at setting up MariaDB in Amazon RDS. Log into the console, go to RDS, select Create database. On the Create Database screen, select Standard create and then MariaDB. You will have a list of versions, starting with version 10.2 at the time of this writing up through the most recent release.

The rest of the set-up screen, surprisingly enough, will look eerily familiar if you just went through the MySQL setup steps; mainly because they are identical! You will have the same three options for the Template that you would like to use (Production, Dev/Test, and Free tier) as well as all of the configuration sections that follow.

Since we took the Free tier route with MySQL, let’s mix it up a little bit and go with the Dev/Test version for MariaDB and we can talk about some of the areas that we glossed over when creating the MySQL database.

The first of these is after you create the database instance identifier and have provided the master user information and is entitled DB instance class. There are three options available for instances:

·         Standard classes (includes m classes) – provide a balance of compute, memory, and network resources and is the best all-around choice for many different database workloads.

·         Memory-optimized classes (includes r and x classes) – have large memory allocations to support those database workloads that process large data sets in memory.

·         Burstable classes (includes t classes) – are the only option available for the free tier and are designed to provide a baseline CPU performance with the ability to burst above this baseline as needed.

Selecting one of these options changes the instances that are available in the instance drop-down from which you make your selection. Selecting the standard classes as shown in Figure 1 will present a drop-down of the m-class instances.

Figure 1. DB Instance class selection for MariaDB (and MySQL)

Selecting one of the other options will filter the list in the drop-down to the applicable classes.

Caution: The lowest m instance class, db.m5.large, with 2vCPUs, 8 GB RAM, and 4,750 Mbps network connectivity will run you $124.83 a month in US East 2, so even a momentary creation has the chance to cost you! The t instance classes are the ones that include the free tier versions.

The next section in the setup is the storage section, with the same options that you had when going through the MySQL steps, though the default values may be different based upon the instance class that you selected. After the storage section is the second “greyed out” area that we saw when we walked through setting up MySQL, Availability & durability.

One of the best features of RDS is how it makes the installation and configuration of a new RDBMS painless when you think about what you would have to do to manage the configuration and maintenance of a standby instance on your own. For those instances where your data needs to be as available as possible, the ability to create (and forget about) a standby instance by checking a radio button can’t be overlooked. Creating a replica will configure a synchronous standby replica in a different Availability Zone than the primary DB instance. In the case of a planned or unplanned outage of the main instance, RDS will automatically failover to the standby. When using a multi-AZ deployment, however, you will be paying approximately twice as much for the duplicated instances as shown in Figure 2.

Figure 2. Estimated monthly costs with standby enabled

Once you have selected the appropriate availability option, in this case we chose to enable a standby instance, the rest of your experience will be the same as it was for MySQL, setting up Database authentication and Additional configuration. You can keep the defaults in these sections and go ahead and create your database or change the values as desired to get a better understanding of each area.

With identical pricing between MySQL and MariaDB, and similar APIs and other interactions, you may be wondering what the differences are between the two.

Selecting between MySQL and MariaDB

My recommendation when you are trying to select between MySQL and MariaDB? All other things being equal, go with MariaDB. Why? Primarily because of the advanced capability that MariaDB offers such as its optimization for performance and its ability to work with large data sets. MariaDB has also spent a lot of effort adding query optimizations for queries that use joins, sub-queries, or derived tables; so, its overall performance is better than you will find with MySQL. Lastly, MariaDB provides better monitoring through the introduction of micro-second precision and extended user statistics.

However, there are occasions when MySQL makes more sense than does MariaDB, generally, when you are using some of the features available in MySQL that are not available in MariaDB, such as geographical processing, JSON stored as binary objects rather than text, or MySQL authentication features such as the ability to authenticate to the database via roles or the ability for a user to activate multiple roles at the same time.

The key is that both are available, and both provide support for .NET development in AWS. However, you do not have to limit your choices to just MariaDB or MySQL, as there is another open-source database that is supported in Amazon RDS that is worth a review. And that’ll be the next post!