Migrating from Azure to AWS

About Ferret

Ferret is a company providing a Relationship Intelligence solution. It collects data from tens of thousands of sources, uses AI and machine learning to classify this data, and allows its customers to have access to exclusive databases, such as historical and real-time negative news, politically exposed people, papers and leaks, illegal activity, social media sentiment analysis, and more.

The Business Challenge

Ferret started developing and deploying their Relationship Intelligence solution on Microsoft Azure however they faced major issues being the most critical issues were related to training their ML Model’s as Azure infrastructure could not scale to provide the capacity they required. They also faced a lot of challenges to import the large datasets from on-premise to Azure and could not successfully import the Production DB using their tool of choice (Pentaho Kettle) or Azure Data Factory as the data, application, and its’ DevOps team worked in silos and were not able to solve data migration issues end-to-end.

The suggested solution to split their workloads between different Azure data centers produced a significant delay and issues with data integrity. Moreover, providing a big impact on their confidence to deliver and launch their solution on time; compromising their business operations.

Moreover, the fees of 80 K per year for only the test environment have already superseded their expectations.

Ferret understand the benefits of being in the cloud and, most of them align with their company’s needs. Therefore, after not obtaining the desired cloud outcomes at Azure, the company decided to invest in migrating its workload to AWS.

Ferret required a partner with extensive knowledge and expertise with cloud and AWS solutions to overcome the issues with their complex infrastructure to assist them deliver a stable and reliable application by their deadline on a more cost-effective basis.

The Assessment Phase

DNX Solutions was engaged to create and execute a roadmap for Ferret’s migration from Azure to AWS ensuring the company obtained the AWS Migration values of cost-saving, staff productivity, operational resilience along fast time to market.

During the discovery phase, utilising AWS Migration Readiness Assessment ( MRA) and workshops,  DNX determined what was Ferret’s business and technical capabilities ( the interdependencies, storage constraints, release process and level of security) .

With all necessary information and, based on AWS “The 7R’s” best practices ( retain, retire, relocate, rehost, repurchase, replatform and refactor)  DNX developed and proposed the following migration plan:

The Mobilise Phase

Migration Strategy

Due to Ferret’s implementation time constraint and focus on fast delivery DNX designed an AWS platform that relies on the following migration strategies:

• Re-hosting (lift-and-shift)
• Re-platforming

The main focus was to do a re-hosting of all the relevant services from Azure to AWS but to do a re-platforming of specific services based on AWS managed services which can provide better control over the solution while reducing maintenance overhead and delivery time.

With this statement in mind,  DNX Solutions followed the following design principles:

• Kubernetes only for Stateless apps.
• AWS managed services over self-maintained services.
• Bamboo for mobile app and Bitbucket Pipelines + Argo CD for cloud apps
• Az Blob Storage to AWS S3
• Az Cosmos DB to Atlas MongoDB
• k8s Kafka/Zookeeper to AWS MSK (Managed Service for Kafka)
• k8s Elasticsearch/Kibana to AWS Elasticsearch
• k8s neo4j to neo4j on AWS EC2
  Az AKS to AWS EKS

The Solution

Cloud Foundation

The project started with the implementation of our Cloud Foundation which combines years of cloud experience in a platform built with simplicity in-mind, infrastructure-as-code, open-source technologies, and is designed for AWS with well-architected principles. 

Application Modernisation

The second phase involved migrating applications running on Azure Kubernetes Service to the AWS Managed Kubernetes Service (aka EKS). At Ferret, they used to have StatefulSets running and hosting their data solutions, such as Elasticsearch, Kafka, etc. These services were replatformed to AWS Managed services so that we could offload the work of maintaining these pieces of infrastructure.

Next, we migrated the Kubernetes stateless apps to EKS. One of the first choices was to use Spot Instances, which can be up to 70% cheaper than regular instances. We also used our open source projects to create the EKS Cluster and added custom Kubernetes controllers to deploy ArgoCD, manage external secrets, configure AWS Load Balancer, control DNS, push to CloudWatch, etc.

This step allowed us to not only migrate the applications, but also set up a GitOps workflow using ArgoCD, making things more efficient and empowering developers to have more control over their kubernetes deployments. Additionally, we configured GPU instances on kubernetes to run a very specific AI workload.

Furthermore, since one of the goals was to reduce operational costs, their hosted pipeline was migrated to a managed pipeline solution.

Just to mention a few other services involved in this phase:

• Amazon Elasticsearch Service
• Amazon Managed Streaming for Apache Kafka (Amazon MSK)
• System Manager (SSM Parameter Store)
• Amazon S3

DNX provided transfer knowledge throughout different sessions to enable the customer to understand AWS concepts to properly maintain the application safe, reliable, and at a predictable cost.

DNX also created cloud watch performance dashboards and set up alarms to inform Ferret of any potential problems and to keep the application safe.

Data Platform

Ferret was not able successfully migrate their massive on-premise datasets to Azure. In their initial attempt to migrate using their on-premise ETL tool (Pentaho Kettle), they faced multiple security and connectivity issues and the tool has been discarded. A second attempt was made using Azure Data Factory, however, the performance achieved seemed not feasible as the estimated time would take a whole week, and it failed multiple times. They ended up only uploading a small amount of data to enable development and testing, and raise the data migration as an issue and they have never been able to solve it.

DNX proposed the below architecture for the  Data Platform. In this phase, we worked to Ferret’s instruction which was to configure their Mongodb cluster and permit network connectivity to their external tool, enabling them to feed their data.

They were using CosmoDB and a hosted Elasticsearch cluster at Azure. They were using an on-premise Pentaho Kettle to migrate the data. 

We proposed S3 for staging, a managed Elasticsearch, and Atlas Mongo to replace CosmoDB. The initial recommendation was to use Glue for ETL. However, after further discussion with the customer, we decided to maintain the Pentaho ETL as Ferret’s developers were familiar with this tool and helped them to connect into our secure network to import the data and move it to the required endpoints. 

Conclusion

Everything described in this case study was planned, designed, and implemented with key principles in mind, such as high availability, scalability, disaster recovery, security, elasticity, fault tolerance, and cost optimisation.

These new environments and AWS services provided Ferret the following benefits:

• Operational Excellence
  By moving to AWS, DNX worked on the well-architect platform, enabling the customer to have an automated deployment process, self-healing applications, and more efficient application management.
• Creating more cost-effective IT environments
  Creating more cost-effective IT environments by optimizing compute, storage, and database costs or moving away from running their IT infrastructures with Azure,  Ferret will spend 50%  less on AWS fees on Kubernetes cluster with CPU and GPU using spot instances running a comparable infrastructure.
• Improved time-to-market 
  We were able to reduce data migration time from 24h to 5h by using parallelism and optimising packet sizes, whilst reducing cost and complexity of the overall solution.
  Shifting IT staff focus to differentiated work and strategic business initiatives, including substantial gains on average 62% more efficient and application developers will be 25% more productive with AWS.
• Instilling IT and business operations
  with the agility required to deliver cost-effective IT resources on an on-demand basis to address business opportunities as they arise, with interviewed organizations delivering almost three times more new application features with AWS, helping them win more business and increase revenue