Migrating from GOV.UK PaaS

Migrating from GOV.UK PaaS

Last updated

Government cloud hosting strategy

The Government Digital Service (GDS) has provided GOV.UK PaaS since 2015, supporting and providing a public cloud platform for departments using a shared hosting and responsibility model.

Following an extensive analysis period, GDS has concluded that, while the platform has been successful in its aims, the underlying technology would now require investment before it could meet its goals in the long term.

Faced with this need for re-investment GDS has decided to decommission the platform, in order to focus its budget and energy on other GDS products for common use by Government.

GDS will not be providing a replacement hosting service.

Tenants should look to their own department, or parent department, and use existing support for public cloud hosting services.

In deciding where to host their service next, tenants should consider the following principles from the Central Digital & Data Office (CDDO):

  • Tenants should continue to use public cloud services, or internal platforms based on public cloud offerings. These continue to be the platforms of choice for government services.
  • The Central Digital & Data Office (CDDO) has secured Director General level agreement to help share services. Larger departments are prepared to share knowledge, patterns and platforms with the rest of government. If tenants have no existing cloud engineering capability, you could approach the CDDO to benefit from the sharing agreements.
  • When selecting a cloud platform, the tenant should assure themselves that:
    • the public sector cloud supplier is regionally appropriate, i.e. that the data will reside in locations suitable for your workload; with GOV.UK PaaS this was predominantly the AWS eu-west-2 (London) region, but some tenancies were hosted in the AWS eu-west-1 (Ireland) region
    • billing processes are clearly defined and understood
    • any ‘service wrap’ provided by the department or supplier is appropriate and proportionate to the needs of your application
    • you have suitable arrangements in place for retention or archiving of data and its safe transit through migration
  • they understand and are prepared to manage and accept any risks of using the new platform and security assurance is carried out before the new service transition date

Introduction

The purpose of this guidance

To help GOV.UK PaaS tenants plan to move their services to an alternative hosting platform by 23 December 2023 and understand how the GOV.UK PaaS team can assist in the migration.

What does this mean for my service?

By 23 December 2023 all services hosted on GOV.UK PaaS will need to have migrated to an alternate hosting platform.

Any services that are not hosted on another platform by that date will go offline.

All platform and tenant data will be permanently deleted by 28 February 2024.

If you miss the deadline, and you need to migrate your applications, contact us via the normal support routes. If you contact us before we have deleted all tenant data, we will be able to provide this data. We will not be able to provide access to the applications or their configuration on GOV.UK PaaS.

Key dates

These are some key dates in the lifetime of the platform until decommission.

MilestoneDateWhat this means for you
Announcement of the decision to decommission12 July 2022Public notification of the decision to decommission the platform and the overall timeline to retirement. Potential tenants cannot request a 90-day trial account for evaluation. Trial tenants cannot request an upgrade to a billable account in order to deploy a beta or production service.
Festive season code freeze16 December 2022 until 9 January 2023Annual platform freeze over the festive period for non essential engineering. This should not affect you.
Deprecation of PostgreSQL 10 plans7 February 2023From this date we will be deprecating all our PostgreSQL 10 plans. You will no longer be able to create new PostgreSQL 10 databases on GOV.UK PaaS. You will need to upgrade any existing plans to ensure continuity of operation of your application. This is happening because our provider is retiring their Postgres 10 offering.
End of the financial year31 March 2023After this date you may have additional funds available from your department to help the migration.
Automatic upgrade of PostgreSQL 10 databases to PostgreSQL 1417 April - 18 July 2023Between 17 April and 18 July, AWS will forcibly upgrade any PostgreSQL 10 databases to PostgreSQL 14 during a maintenance window. GOV.UK PaaS has no plans to support PostgreSQL 14 through the marketplace using cf create-service. Any databases forcibly upgraded by AWS will still be functional, but you will not be able to manage them through Cloud Foundry, other than to delete them or connect via Conduit. We recommend upgrading your PostgreSQL 10 databases to at least PostgreSQL 11 before 17 April if you expect to be with us by that point. This change may break your application. You are advised to ensure your applications work with the versions of the relevant libraries that are compatible with PostgreSQL14, if you plan to allow AWS to automatically upgrade your database. We will be in touch on this topic regularly.
Deprecation of Redis 330 June 2023From this date we will be deprecating all our Redis 3 plans. You will have to upgrade any existing plans to ensure continuity of operation of your application. This is happening because our provider is retiring their Redis 3 offering.
Forced upgrade of PostgreSQL 10 databases to PostgreSQL 1418 July 2023From this date AWS will force the upgrade of PostgreSQL 10 databases to PostgreSQL 14, regardless of maintenance window. This change may break your application.
GOV.UK PaaS decommission23 December 2023After this date services hosted on the platform will stop operating or responding to requests from the web. The GOV.UK PaaS API will not respond to requests.
Final bills issued by GDS Business OperationsJanuary - March 2024You will receive your final bill.
Deletion of tenant data and GOV.UK PaaS platform data28 February 2024After this date all tenant and platform data will be destroyed.
Deadline to pay final bills31 March 2024You must pay any outstanding bills by this date.
End of financial year 2023/202431 March 2024The end of the financial year
Deletion of tenant data and paas platform dataApril - July 2024Your data will be finally deleted

Support for GOV.UK PaaS

GDS will continue to provide a team of site reliability engineers to perform critical platform operations and timely support

The GOV.UK PaaS platform will be maintained to the usual standards until the 23 December 2023.

PaaS maintenance schedule
MaintenanceFrequencyWhat this means for you
Monthly buildpack upgradesregular upgrades, typically monthlyRegular communication from the GOV.UK PaaS team highlighting any major changes to language support, deprecations.
Ad hoc security patchesas required within 5 days of reportingThe platform will be maintained to our usual standards for security patching until it is retired.
Annual platform pentestJanuary to March of each yearThe platform will be assured to the usual standards until it is demised. This means annual health checks and remediation of any priority findings.
monthly paas-announce email to tenantsregular monthly updatesRegular communication from GOV.UK PaaS team highlighting any key milestones or guidance for tenants.
PaaS IT Health Check for 2023/2024January to March of each yearAnnual health check for the coming year to ensure we are independently assessed and we remediate any findings

Unless you no longer require your service, it must be migrated before this date.

You will receive regular communications from the GOV.UK PaaS team via email.

We offer help and advice though our usual channels:

Changing responsibility model

GOV.UK PaaS is designed and built with a shared responsibility model. After migration, tenants will assume the responsibilities that the GOV.UK PaaS team once owned for their own infrastructure.

Examples include:

  • Security information and event management (SIEM) threat detection compliance and security incident management
  • Information assurance and IT Health checks
  • Infrastructure scaling
  • Infrastructure cost management
  • Infrastructure-level log management and retention
  • User management
  • Ingress traffic routing
  • Egress traffic routing
  • TLS certificates management
  • Secrets management
  • Platform/Operating system stack upgrades -> Upgrade deployment strategy
  • Platform monitoring and alerting
  • Tenant service monitoring and alerting
  • Back-up management
  • Disaster recovery
  • Securing SSH access to applications

Prepare to migrate

Before tenants migrate their services, there are a number of things to consider.

Audit your services

Understand all the services that you need to migrate. You could consider:

  • The relative importance of your services: which ones are higher priority? Should any be retired instead of migrated?
  • The technical effort required to migrate each service to another platform
  • Who owns the service? Smaller services may not have an obvious owner due to changes in the team or department structure
  • The “7 Rs” are commonly used cloud migration strategies

Think about your timing and budget

  • Do you have the budget to fund the migration?
  • Can you raise a business case for the year 2023/2024?
  • What is the timeframe for your migration?
  • Are there any events or features that may constraint your migration plan? For example if your service has a - higher number of users in April, you may decide to migrate a month after when it will have less of an impact
  • Can you identify a sequence to migrate the services if you have more than one?
  • You’ll need to explain to stakeholders that any data migration will incur some downtime. You can avoid this if absolutely necessary (see the Data migration section below)

Review your skills

  • Do you have the skills to migrate your services available in your department? You will need:
    • people with the skills to work with the infrastructure (defined in the DDAT framework as an infrastructure devops engineer role and also referred to as Site Reliability Engineering in the wider technology marketplace), to enable the migration, change your pipelines
    • people with the skills to refactor your applications to accommodate the new hosting arrangement (i.e. software developers)
  • Do you need to outsource to a delivery partner?
  • Do you need to bring in contractors to supplement your team?
  • Do you need to procure cloud migration expertise from the Digital Marketplace to assist with the migration?

Consider your department’s cloud hosting strategy

  • Understand your department's hosting standards and preferred cloud provider
  • Determine if there is a departmental strategy to follow
  • Determine if there is a departmental hosting platform available in-house. Some departments may have multiple hosting platforms for different types of workload. (for example one platform for general purpose web - hosting for OFFICIAL level services and one for data heavy workloads for data scientists)

Assess your security needs

You will need to review the threat model of your service and re-assess your risks so that they can be accepted by your senior responsible owner (SRO)

Decide where to host your services

There are many possible hosting options, but not all of them will necessarily make sense for your service. You should plan to spend time and effort exploring options and evaluating technical solutions according to your specific needs.

You should prefer known technologies with a proven track record like your public cloud provider’s virtual machines or basic container hosting offerings.

You should consider services with higher levels of abstraction than Public Cloud IaaS, such as:

  • AWS Copilot (to provision ECS)
  • AWS ECS on Fargate
  • AWS Lambda
  • Google App Engine
  • Google Cloud Run
  • Heroku
  • Microsoft Azure App Service

These are just some examples rather than endorsements so ensure that whatever you choose meets the needs of your service.

When making your technology choices, consider not only what is suitable for your team and department, but also what is suitable for each service in turn; not all services will be best placed running on the same technology.

For example, a static website for documentation could easily be hosted on GitHub Pages, or in an AWS S3 bucket with AWS CloudFront in front; but a complex set of microservices with multiple persistent stores would be better on ECS on Fargate, or even Kubernetes if your team/department has the necessary skills to run it.

Review the existing guidance

Here is some of the existing guidance relevant to cloud hosting.

GuidanceDescription
Government Cloud First PolicyThe government introduced a ‘Cloud First’ policy in 2013 for all technology decisions.
Technical code of practice

Point 5: Use cloud first		The Technology Code of Practice is a set of criteria to help public sector organisations design, build and buy technology.
Service Standard

9. Create a secure service which protects users’ privacy

11. Choose the right tools and technology

14. Operate a reliable service		The Service Standard helps teams to create and run great public services.
Service Manual

Technology		Helping teams to create and run great public services that meet the Service Standard.
CDDO Creating and implementing a cloud hosting strategyThis guidance outlines how to create and implement a cloud strategy, and when to consider a single, hybrid or multi-cloud solution.
NCSC cloud security guidance Cloud Security Principles

Approach

  • Keep it simple
  • Find the largest building blocks you can and consider solutions that take care of the underlying infrastructure, allowing you to focus on the specifics of your service
  • Use mature, well documented technologies and that you can readily get skills for in the marketplace
  • Consider running some technical spikes to evaluate alternatives and to inform your decision

Publishing static pages to GitHub Pages

The GDS technical writers provide a commonly used tool to generate static documentation from templates called the Tech Doc Templates.

Many teams use this to publish technical documentation for their products or team manuals to GOV.UK PaaS using the static buildpack.

For example:

The decommission of GOV.UK PaaS requires that these websites are migrated.

Teams need to decide on an appropriate alternative. This guidance documents the approach taken by the PaaS team in the hope that it will prove useful to other teams.

A common pattern uses GitHub Actions to publish the content to GitHub Pages.

The GOV.UK PaaS team used this approach to move our technical documentation from our own Cloud Foundry based platform to GitHub Pages.

The content is generated by the Middleman static site publishing tool from a set of templates defined in Markdown and Ruby ERBs however the approach can be adapted for use with other static publishing tools like Jekyll or Hugo by modifying the pipeline to replace the step that generates the HTML content.

Using the PaaS tech docs as a concrete example:

The GitHub deploy.yml workflow performs the following steps when changes are made to the main branch.

  • check out the repository source code using actions/checkout to work with
  • set up Nodejs tools using actions/setup-node
  • set up Ruby tools using actions/setup-ruby
  • set up Python tools using actions/setup-python
  • install Python and Nodejs dependencies
  • run tests in Nodejs
  • run tests in Python
  • run Middleman using bundle exec middleman build to generate static content in the /build directory
  • create an artifact from the generated /build directory using the actions/upload-github-pages-artifact storing the content in a gzip archive called github-pages
  • publish to GitHub pages using the actions/deploy-pages
  • notify the PaaS team via Slack if there are any problems

GitHub pages takes care of the set up of TLS for you and manages the underlying infrastructure to host the pages.

The content is deployed to alphagov.github.io/paas-tech-docs

You can configure a custom domain with a preferred name. We use a custom domain set to docs.cloud.service.gov.uk

Ongoing support for your service

Once your service is migrated to a new hosting platform you will need to look after it on an ongoing basis for a long time, potentially many years

  • How is your team going to support your service over the longer term? Will there be a long lived team?
  • Do you need and can you afford to offer out of hours support, including cyber security?
  • Consider how you are going to support debugging of your applications (GOV.UK PaaS permits direct SSH access to containers but other hosting providers may not)
  • Consider how you are going to secure an engineer’s access to backing services to allow you to support your service
  • Consider how you are going to patch container vulnerabilities in a timely manner

Data migration

Data migration may involve downtime. If you need zero downtime it will not be easy or cheap

Use documented PaaS data techniques

Our technical documentation covers a number of topics relevant for migrating services.

These documented processes are self-service and can be performed by tenants independent of the GOV.UK PaaS team.

The documentation is mature and tested and allows you to coordinate any system downtime yourselves.

Connecting to GOV.UK PaaS backing services

Some GOV.UK PaaS backing services (PostgreSQL, MySQL, InfluxDB, OpenSearch and Redis) only accept connections from GOV.UK PaaS apps. You’ll need to connect through a GOV.UK PaaS app to access one of these backing services from your local machine.

Conduit lets you connect to your remote backing service instances from your local system. This allows you to use the standard tool for your backing service, for example, psql for PostgreSQL or mysql for MySQL CLI tools to make backups and interrogate your backing services.

InfluxDB times series data

Time series data is often scrapped during migration.

MySQL relational data

OpenSearch non-relational data

OpenSearch is not intended as a persistent data store, and we don’t recommend moving the data you store in it. You should instead rebuild your indices at the destination.

PostgreSQL relational data

Redis cache data

Redis is not intended as a persistent data store, and we don’t recommend moving the data you store in it. You should instead rebuild your indices at the destination. If you have been using Redis for persistent data, you will need to move it.

S3 file storage

Consider your data migration strategy options

  • Can you reduce the volume of data to migrate by archiving legacy data?
  • Can you reduce the volume data to migrate during the production migration by only synchronising changes?
  • Can you migrate your data ahead of your applications?

Do you have more complex data migration needs?

Some tenants may face data migration challenges using the documented approaches above.

Things that may affect the decision include:

  • the complexity of your data schema
  • the need for high availability
  • the difficulty of exporting a database dump owing to a variety of factors like size of data, connection speed, or the reliability of the connection

We recommend you make a small number of attempts to export a dump of your databases using conduit in order to assess how reliably you can do it.

If you think that your service will face issues migrating data using the documented approaches you should get in contact with the team using our support form to discuss your needs further.

PaaS CDN Migration

We implement Cloud Foundry CDN routes as AWS Cloudfront distributions behind the scenes.

If you are migrating from a Cloud Foundry CDN route, you will need to remove the domain from your cdn-route service before you can add it to your new AWS Cloudfront distribution. If you don't do this, you will receive a 'CNAMEAlreadyExists' error. AWS documents this issue.

You can either remove the cdn-route service:

cf delete-service my-cdn-route

or update the service to remove the domain you are migrating (if you have multiple domains):

cf update-service my-cdn-route -c '{"domain": "<remaining domains>"}'

Removing the domain and re-adding it to your new AWS Cloudfront Distribution will cause your website to go down. Unfortunately changes to AWS Cloudfront can take between 5 and 20 minutes to apply.

AWS support can transfer your domain instantly between AWS Cloudfront Distributions, however you will still get a small amount downtime as you cutover the DNS yourself. This requires some coordination and prerequisite work with AWS support and will require you to discover your current Distribution ID.

To avoid downtime you can migrate your domain to something other than AWS Cloudfront for a short time or use the wildcard approach provided by AWS.

Changes to your service when migrating

You will have to make some changes to the source code of your service, its supporting stack and pipelines to allow it to migrate to an alternative hosting provider.

Apps hosted on the GOV.UK PaaS need to follow 12 factor principles which is an established cloud best-practice that will help moving containers to another platform.

If you are using GOV.UK PaaS buildpacks you will have to find an alternative to create your containers. Cloud Native buildpacks or Packer are some examples but other solutions are available.

You will have to provision a container registry for your container images. Our tenants currently use ECR, ACR, GCR, Dockerhub, GHCR, or Quay.io, but other solutions are available.

You will need to update your pipelines to build your containers and target your chosen hosting platform. Our tenants use all of the commonly available cicd tools including Jenkins, Concourse, GitHub actions, CircleCI.

You may want to consider modifying your pipelines to deploy to the new hosting platform in parallel to the existing GOV.UK PaaS platform, so that you can develop and maintain their existing system whilst exploring their new hosting platform.

GOV.UK PaaS apps get credentials for their databases from the VCAP_SERVICES environment variable, you will need to refactor your application to solve this problem without using the Cloud Foundry conventions or you will have to implement the environment variable mapping in your platform outside of the application code.

You will need to find other means to bring your own domains to use with your applications and generate TLS certificates. GOV.UK PaaS makes use of AWS CloudFront and ACM.

PaaS uses log drains to ship logs to a permanent log service for analysis and storage. You will have to find an alternative way to do this. Using a self managed ELK stack, Logit.io and Papertrail are some examples used by our tenants but other solutions are available.

You will need to reconfigure your uptime monitoring to monitor your application. Tenants are currently using tools such as pingdom and cronitor.

You will need to find other means to implement your preferred observability tools to collect metrics and raise alerts to support staff. Tenants are using tools such as prometheus, alertmanager, grafana, sentry pagerduty.

Finalise your GOV.UK PaaS account

When migrating from GOV.UK PaaS to alternate hosting the last step is to finalise your usage of the platform and reduce your spend on the account to zero before paying the final bill.

Once your service has been migrated to another cloud hosting provider you should complete the following tasks:

  1. stop all running applications hosted on the platform using cf stop <APP_NAME>
  2. delete all stopped applications using cf delete <APP_NAME>
  3. remove all objects from your S3 buckets, these must be empty before deletion. See how to connect to an S3 bucket from outside the paas for instructions on how to get credentials for use with the AWS CLI and Delete an S3 bucket
  4. delete all backing services and user provided services using cf delete-service. For example to delete a postgresql backing service see Delete a PostgreSql service
  5. notify the GOV.UK PaaS team using the support form to tell us that you have removed all your apps and services from the organisation/space and that you want to receive a final bill. Please specify the name of the Cloud Foundry organisation that you wish to decommission, the name of the service and the date the service is decommissioned (You can also notify us of any spaces you have decommissioned. We'll check on our side if there's any leftover services.)

This will prevent your account accruing any more charges.

The GOV.UK PaaS team will then:

  1. remove any user access to the Cloud Foundry organisation
  2. set the Cloud Foundry organisation to ‘suspended’ meaning that it is no longer usable
  3. generate the final bill
  4. confirm that the organisation is now no longer usable and the data and backups have been destroyed

If you notify us that one of your services is decommissioned but the PaaS team discovers that you still have running apps and backing services, we will:

  1. send you a reminder to delete the apps and backing services
  2. wait a week and if you still have running apps we will stop the apps and send another reminder to delete the apps and services
  3. wait a week and unless you have notified us otherwise we will delete the apps and the backing services

How to contact us

If you have any questions about your migration please contact us via the GOV.UK PaaS Team (support form).

Feedback

Please let us know if this guidance is useful or not useful or raise a GitHub issue to give us feedback