Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1693839

Summary: (RFE) Support for IPI/UPI Installation on AWS/Azure GovCloud with Custom DNS
Product: OpenShift Container Platform Reporter: Daniel Domkowski <ddomkows>
Component: InstallerAssignee: Abhinav Dahiya <adahiya>
Installer sub component: openshift-installer QA Contact: Johnny Liu <jialiu>
Status: NEW --- Docs Contact:
Severity: low    
Priority: high CC: degts, jhocutt, trking
Version: 4.1   
Target Milestone: ---   
Target Release: 4.1.z   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Deadline: 2019-10-31   

Description Daniel Domkowski 2019-03-28 18:52:17 UTC
1. Proposed title of this feature request
(RFE) Support for IPI/UPI Installation on AWS/Azure GovCloud with Custom DNS

2. Who is the customer behind the request?
US Federal Customers 

Account: Department of Veterans Affairs - 1366923
TAM customer: Yes
CSM customer: Yes
Strategic: Yes

3. What is the nature and description of the request?
To provide OpenShift 4.x installability in cloud provider regions specific to the US Federal Government. 

4. Why does the customer need this? (List the business requirements here)
Under FedRAMP, US Federal Government agencies have a mandate to move significant portions of their infrastructure and application portfolio to cloud providers using a standardized approach for security assessment, authorization, and continuous monitoring. 

AWS and Microsoft Azure both provide regions and cloud environments specific to FedRAMP and dedicated to US Government customers (i.e. AWS GovCloud and Azure Government), allowing agencies to consume a subset of cloud products and services offered in their counterpart, commercial cloud regions. For example, the AWS DNS service, Route53, is one of those services that does not have true feature parity between the commercial and GovCloud regions. As a result, the IPI/UPI installation method in OpenShift 4.0 for the AWS commercial would not work in the GovCloud region. 

In parallel efforts, both the US Intelligence Community (IC) and Department of Defense (DoD) have specific contracts with cloud providers (e.g. C2C for IC, JEDI for DoD) to offer segregated cloud environments, which offers separate products, services, and regions from FedRAMP.

In order to make use of new functionality in OpenShift 4.x, the ask from NAPS on behalf of Federal customers is to support a UPI installer in the AWS GovCloud first, and add support incrementally from there (e.g. IPI in AWS, UPI in Azure, IPI in Azure, UPI in C2C, etc.).

5. How would the customer like to achieve this? (List the functional requirements here)
Federal customers have yet to opt into DNS services like Route53 since they have yet to be offered by cloud providers under FedRAMP, and have often utilized custom DNS solutions as a result. The ask here is that we initially provide custom DNS support for the UPI Installer so that government agencies can install OpenShift 4.x in government cloud regions.

It is also important to note that DNS may not be the only blocker to government cloud support with OpenShift 4.x, and further investigation may be necessary to explore all barriers to entry. 


6. For each functional requirement listed, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented.
Within NAPS, several Red Hat employees have access to GovCloud regions through credentials provided by their customers, and can support manual testing. 

The Department of Veterans Affairs has one program that would be interested in testing Beta versions in their AWS GovCloud VPCs.

7. Is there already an existing RFE upstream or in Red Hat Bugzilla?
There is an RFE for custom DNS

8. Does the customer have any specific timeline dependencies and which release would they like to target (i.e. RHEL5, RHEL6)?
OpenShift 4.x, Late Summer, Early Fall 2019. 

9. Is the sales team involved in this request and do they have any additional input?
NAPS is involved. 

10. List any affected packages or components.
OpenShift 4.x.

11. Would the customer be able to assist in testing this functionality if implemented?
Yes

Comment 1 W. Trevor King 2019-03-28 23:18:39 UTC
> For example, the AWS DNS service, Route53, is one of those services that does not have true feature parity between the commercial and GovCloud regions.

Can you link docs for the differences?  Or is "not true feature parity" really "no GovCloud customers will be able to use Route 53, details are unimportant"? ;)

> The ask here is that we initially provide custom DNS support for the UPI Installer so that government agencies can install OpenShift 4.x in government cloud regions.

UPI is user provided, so while our CloudFormation templates show Route 53 DNS as an example, you're free to provide that infrastructure through whatever channels you like.  Are there knock on affects like "network loadbalancers require Route 53, so if you bring your own DNS you need to bring your own load balancers too" (<- hypothetical invention)?

> It is also important to note that DNS may not be the only blocker to government cloud support with OpenShift 4.x, and further investigation may be necessary to explore all barriers to entry. 

We don't publish AMIs to GovCloud (and since we aren't a government agency, we might be stuck with that).  So to get RHCOS AMIs into GovCloud clusters (and certainly into disconnected AWS Outposts clusters (do they do disconnected?) or similar, you'd need a way to get AMIs in.  We currently use copy-and-encrypt AMIs for installer-launched machines, so we could pull those in from an external region, but you'd need to work with the machine-API folks to get support for copy-and-encrypt for machines they launch.

Comment 2 Daniel Domkowski 2019-03-29 01:03:25 UTC
> For example, the AWS DNS service, Route53, is one of those services that does not have true feature parity between the commercial and GovCloud regions.

> Can you link docs for the differences?  Or is "not true feature parity" really "no GovCloud customers will be able to use Route 53, details are unimportant"? ;)

At first glance, this seemed to suggest that Route 53 WAS supported in AWS GovCloud. But after reading more closely, the steps in the link below just describe how you can use the Route 53 service in the commercial cloud to route users to resources in the AWS GovCloud. In other words, you would need credentials in both types of regions, which I believe is an unlikely use case for GovCloud agencies, and may even be impossible for the OCP installer? 

https://docs.aws.amazon.com/govcloud-us/latest/ug-west/setting-up-route53.html

Anecdotal evidence suggests that Federal customers have typically been bringing their own DNS even when they deploy OCP to commercial clouds, but I hope to gather more data on this topic to see if any are moving to Route 53.

> The ask here is that we initially provide custom DNS support for the UPI Installer so that government agencies can install OpenShift 4.x in government cloud regions.

> UPI is user provided, so while our CloudFormation templates show Route 53 DNS as an example, you're free to provide that infrastructure through whatever channels you like.  Are there knock on affects like "network loadbalancers require Route 53, so if you bring your own DNS you need to bring your own load balancers too" (<- hypothetical invention)?

I was under the impression that a custom DNS solution in the installer (even UPI) would be its own epic. I'm uncertain about the network load balancers question so others should weigh in. 


> We don't publish AMIs to GovCloud (and since we aren't a government agency, we might be stuck with that).  So to get RHCOS AMIs into GovCloud clusters (and certainly into disconnected AWS Outposts clusters (do they do disconnected?) or similar, you'd need a way to get AMIs in.  We currently use copy-and-encrypt AMIs for installer-launched machines, so we could pull those in from an external region, but you'd need to work with the machine-API folks to get support for copy-and-encrypt for machines they launch.

We have several folks in NAPS with AWS GovCloud keys and are able to upload AMIs into the GovCloud region in a community AMIs section.