Incomplete Features
When this image was assembled, these features were not yet completed. Therefore, only the Jira Cards included here are part of this release
Provide a simple way to get a VM-friendly networking setup, without having to configure the underlying physical network.
Goal
Provide a network solution working out of the box, meeting expectations of a typical VM workload.
User Stories
- As an owner of a VM that is connected only to a secondary overlay network, I want to fetch resources from outside networks (internet).
- As a developer migrating my VMs to OCP, I do not want to change my application to support multiple NICs.
- My application needs access to a flat network connecting it to other VMs and Pods.
- I want to expose my selected applications over the network to users outside the cluster.
- I'm limited by public cloud networking restrictions and I rely on their LoadBalancer to route traffic to my applications.
- As a developer who defined a custom primary network in their project,
I want to connect my VM to this new primary network, so it can utilize it for east/west/north/south, while still being able to connect to KAPI.
Non-Requirements
- Service mesh integration is not a part of this
- Seamless live-migration is not a must
- UI integration is tracked in CNV-46603
Notes
- This epics tracks graduation of features previously developed upstream on OVN Kubernetes, and additional VM-specific work.
- Once we have both ingress and egress, we may be able to obsolete the custom solution on OVN-K introduced for HyperShift.
- From binding plugins epic:
- https://docs.google.com/document/d/1GZOxEh21TQilgnHULlVq0_6g0K8XKzZIItlZ_Yjv-x0/edit#heading=h.tv7nbn6fqsz2
- https://docs.google.com/document/d/1wt3z9EH5LKk02IQK7xlIxBbu-DSnw2jSgbnyYE96VDA/edit#heading=h.e9t0h9kcw91
- https://github.com/openshift/enhancements/pull/1623
- This could come in a form of a network binding. Passt should be used to allow integration with service meshes, while leveraging the L2 overlay for same IP inside and outside the guest
- There are two options how to implement this:
- Plug each individual Pod NIC as a VNIC to the guest
- Let Pod routing do its job and use Passt binding to connect the guest to it through a single VNIC
- Most likely this won't work with any existing core binding of KubeVirt - we will need to introduce our own binding plugin
- Leveraging passt would make it easier to create our plugin, since it already handles user-space networking for us and DHCP for the guest
- We could host the new binding plugin under the same repo as IPAM claims controller. Basic passt plugin template is available here: https://github.com/kubevirt/kubevirt/tree/main/cmd/cniplugins/passt-binding
- There was a presentation about binding plugins on KubeVirt Summit 2024
Ensure the feature can be used on non dev|tech preview clusters.
Involves a PR to the OpenShift API - need to involve the API team.
This task requires periodic e2e tests in openshift/origin and openshift/ovn-kubernetes asserting the correct behavior of the gated feature. Currently focused on that (must add a virt-aware TP lane).
This is the script that decied if FG can be GA or not
https://github.com/openshift/api/blob/master/tools/codegen/cmd/featuregate-test-analyzer.go
Interesing slack thread https://redhat-internal.slack.com/archives/CE4L0F143/p1729612519105239?thread_ts=1729607662.848259&cid=CE4L0F143
Goal
Primary used-defined networks can be managed from the UI and the user flow is seamless.
User Stories
- As a cluster admin,
I want to use the UI to define a ClusterUserDefinedNetwork, assigned with a namespace selector. - As a project admin,
I want to use the UI to define a UserDefinedNetwork in my namespace. - As a project admin,
I want to be queried to create a UserDefinedNetwork before I create any Pods/VMs in my new project. - As a project admin running VMs in a namespace with UDN defined,
I expect the "pod network" to be called "user-defined primary network",
and I expect that when using it, the proper network binding is used. - As a project admin,
I want to use the UI to request a specific IP for my VM connected to UDN.
UX doc
Non-Requirements
- <List of things not included in this epic, to alleviate any doubt raised during the grooming process.>
Notes
- The user-defined networks design, including the API, is available here: https://github.com/openshift/enhancements/blob/master/enhancements/network/user-defined-network-segmentation.md
- https://github.com/openshift/networking-console-plugin/pull/150
- https://github.com/openshift/networking-console-plugin/pull/142
- https://github.com/openshift/networking-console-plugin/pull/146
- https://github.com/openshift/networking-console-plugin/pull/145
- https://github.com/openshift/networking-console-plugin/pull/137
Placeholder feature for ccx-ocp-core maintenance tasks.
This is epic tracks "business as usual" requirements / enhancements / bug fixing of Insights Operator.
It looks like the insights-operator doesn't work with IPv6, there are log errors like this:
E1209 12:20:27.648684 37952 run.go:72] "command failed" err="failed to run groups: failed to listen on secure address: listen tcp: address fd01:0:0:5::6:8000: too many colons in address"
It's showing up in metal techpreview jobs.
The URL isn't being constructed correctly, use NetJoinHostPort over Sprintf. Some more details here https://github.com/stbenjam/no-sprintf-host-port. There's a non-default linter in golangci-lint for this.
Component Readiness has found a potential regression in the following test:
[sig-architecture] platform pods in ns/openshift-insights should not exit an excessive amount of times
Test has a 56.36% pass rate, but 95.00% is required.
Sample (being evaluated) Release: 4.18
Start Time: 2024-12-02T00:00:00Z
End Time: 2024-12-09T16:00:00Z
Success Rate: 56.36%
Successes: 31
Failures: 24
Flakes: 0
View the test details report for additional context.
Background
The admin console's alert rule details page is provided by https://github.com/openshift/monitoring-plugin, but the dev console's equivalent page is still provided by code in the console codebase.
Outcomes
That dev console page is loaded from monitoring-plugin and the code for the page is removed from the console codebase.
Ensure removal of deprecated patternfly components from kebab-dropdown.tsx and alerting.tsx once this story and OU-257 are completed.
Background
Technical debt led to a majority of the alerting components and all routing components being placed in a single file. This made the file very large, difficult to work with, and confusing for others to work with the routing items.
Outcomes
Alerting components have all been moved into their own separate files and the routing for the monitoring plugin is moved out of alerting.tsx into a routing specific file.
Background
In order for customers or internal teams to troubleshoot better, they need to be able to see the dashboards created using Perses inside the OpenShift console. We will use the monitoring plugin which already supports console dashboards comming from Grafana, to provide the Perses dashboard funcionallity
Outcomes
Create a component in the monitoring plugin that can render a Perses dashboard based on the dashboard schema returned by the Perses API.
There are 2 dropdowns, one for selecting namespaces and another for selecting the dashboards in the selected namespace.
Steps
- Use the same dashboards route for the Perses dashboards component
- Use the monitoring-console-plugin proxy to call the Perses API, this was covered by https://issues.redhat.com/browse/OU-432
- Choose the component being rendered: Dashboards or Perses Dashboards if Perses is detected in the cluster
- Perses dashboards
- Add the namespace dropdown, visible only if Perses is detected on the cluster using an API call to the proxy
- Create a Perses: hello world component to be rendered in the dashboard page
- When selecting a Perses project on the dropdown, render the Perses hello world component.
- When selecting the legacy namespace, render the current Dashboard component
- The implementation of the Perses component will be done in https://issues.redhat.com/browse/OU-618
- Dashboards. Keep the page as it is
- Perses dashboards
Previous work
Create dashboard flow chart
Background
In order to allow customers and internal teams to see dashboards created using Perses, we must add them as new elements on the current dashboard list
Outcomes
- When navigating to the Monitoring / Dashboards. Perses dashboards are listed with the current console dashboards. The extension point is backported to 4.14
Steps
- COO (monitoring-console-plugin)
- Add the Perses dashboards feature called "perses-dashboards" in the monitoring plugin.
- Create a function to fetch dashboards from the Perses API
- CMO (monitoring-plugin)
- An extension point is added to inject the function to fetch dashboards from Perses API and merge the results with the current console dashboards
tldr: three basic claims, the rest is explanation and one example
- We cannot improve long term maintainability solely by fixing bugs.
- Teams should be asked to produce designs for improving maintainability/debugability.
- Specific maintenance items (or investigation of maintenance items), should be placed into planning as peer to PM requests and explicitly prioritized against them.
While bugs are an important metric, fixing bugs is different than investing in maintainability and debugability. Investing in fixing bugs will help alleviate immediate problems, but doesn't improve the ability to address future problems. You (may) get a code base with fewer bugs, but when you add a new feature, it will still be hard to debug problems and interactions. This pushes a code base towards stagnation where it gets harder and harder to add features.
One alternative is to ask teams to produce ideas for how they would improve future maintainability and debugability instead of focusing on immediate bugs. This would produce designs that make problem determination, bug resolution, and future feature additions faster over time.
I have a concrete example of one such outcome of focusing on bugs vs quality. We have resolved many bugs about communication failures with ingress by finding problems with point-to-point network communication. We have fixed the individual bugs, but have not improved the code for future debugging. In so doing, we chase many hard to diagnose problem across the stack. The alternative is to create a point-to-point network connectivity capability. this would immediately improve bug resolution and stability (detection) for kuryr, ovs, legacy sdn, network-edge, kube-apiserver, openshift-apiserver, authentication, and console. Bug fixing does not produce the same impact.
We need more investment in our future selves. Saying, "teams should reserve this" doesn't seem to be universally effective. Perhaps an approach that directly asks for designs and impacts and then follows up by placing the items directly in planning and prioritizing against PM feature requests would give teams the confidence to invest in these areas and give broad exposure to systemic problems.
Relevant links:
- Documentation:
- Edge Diagnostics Scratchpad, our team's internal diagnostic guide.
- Troubleshooting OCP networking issues - The complete guide, the SDN team's diagnostic guide.
- Linux Performance, Brendan Gregg's guide to analyzing Linux performance issues.
- RFC: A proper feedback loop on Alerts.
- OpenShift Router Reload Technical Overview on Access.
- Performance Scaling HAProxy with OpenShift on Access.
- How to collect worker metrics to troubleshoot CPU load, memory pressure and interrupt issues and networking on worker nodes in OCP 4 on Access.
- OpenShift Performance and Scale Knowledge Base on Mojo, results from OpenShift scalability testing.
- Scalability and performance, OCP 4.5 documentation about the router's currently known scalability limits.
- Scaling OpenShift Container Platform HAProxy Router, OCP 3.11 documentation about the manual performance configuration that was possible in OCP 3.
- Timing web requests with cURL and Chrome from the Cloudflare blog.
- tcpdump advanced filters, some useful tcpdump commands.
- OpenShift SDN - Networking, OCP 3.11 documentation on the SDN (useful background reading).
- Ingress Operator and Controller Status Conditions, design document for improved status condition reporting.
- Observability tips for HAProxy, a slide deck by Willy Tarreau.
- Interesting Traces - Out of Order versus Retransmissions, analysis using tshark.
- The PCP Book: A Complete Documentation of Performance Co-Pilot, by Yogesh Babar.
- Debugging kernel networking bug, brief guide to using SystemTap on RHCOS.
- Troubleshooting throughput issues from the OCP 4.5 documentation.
- Troubleshooting OpenShift Clusters and Workloads.
- Red Hat Enterprise Linux Network Performance Tuning Guide (PDF).
- openshift/enhancements#289 stability: point to point network check, a diagnostic built into the kube-apiserver operator.
- Diagnostic tools:
- dropwatch to watch for packet drops.
- ethtool to check NIC configuration.
- iovisor/bcc: BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more to trace and diagnose various issues in the networking stack.
- r-curler to gather timing information about HTTP/HTTPS connections.
- route-monitor, to monitor routes for reachability.
- hping(3), a programmable packet generator.
- OpenTracing / Jaeger in OpenShift.
- node-problem-detector, a possible integration point for new diagnostics.
- Using SystemTap by Brendan Gregg.
- DTrace SystemTap cheatsheet (PDF).
- Visualization and more sophisticated diagnostic tools:
- eldadru/ksniff, kubectl plugin for tcpdump & Wireshark.
- ironcladlou/ditm, Dan's "Dan in the Middle" tool.
- Skydive, network diagnostic and visualization tool.
- ali, a "load testing tool capable of performing real-time analysis" with visualization.
- Testing tools:
- Case studies:
- BZ1763206 is an example of diagnosing DNS latency/timeouts.
- BZ1829779 Investigation details the diagnosis of route latency.
- BZ1845545 is an example of diagnosing misconfigured DNS for an external LB.
- Debugging network stalls on Kubernetes, from the GitHub Blog, about diagnosing Kubernetes performance issues related to ksoftirqd.
Add a NID alias to OWNERS_ALIASES and update the OWNERS file in test/extended/router and add OWNERS file to test/extended/dns
Feature Overview
As a cluster-admin, I want to run update in discrete steps. Update control plane and worker nodes independently.
I also want to back-up and restore incase of a problematic upgrade.
Background:
This Feature is a continuation of https://issues.redhat.com/browse/OCPSTRAT-180.
Customers are asking for improvements to the upgrade experience (both over-the-air and disconnected). This is a feature tracking epics required to get that work done.Below is the list of done tasks.
OTA-700Reduce False Positives (such as Degraded)OTA-922- Better able to show the progress made in each discrete step- [Cover by status command]Better visibility into any errors during the upgrades and documentation of what they error means and how to recover.
Goals
- Have an option to do upgrades in more discrete steps under admin control. Specifically, these steps are:
- Control plane upgrade
- Worker nodes upgrade
- Workload enabling upgrade (i..e. Router, other components) or infra nodes
- An user experience around an end-2-end back-up and restore after a failed upgrade
- MCO-530 - Support in Telemetry for the discrete steps of upgrades
References
Epic Goal
- Eliminate the gap between measured availability and Available=true
Why is this important?
- Today it's not uncommon, even for CI jobs, to have multiple operators which blip through either Degraded=True or Available=False conditions
- We should assume that if our CI jobs do this then when operating in customer environments with higher levels of chaos things will be even worse
- We have had multiple customers express that they've pursued rolling back upgrades because the cluster is telling them that portions of the cluster are Degraded or Unavailable when they're actually not
- Since our product is self-hosted, we can reasonably expect that the instability that we experience on our platform workloads (kube-apiserver, console, authentication, service availability), will also impact customer workloads that run exactly the same way: we're just better at detecting it.
Scenarios
- In all of the following, assume standard 3 master 0 worker or 3 master 2+ worker topologies
- Add/update CI jobs which ensure 100% Degraded=False and Available=True for the duration of upgrade
- Add/update CI jobs which measure availability of all components which are not explicitly defined as non-HA (ex: metal's DHCP server is singleton)
- Address all identified issues
Acceptance Criteria
- openshift/enhancements CONVENTIONS outlines these requirements
- CI - Release blocking jobs include these new/updated tests
- Release Technical Enablement - N/A if we do this we should need no docs
- No outstanding identified issues
Dependencies (internal and external)
- ...
Previous Work (Optional):
- Clayton, David, and Trevor identified many issues early in 4.8 development but were unable to ensure all teams addressed them that list is in this query, teams will be asked to address everything on this list as a 4.9 blocker+ bug and we will re-evaluate status closer to 4.9 code freeze to see which may be deferred to 4.10
https://bugzilla.redhat.com/buglist.cgi?columnlist=product%2Ccomponent%2Cassigned_to%2Cbug_severity%2Ctarget_release%2Cbug_status%2Cresolution%2Cshort_desc%2Cchangeddate&f1=longdesc&f2=cf_environment&j_top=OR&list_id=12012976&o1=casesubstring&o2=casesubstring&query_based_on=ClusterOperator%20conditions&query_format=advanced&v1=should%20not%20change%20condition%2F&v2=should%20not%20change%20condition%2F
Open questions::
- …
Done Checklist
- CI - CI is running, tests are automated and merged.
- DEV - Tests in place
- DEV - No outstanding failing tests
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
These are alarming conditions which may frighten customers, and we don't want to see them in our own, controlled, repeatable update CI. This example job had logs like:
Feb 18 21:11:25.799 E clusteroperator/openshift-apiserver changed Degraded to True: APIServerDeployment_UnavailablePod: APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver ()
And the job failed, but none of the failures were "something made openshift-apiserver mad enough to go Degraded".
Definition of done:
- Same as
OTA-362 - File bugs or the existing issues
- If bug exists then add the tests to the exception list.
- Unless tests are in exception list , they should fail if we see degraded != false.
Feature Overview
This feature aims to enable customers of OCP to integrate 3rd party KMS solutions for encrypting etcd values at rest in accordance with:
https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/
Goals
- Bring KMS v2 API to beta|stable level
- Create/expose mechanisms for customers to plug in containers/operators which can serve the API server's needs (can it be an operator, something provided via CoreOS layering, vanilla container spec provided to API server operator?)
- Provide similar UX experience for all of self-hosted, hypershift, SNO scenarios
- Provide example container/operator for the mechanism
General Prioritization for the Feature
- Approved design for detection & actuation for stand-alone OCP clusters.
- How to detect a problem like an expired/lost key and no contact with the KMS provider?
- How to inform/notify the situation, even at node level, of the situation
- Tech Preview (Feature gated) enabling Kube-KMS v2 for partners to start working on KMS plugin provider integrations:
- Cloud: (priority Azure > AWS > Google)
- Azure KMS
- Azure Dedicated HSM
- AWS KMS
- AWS CloudHSM
- Google Cloud HSM
- On-premise:
- HashiCorp Vault
- EU FSI & EU Telco KMS/HSM top-2 providers
- Cloud: (priority Azure > AWS > Google)
- GA after at least one stable KMS plugin provider
Scenario:
For an OCP cluster with external KMS enabled:
- The customer loses the key to the external KMS
- The external KMS service is degraded or unavailable
How doe the above scenario(s) impact the cluster? The API may be unavailable
Goal:
- Detection: The ability to detect these failure condition(s) and make it visible to the cluster admin.
- Actuation: To what extent can we restore the cluster? ( API availability, Control Plane operators). Recovering customer data is outside of the scope
Investigation Steps:
Detection:
- How do we detect issues with the external KMS?
- How do we detect issues with the KMS plugins?
- How do we surface the information that an issue happened with KMS?
- Metrics / Alerts? Will not work with SNO
- ClusterOperatorStatus?
Actuation:
- Is the control-plane self-recovering?
- What actions are required for the user to recover the cluster partially/completely?
- Complete: kube-apiserver? KMS plugin?
- Partial: kube-apiserver? Etcd? KMS plugin?
User stories that might result in KCS:
- KMS / KMS plugin unavailable
- Is there any degradation? (most likely not with kms v2)
- KMS unavailable and DEK not in cache anymore
- Degradation will most likely occur, but what happens when the KMS becomes available again? Is the cluster self-recovering?
- Key has been deleted and later recovered
- Is the cluster self-recovering?
- KMS / KMS plugin misconfigured
- Is the apiserver rolled-back to the previous healthy revision?
- Is the misconfiguration properly surfaced?
Plugins research:
- What are the pros and cons of managing the plugins ourselves vs leaving that responsibility to the customer?
- What is the list of KMS we need to support?
- Do all the KMS plugins we need to use support KMS v2? If not reach out to the provider
- HSMs?
POCs:
- Have a running POC of KMS on OCP to iterate over the user stories and start testing things out
- Have a hacked version of o/k with https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/3926-handling-undecryptable-resources to be able to easily take actions to fix the clusters as it will be for the customers in 4.17.
Acceptance Criteria:
- Document the detection and actuation process in a KEP.
- Generate new Jira work items based on the new findings.
We did something similar for the aesgcm encryption type in https://github.com/openshift/api/pull/1413/.
Feature Overview (aka. Goal Summary)
Volume Group Snapshots is a key new Kubernetes storage feature that allows multiple PVs to be grouped together and snapshotted at the same time. This enables customers to takes consistent snapshots of applications that span across multiple PVs.
This is also a key requirement for backup and DR solutions.
https://kubernetes.io/blog/2023/05/08/kubernetes-1-27-volume-group-snapshot-alpha/
https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3476-volume-group-snapshot
Goals (aka. expected user outcomes)
Productise the volume group snapshots feature as tech preview have docs, testing as well as a feature gate to enable it in order for customers and partners to test it in advance.
Requirements (aka. Acceptance Criteria):
The feature should be graduated beta upstream to become TP in OCP. Tests and CI must pass and a feature gate should allow customers and partners to easily enable it. We should identify all OCP shipped CSI drivers that support this feature and configure them accordingly.
Use Cases (Optional):
- As a storage vendor I want to have early access to the VolumeGroupSnapshot feature to test and validate my driver support.
- As a backup vendor I want to have early access to the VolumeGroupSnapshot feature to test and validate my backup solution.
- As a customer I want early access to test the VolumeGroupSnapshot feature in order to take consistent snapshots of my workloads that are relying on multiple PVs.
Out of Scope
CSI drivers development/support of this feature.
Background
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
Customer Considerations
Drivers must support this feature and enable it. Partners may need to change their operator and/or doc to support it.
Documentation Considerations
Document how to enable the feature, what this feature does and how to use it. Update the OCP driver's table to include this capability.
Interoperability Considerations
Can be leveraged by ODF and OCP virt, especially around backup and DR scenarios.
Epic Goal*
What is our purpose in implementing this? What new capability will be available to customers?
Add Volume Group Snapshots as Tech Preview. This is a key new Kubernetes storage feature that allows multiple PVs to be grouped together and snapshotted at the same time. This enables customers to takes consistent snapshots of applications that span across multiple PVs.
We will rely on the newly beta promoted feature. This feature is driver dependent.
This will need a new external-snapshotter rebase + removal of the feature gate check in csi-snapshot-controller-operator. Freshly installed or upgraded from older release, will have group snapshot v1beta1 API enabled + enabled support for it in the snapshot-controller (+ ship corresponding external-snapshotter sidecar).
No opt-in, no opt-out.
OCP itself will not ship any CSI driver that supports it.
Why is this important? (mandatory)
What are the benefits to the customer or Red Hat? Does it improve security, performance, supportability, etc? Why is work a priority?
This is also a key requirement for backup and DR solutions specially for OCP virt.
Scenarios (mandatory)
Provide details for user scenarios including actions to be performed, platform specifications, and user personas.
- As a storage vendor I want to have early access to the VolumeGroupSnapshot feature to test and validate my driver support.
- As a backup vendor I want to have early access to the VolumeGroupSnapshot feature to test and validate my backup solution.
- As a customer I want early access to test the VolumeGroupSnapshot feature in order to take consistent snapshots of my workloads that are relying on multiple PVs
Dependencies (internal and external) (mandatory)
What items must be delivered by other teams/groups to enable delivery of this epic.
External snapshotter rebase to the upstream version that include the beta API.
Contributing Teams(and contacts) (mandatory)
Our expectation is that teams would modify the list below to fit the epic. Some epics may not need all the default groups but what is included here should accurately reflect who will be involved in delivering the epic.
- Development - STOR / ODF
- Documentation - STOR
- QE - STOR / ODF
- PX -
- Others -
Acceptance Criteria (optional)
Provide some (testable) examples of how we will know if we have achieved the epic goal.
Since we don't ship any driver with OCP that support the feature we need to have testing with ODF
Drawbacks or Risk (optional)
Reasons we should consider NOT doing this such as: limited audience for the feature, feature will be superseded by other work that is planned, resulting feature will introduce substantial administrative complexity or user confusion, etc.
We're looking at enabling it by default which could introduce risk. Since the feature has recently landed upstream, we will need to rebase on a newer external snapshotter that we initially targeted.
When moving to v1 there may be non backward compatible changes.
Done - Checklist (mandatory)
The following points apply to all epics and are what the OpenShift team believes are the minimum set of criteria that epics should meet for us to consider them potentially shippable. We request that epic owners modify this list to reflect the work to be completed in order to produce something that is potentially shippable.
- CI Testing - Basic e2e automationTests are merged and completing successfully
- Documentation - Content development is complete.
- QE - Test scenarios are written and executed successfully.
- Technical Enablement - Slides are complete (if requested by PLM)
- Engineering Stories Merged
- All associated work items with the Epic are closed
- Epic status should be “Release Pending”
As part of https://issues.redhat.com/browse/COS-2572, the OpenShift installer needs to be adapted so that it can work with bootimages which have no OCP content.
Feature Overview
iSCSI boot is supported in RHEL and since the implementation of OCPSTRAT-749 it's also available in RHCOS.
Customers require using this feature in different bare metal environments on-prem and cloud-based.
Assisted Installer implements support for it in Oracle Cloud Infrastructure (MGMT-16167) to support their bare metal standard "shapes".
This feature extends this support to make it generic and supported in the Agent-Based Installer, the Assisted Installer and in ACM/MCE.
Goals
Support iSCSI boot in bare metal nodes, including platform baremetal and platform "none".
Requirements
Assisted installer can boot and install OpenShift on nodes with iSCSI disks.
Agent-Based Installer can boot and install OpenShift on nodes with iSCSI disks.
MCE/ACM can boot and install OpenShift on nodes with iSCSI disks.
The installation can be done on clusters with platform baremetal and clusters with platform "none".
Epic Goal
Support booting from iSCSI using ABI starting OCP 4.16.
The following PRs are the gaps between release-4.17 branch and master that are needed to make the integration work on 4.17.
https://github.com/openshift/assisted-service/pull/6665
https://github.com/openshift/assisted-service/pull/6603
https://github.com/openshift/assisted-service/pull/6661
The feature has to be backported to 4.16 as well. TBD - list all the PRs that have to be backported.
Instructions to test the AI feature with local env - https://docs.google.com/document/d/1RnRhJN-fgofnVSBTA6mIKcK2_UW7ihbZDLGAVHSdpzc/edit#heading=h.bf4zg53460gu
Why is this important?
- Oracle has a client with disconnected env waiting for it - slack discussion.
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement details and documents.
- iSCSI boot is enabled on ocp >= 4.16
Dependencies (internal and external)
- https://issues.redhat.com/browse/MGMT-16167 - AI support boot for iSCSI for COI
- https://issues.redhat.com/browse/MGMT-17556 - AI generic support for iSCSI boot
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
When testing the agent-based installer using iSCSI with dev-scripts https://github.com/openshift-metal3/dev-scripts/pull/1727 it was found that the installer was not able to complete the installation when using multiple hosts. This same problem did not appear when using SNO.
The iscsi session from all the hosts work do their targets fine until coreos-installer is run, at which time (before reboot) the connection to the target is lost and the coreos-installer fails
Jan 09 16:12:23 master-1 kernel: session1: session recovery timed out after 120 secs Jan 09 16:12:23 master-1 kernel: sd 7:0:0:0: rejecting I/O to offline device Jan 09 16:12:23 master-1 kernel: I/O error, dev sdb, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 2 Jan 09 16:12:23 master-1 installer[2937]: time="2025-01-09T16:12:23Z" level=info msg="\nError: syncing data to disk\n\nCaused by:\n Input/output error (os error 5)\n\nResetting partition table\n" Jan 09 16:12:23 master-1 installer[2937]: time="2025-01-09T16:12:23Z" level=warning msg="Retrying after error: failed executing /usr/bin/nsenter [--target 1 --cgroup --mount --ipc --pid -- coreos-installer install --insecure -i /opt/install-dir/master-f3c24588-2129-483f-9dfb-8a8fe332a4bf.ign --append-karg rd.iscsi.firmware=1 --append-karg ip=enp6s0:dhcp --copy-network /dev/sdb], Error exit status 1, LastOutput \"Error: syncing data to disk\n\nCaused by:\n Input/output error (os error 5)\n\nResetting partition table\nError: syncing partition table to disk\n\nCaused by:\n Input/output error (os error 5)\""
On the host it can be seen that session shows as logged out
Iface Name: default Iface Transport: tcp Iface Initiatorname: iqn.2023-01.com.example:master-1 Iface IPaddress: [default] Iface HWaddress: default Iface Netdev: default SID: 1 iSCSI Connection State: Unknown iSCSI Session State: FREE Internal iscsid Session State: Unknown
The problem occurs because the iscsid service is not running. If it is started by iscsadm then coreos-installer can successfully write the image to disk.
Goal:
Graduate to GA (full support) Gateway API with Istio to unify the management of cluster ingress with a common, open, expressive, and extensible API.
Description:
Gateway API is the evolution of upstream Kubernetes Ingress APIs. The upstream project is part of Kubernetes, working under SIG-NETWORK. OpenShift is contributing to the development, building a leadership position, and preparing OpenShift to support Gateway API, with Istio as our supported implementation.
The plug-able nature of the implementation of Gateway API enables support for additional and optional 3rd-party Ingress technologies.
The team agrees, we should be running the upstream GWAPI conformance tests, as they are readily available and we are an integration product with GWAPI.
We need to answer these questions asked at the March 23, 2023 GWAPI team meeting:
Would it make sense to do it as an optional job in the cluster-ingress-operator?
Is OSSM running the Gateway Conformance test in their CI?
Review what other implementers do with conformance tests to understand what we should do (Do we fork the repo? Clone it? Make a new repo?)
Epic Goal*
Drive the technical part of the Kubernetes 1.31 upgrade, including rebasing openshift/kubernetes repositiry and coordination across OpenShift organization to get e2e tests green for the OCP release.
Why is this important? (mandatory)
OpenShift 4.18 cannot be released without Kubernetes 1.31
Scenarios (mandatory)
Dependencies (internal and external) (mandatory)
What items must be delivered by other teams/groups to enable delivery of this epic.
Contributing Teams(and contacts) (mandatory)
Our expectation is that teams would modify the list below to fit the epic. Some epics may not need all the default groups but what is included here should accurately reflect who will be involved in delivering the epic.
- Development -
- Documentation -
- QE -
- PX -
- Others -
Acceptance Criteria (optional)
Provide some (testable) examples of how we will know if we have achieved the epic goal.
Drawbacks or Risk (optional)
Reasons we should consider NOT doing this such as: limited audience for the feature, feature will be superseded by other work that is planned, resulting feature will introduce substantial administrative complexity or user confusion, etc.
Done - Checklist (mandatory)
The following points apply to all epics and are what the OpenShift team believes are the minimum set of criteria that epics should meet for us to consider them potentially shippable. We request that epic owners modify this list to reflect the work to be completed in order to produce something that is potentially shippable.
- CI Testing - Basic e2e automationTests are merged and completing successfully
- Documentation - Content development is complete.
- QE - Test scenarios are written and executed successfully.
- Technical Enablement - Slides are complete (if requested by PLM)
- Engineering Stories Merged
- All associated work items with the Epic are closed
- Epic status should be “Release Pending”
PRs:
Retro: Kube 1.31 Rebase Retrospective Timeline (OCP 4.18)
Retro recording: https://drive.google.com/file/d/1htU-AglTJjd-VgFfwE3z_dH5tKXT1Tes/view?usp=drive_web
Feature Overview (aka. Goal Summary)
OVN Kubernetes BGP support as a routing protocol for User Defined Network (Segmentation) pod and VM addressability.
Goals (aka. expected user outcomes)
OVN-Kubernetes BGP support enables the capability of dynamically exposing cluster scoped network entities into a provider’s network, as well as program BGP learned routes from the provider’s network into OVN.
OVN-Kubernetes currently has no native routing protocol integration, and relies on a Geneve overlay for east/west traffic, as well as third party operators to handle external network integration into the cluster. This enhancement adds support for BGP as a supported routing protocol with OVN-Kubernetes. The extent of this support will allow OVN-Kubernetes to integrate into different BGP user environments, enabling it to dynamically expose cluster scoped network entities into a provider’s network, as well as program BGP learned routes from the provider’s network into OVN. In a follow-on release, this enhancement will provide support for EVPN, which is a common data center networking fabric that relies on BGP.
Requirements (aka. Acceptance Criteria):
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
| Deployment considerations | List applicable specific needs (N/A = not applicable) |
| Self-managed, managed, or both | |
| Classic (standalone cluster) | |
| Hosted control planes | |
| Multi node, Compact (three node), or Single node (SNO), or all | |
| Connected / Restricted Network | |
| Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
| Operator compatibility | |
| Backport needed (list applicable versions) | |
| UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
| Other (please specify) |
Use Cases (Optional):
- Integration with 3rdparty load balancers that send packets directly to OpenShift nodes with the destination IP address of a targeted pod, without needing custom operators to detect which node a pod is scheduled to and then add routes into the load balancer to send the packet to the right node.
Questions to Answer (Optional):
Out of Scope
- Support of any other routing protocol
- Running separate BGP instances per VRF network
- Support for any other type of L3VPN with BGP, including MPLS
- Providing any type of API or operator to automatically connect two Kubernetes clusters via L3VPN
- Replacing the support that MetalLB provides today for advertising service IPs
- Asymmetric Integrated Routing and Bridging (IRB) with EVPN
Background
BGP
Importing Routes from the Provider Network
Today in OpenShift there is no API for a user to be able to configure routes into OVN. In order for a user to change how cluster traffic is routed egress into the cluster, the user leverages local gateway mode, which forces egress traffic to hop through the Linux host's networking stack, where a user can configure routes inside of the host via NM State. This manual configuration would need to be performed and maintained across nodes and VRFs within each node.
Additionally, if a user chooses to not manage routes within the host and use local gateway mode, then by default traffic is always sent to the default gateway. The only other way to affect egress routing is by using the Multiple External Gateways (MEG) feature. With this feature the user may choose to have multiple different egress gateways per namespace to send traffic to.
As an alternative, configuring BGP peers and which route-targets to import would eliminate the need to manually configure routes in the host, and would allow dynamic routing updates based on changes in the provider’s network.
Exporting Routes into the Provider Network
There exists a need for provider networks to learn routes directly to services and pods today in Kubernetes. Metal LB is already one solution whereby load balancer IPs are advertised by BGP to provider networks, and this feature development does not intend to duplicate or replace the function of Metal LB. Metal LB should be able to interoperate with OVN-Kubernetes, and be responsible for advertising services to a provider’s network.
However, there is an alternative need to advertise pod IPs on the provider network. One use case is integration with 3rd party load balancers, where they terminate a load balancer and then send packets directly to OCP nodes with the destination IP address being the pod IP itself. Today these load balancers rely on custom operators to detect which node a pod is scheduled to and then add routes into its load balancer to send the packet to the right node.
By integrating BGP and advertising the pod subnets/addresses directly on the provider network, load balancers and other entities on the network would be able to reach the pod IPs directly.
EVPN
Extending OVN-Kubernetes VRFs into the Provider Network
This is the most powerful motivation for bringing support of EVPN into OVN-Kubernetes. A previous development effort enabled the ability to create a network per namespace (VRF) in OVN-Kubernetes, allowing users to create multiple isolated networks for namespaces of pods. However, the VRFs terminate at node egress, and routes are leaked from the default VRF so that traffic is able to route out of the OCP node. With EVPN, we can now extend the VRFs into the provider network using a VPN. This unlocks the ability to have L3VPNs that extend across the provider networks.
Utilizing the EVPN Fabric as the Overlay for OVN-Kubernetes
In addition to extending VRFs to the outside world for ingress and egress, we can also leverage EVPN to handle extending VRFs into the fabric for east/west traffic. This is useful in EVPN DC deployments where EVPN is already being used in the TOR network, and there is no need to use a Geneve overlay. In this use case, both layer 2 (MAC-VRFs) and layer 3 (IP-VRFs) can be advertised directly to the EVPN fabric. One advantage of doing this is that with Layer 2 networks, broadcast, unknown-unicast and multicast (BUM) traffic is suppressed across the EVPN fabric. Therefore the flooding domain in L2 networks for this type of traffic is limited to the node.
Multi-homing, Link Redundancy, Fast Convergence
Extending the EVPN fabric to OCP nodes brings other added benefits that are not present in OCP natively today. In this design there are at least 2 physical NICs and links leaving the OCP node to the EVPN leaves. This provides link redundancy, and when coupled with BFD and mass withdrawal, it can also provide fast failover. Additionally, the links can be used by the EVPN fabric to utilize ECMP routing.
Customer Considerations
- For customers using MetalLB, it will continue to function correctly regardless of this development.
Documentation Considerations
Interoperability Considerations
- Multiple External Gateways (MEG)
- Egress IP
- Services
- Egress Service
- Egress Firewall
- Egress QoS
Epic Goal
OVN Kubernetes support for BGP as a routing protocol.
Planning Done Checklist
The following items must be completed on the Epic prior to moving the Epic from Planning to the ToDo status
Priority+ is set by engineering- Epic must be Linked to a +Parent Feature
- Target version+ must be set
- Assignee+ must be set
- (Enhancement Proposal is Implementable
- (No outstanding questions about major work breakdown
- (Are all Stakeholders known? Have they all been notified about this item?
- Does this epic affect SD? {}Have they been notified{+}? (View plan definition for current suggested assignee)
- Please use the “Discussion Needed: Service Delivery Architecture Overview” checkbox to facilitate the conversation with SD Architects. The SD architecture team monitors this checkbox which should then spur the conversation between SD and epic stakeholders. Once the conversation has occurred, uncheck the “Discussion Needed: Service Delivery Architecture Overview” checkbox and record the outcome of the discussion in the epic description here.
- The guidance here is that unless it is very clear that your epic doesn’t have any managed services impact, default to use the Discussion Needed checkbox to facilitate that conversation.
Additional information on each of the above items can be found here: Networking Definition of Planned
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement
details and documents.
...
Dependencies (internal and external)
1.
...
Previous Work (Optional):
1. …
Open questions::
1. …
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
CNO should deploy the new RouteAdvertisements OVN-K CRD.
When the OCP API flag to enable BGP support in the cluster is set, CNO should enable support on OVN-K through a CLI arg.
Feature Overview (aka. Goal Summary)
Customers in highly regulated environments are required to adopt strong ciphers. For control-plane components, this means all components must support the modern TLSProfile with TLS 1.3.
During internal discussions [1] for RFE-4992 and follow-up conversations, it became clear we need a dedicated CI job to track and validate that OpenShift control-plane components are aligned and functional with the TLSProfiles configured on the system.
[1] https://redhat-internal.slack.com/archives/CB48XQ4KZ/p1713288937307359
Goals (aka. expected user outcomes)
- The resulting user outcome from this internal CI validation will be the support of TLS 1.3 in the OpenShift control-plane components.
Requirements (aka. Acceptance Criteria):
- Create a CI job that would test that the cluster stays fully operational when enabling the Modern TLS profile
- This requires updates to our docs
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
| Deployment considerations | List applicable specific needs (N/A = not applicable) |
| Self-managed, managed, or both | both |
| Classic (standalone cluster) | yes |
| Hosted control planes | yes |
| Multi node, Compact (three node), or Single node (SNO), or all | all |
| Connected / Restricted Network | all |
| Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | all |
| Operator compatibility | ?? |
| Backport needed (list applicable versions) | n/a |
| UI need (e.g. OpenShift Console, dynamic plugin, OCM) | n/a |
| Other (please specify) | unknown |
If we try to enable a Modern TLS profile:
EnvVarControllerDegraded: no supported cipherSuites not found in observedConfig
also, if we do manage to pass along the Modern TLS profile cipher suit, we see:
http2: TLSConfig.CipherSuites is missing an HTTP/2-required AES_128_GCM_SHA256 cipher (need at least one of TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 or TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256)
Feature Overview
This is Image mode on OpenShift. It uses the rpm-ostree native containers interface and not bootc but that is an implementation detail.
In the initial delivery of CoreOS Layering, it is required that administrators provide their own build environment to customize RHCOS images. That could be a traditional RHEL environment or potentially an enterprising administrator with some knowledge of OCP Builds could set theirs up on-cluster.
The primary virtue of an on-cluster build path is to continue using the cluster to manage the cluster. No external dependency, batteries-included.
On-cluster, automated RHCOS Layering builds are important for multiple reasons:
- One-click/one-command upgrades of OCP are very popular. Many customers may want to make one or just a few customizations but also want to keep that simplified upgrade experience.
- Customers who only need to customize RHCOS temporarily (hotfix, driver test package, etc) will find off-cluster builds to be too much friction for one driver.
- One of OCP's virtues is that the platform and OS are developed, tested, and versioned together. Off-cluster building breaks that connection and leaves it up to the user to keep the OS up-to-date with the platform containers. We must make it easy for customers to add what they need and keep the OS image matched to the platform containers.
Goals & Requirements
- The goal of this feature is primarily to bring the 4.14 progress (
OCPSTRAT-35) to a Tech Preview or GA level of support. - Customers should be able to specify a Containerfile with their customizations and "forget it" as long as the automated builds succeed. If they fail, the admin should be alerted and pointed to the logs from the failed build.
- The admin should then be able to correct the build and resume the upgrade.
- Intersect with the Custom Boot Images such that a required custom software component can be present on every boot of every node throughout the installation process including the bootstrap node sequence (example: out-of-box storage driver needed for root disk).
- Users can return a pool to an unmodified image easily.
- RHEL entitlements should be wired in or at least simple to set up (once).
- Parity with current features – including the current drain/reboot suppression list, CoreOS Extensions, and config drift monitoring.
This work describes the tech preview state of On Cluster Builds. Major interfaces should be agreed upon at the end of this state.
As a cluster admin of user provided infrastructure,
when I apply the machine config that opts a pool into On Cluster Layering,
I want to also be able to remove that config and have the pool revert back to its non-layered state with the previously applied config.
As a cluster admin using on cluster layering,
when an image build has failed,
I want it to retry 3 times automatically without my intervention and show me where to find the log of the failure.
As a cluster admin,
when I enable On Cluster Layering,
I want to know that the builder image I am building with is stable and will not change unless I change it
so that I keep the same API promises as we do elsewhere in the platform.
To test:
As a cluster admin using on cluster layering,
when I try to upgrade my cluster and the Cluster Version Operator is not available,
I want the upgrade operation to be blocked.
As a cluster admin,
when I use a disconnected environment,
I want to still be able to use On Cluster Layering.
As a cluster admin using On Cluster layering,
When there has been config drift of any sort that degrades a node and I have resolved the issue,
I want to it to resync without forcing a reboot.
As a cluster admin using on cluster layering,
when a pool is using on cluster layering and references an internal registry
I want that registry available on the host network so that the pool can successfully scale up
(MCO-770, MCO-578, MCO-574 )
As a cluster admin using on cluster layering,
when a pool is using on cluster layering and I want to scale up nodes,
the nodes should have the same config as the other nodes in the pool.
Maybe:
Entitlements: MCO-1097, MCO-1099
Not Likely:
As a cluster admin using on cluster layering,
when I try to upgrade my cluster,
I want the upgrade operation to succeed at the same rate as non-OCL upgrades do.
The original scope of this task is represented across this story & MCO-1494.
With OCL GA'ing soon, we'll need a blocking path within our e2e test suite that must pass before a PR can be merged. This story represents the first stage in creating the blocking path:
- Migrate the tests from e2e-gcp-op-techpreview into a new test suite called e2e-ocl. This can be done by moving the tests in the MCO repo from the test/e2e-techpreview folder to a new test/e2e-ocl folder. There might be some minor cleanups such as fixing duplicate function names, etc. but it should be fairly straightforward to do.
- Make a new e2e-gcp-op-ocl job to call the newly created e2e-ocl test suite. This test should first be added as optional for 4.19 so it can be stability tested before it is made blocking for 4.18 & 4.19. This will require a PR to the openshift/release repo to call the new test for 4.19 & master. This should be a pretty straightforward config change.
[REGRESSION] We need to reinvent the wheel for triggering rebuild functionality and the rebuild mechanism as pool labeling and annotation is no longer a favorable way to interact with layered pools
There are a few situations in which a cluster admin might want to trigger a rebuild of their OS image in addition to situations where cluster state may dictate that we should perform a rebuild. For example, if the custom Dockerfile changes or the machine-config-osimageurl changes, it would be desirable to perform a rebuild in that case. To that end, this particular story covers adding the foundation for a rebuild mechanism in the form of an annotation that can be applied to the target MachineConfigPool. What is out of scope for this story is applying this annotation in response to a change in cluster state (e.g., custom Dockerfile change).
Done When:
- BuildController is aware of and recognizes a special annotation on layered MachineConfigPools (e.g., machineconfiguration.openshift.io/rebuildImage).
- Upon recognizing that a MachineConfigPool has this annotation, BuildController will clear any failed build attempts, delete any failed builds and their related ephemeral objects (e.g., rendered Dockerfile / MachineConfig ConfigMaps), and schedule a new build to be performed.
- This annotation should be removed when the build process completes, regardless of outcome. In other words, should the build success or fail, the annotation should be removed.
- [optional] BuildController keeps track of the number of retries for a given MachineConfigPool. This can occur via another annotation, e.g., machineconfiguration.openshift.io/buildRetries=1 . For now, this can be a hard-coded value (e.g., 5), but in the future, this could be wired up to an end-user facing knob. This annotation should be cleared upon a successful rebuild. If the rebuild is reached, then we should degrade.
This work describes the tech preview state of On Cluster Builds. Major interfaces should be agreed upon at the end of this state.
As a cluster admin of user provided infrastructure,
when I apply the machine config that opts a pool into On Cluster Layering,
I want to also be able to remove that config and have the pool revert back to its non-layered state with the previously applied config.
As a cluster admin using on cluster layering,
when an image build has failed,
I want it to retry 3 times automatically without my intervention and show me where to find the log of the failure.
As a cluster admin,
when I enable On Cluster Layering,
I want to know that the builder image I am building with is stable and will not change unless I change it
so that I keep the same API promises as we do elsewhere in the platform.
To test:
As a cluster admin using on cluster layering,
when I try to upgrade my cluster and the Cluster Version Operator is not available,
I want the upgrade operation to be blocked.
As a cluster admin,
when I use a disconnected environment,
I want to still be able to use On Cluster Layering.
As a cluster admin using On Cluster layering,
When there has been config drift of any sort that degrades a node and I have resolved the issue,
I want to it to resync without forcing a reboot.
As a cluster admin using on cluster layering,
when a pool is using on cluster layering and references an internal registry
I want that registry available on the host network so that the pool can successfully scale up
(MCO-770, MCO-578, MCO-574 )
As a cluster admin using on cluster layering,
when a pool is using on cluster layering and I want to scale up nodes,
the nodes should have the same config as the other nodes in the pool.
Maybe:
Entitlements: MCO-1097, MCO-1099
Not Likely:
As a cluster admin using on cluster layering,
when I try to upgrade my cluster,
I want the upgrade operation to succeed at the same rate as non-OCL upgrades do.
As a follow up to https://issues.redhat.com/browse/MCO-1284, the one field we identified that is best updated pre-GA is to make the baseImagePullSecret optional. The builder pod should have the logic to fetch from baseImagePullSecret if the user does not specify this via a MachineOSConfig object.
To make OCL ready for GA, the first step would be graduating the MCO's APIs from v1alpha1 to v1. This requires changes in the openshift/api repo.
Feature Overview
As a cluster admin for standalone OpenShift, I want to customize the prefix of the machine names created by CPMS due to company policies related to nomenclature. Implement the Control Plane Machine Set (CPMS) feature in OpenShift to support machine names where user can set custom names prefixes. Note the prefix will always be suffixed by "<5-chars>-<index>" as this is part of the CPMS internal design.
Acceptance Criteria
A new field called machineNamePrefix has been added to CPMS CR.
This field would allow the customer to specify a custom prefix for the machine names.
The machine names would then be generated using the format: <machineNamePrefix><5-chars><index>
Where:
<machineNamePrefix> is the custom prefix provided by the customer
<5-chars> is a random 5 character string (this is required and cannot be changed)
<index> represents the index of the machine (0, 1, 2, etc.)
Ensure that if the machineNamePrefix is changed, the operator reconciles and succeeds in rolling out the changes.
Epic Goal
- Provide a new field to the CPMS that allows to define a Machine name prefix
- This prefix will supersede the current usage of the control plane label and role combination we use today
- The names must still continue to be suffixed with <chars>-<idx> as this is important to the operation of CPMS
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement details and documents.
Done Checklist
- CI - CI is running, tests are automated and merged.
- DEV - Downstream code and tests merged: <link to meaningful PR or GitHub Issue>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
To be able to gather test data for this feature, we will need to introduce tech preview periodics, so we need to duplicate each of https://github.com/openshift/release/blob/8f8c7c981c3d81c943363a9435b6c48005eec6e3[…]control-plane-machine-set-operator-release-4.19__periodics.yaml and add the techpreview configuration.
It's configured as an env var, so copy each job, add the env var, and change the name to include -techpreview as a suffic
env: FEATURE_SET: TechPreviewNoUpgrade
Bump openshift/api to vendor machineNamePrefix field and CPMSMachineNamePrefix feature-gate into cpms-operator
Define a new feature gate in openshift/api for this feature so that all the implementation can be safe guarded behind this gate.
Description of problem:
Consecutive points are not allowed in machineNamePrefix, but from the prompt "spec.machineNamePrefix: Invalid value: "string": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character.", consecutive points should be allowed. And I can create machine on providers console with consecutive points https://drive.google.com/file/d/1p5QLhkL4VI3tt3viO98zYG8uqb1ePTnB/view?usp=sharing
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-2025-01-08-165032
How reproducible:
Always
Steps to Reproduce:
1.Update machineNamePrefix containing consecutive points in controlplanemachineset, like
machineNamePrefix: huliu..azure
2.Cannot save, get prompt
# * spec.machineNamePrefix: Invalid value: "string": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character.
3. If change to
machineNamePrefix: huliu.azure
then it can save successfully, and rollout succeed.
Actual results:
Cannot save, get prompt
Expected results:
Can save successfully
Additional info:
New feature testing for https://issues.redhat.com/browse/OAPE-22
- Utilize the new field added in openshift/api and add the implementation so that custom name (prefix) formats can be assigned to Control Plane Machines via CPMS.
- All the changes should be behind the feature gate.
- This prefix will supersede the current usage of the control plane label and role combination we use today
- The names must still continue to be suffixed with <chars><idx> as this is important to the operation of CPMS
- Add unit tests and E2E tests.
Provide a new field to the CPMS that allows to define a Machine name prefix
Feature description
Oc-mirror v2 is focuses on major enhancements that include making oc-mirror faster and more robust and introduces caching as well as address more complex air-gapped scenarios. OC mirror v2 is a rewritten version with three goals:
- Manage complex air-gapped scenarios, providing support for the enclaves feature
- Faster and more robust: introduces caching, it doesn’t rebuild catalogs from scratch
- Improves code maintainability, making it more reliable and easier to add features, and fixes, and including a feature plugin interface
There was a selected bundle feature on v2 that needs to be removed in 4.18 because of the its risk.
An alternative solution is required to unblock one of our customers.
Feature Overview (aka. Goal Summary)
Edge customers requiring computing on-site to serve business applications (e.g., point of sale, security & control applications, AI inference) are asking for a 2-node HA solution for their environments. Only two nodes at the edge, because the 3d node induces too much cost, but still they need HA for critical workload. To address this need, a 2+1 topology is introduced. It supports a small cheap arbiter node that can optionally be remote/virtual to reduce onsite HW cost.
Goals (aka. expected user outcomes)
Support OpenShift on 2+1 topology, meaning two primary nodes with large capacity to run workload and control plan, and a third small “arbiter” node which ensure quorum. See requirements for more details.
Requirements (aka. Acceptance Criteria):
- Co-located arbiter node - 3d node in same network/location with low latency network access, but the arbiter node is much smaller compared to the two main nodes. Target resource requirements for the arbiter node: 4 cores / 8 vcpu, 16G RAM, 120G disk (non-spinning), 1x1 GbE network ports, no BMC
- OCP Virt fully functionally, incl. Live migration of VMs (assuming RWX CSI Driver is available)
- Single Node outage is handled seamlessly
- In case the arbiter node is down , a reboot/restart of the two remaining nodes has to work, i.e. the two remaining nodes re-gain quorum and spin-up the workload.
- Scale out of the cluster by adding additional worker nodes should be possible
- Transition the cluster into a regular 3 node compact cluster, e.g. by adding a new node as control plane node, then removing the arbiter node, should be possible
- Regular workload should not be scheduled to the arbiter node (e.g by making it un-schedulabe, or introduce a new node role “arbiter”). Only essential control plane workload (etcd components) should run on the arbiter node. Non-essential control plan workload (i.e. router, registry, console, monitoring etc) should also not be scheduled to the arbiter nodded.
- It must be possible to explicitly schedule additional workload to the arbiter node. That is important for 3d party solutions (e.g. storage provider) which also have quorum based mechanisms.
- must seamlessly integrate into existing installation/update mechanisms, esp. zero touch provisioning etc.
- Added: ability to track TNA usage in the fleet of connected clusters via OCP telemetry data (e.g. number of clusters with TNA topology)
| Deployment considerations | List applicable specific needs (N/A = not applicable) |
| Self-managed, managed, or both | self-managed |
| Classic (standalone cluster) | yes |
| Hosted control planes | no |
| Multi node, Compact (three node), or Single node (SNO), or all | Multi node and Compact (three node) |
| Connected / Restricted Network | both |
| Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | x86_86 and ARM |
| Operator compatibility | full |
| Backport needed (list applicable versions) | no |
| UI need (e.g. OpenShift Console, dynamic plugin, OCM) | no |
| Other (please specify) | n/a |
Questions to Answer (Optional):
- How to implement the scheduling restrictions to the arbiter node? New node role “arbiter”?
- Can this be delivered in one release, or do we need to split, e.g. TechPreview + GA?
Out of Scope
- Storage driver providing RWX shared storage
- …
Background
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
- Two node support is in high demand by telco, industrial and retail customers.
- VMWare supports a two node VSan solution: https://core.vmware.com/resource/vsan-2-node-cluster-guide
- Example edge hardware frequently used for edge deployments with a co-located small arbiter node: Dell PowerEdge XR4000z Server is an edge computing device that allows restaurants, retailers, and other small to medium businesses to set up local computing for data-intensive workloads.
Customer Considerations
See requirements - there are two main groups of customers: co-located arbiter node, and remote arbiter node.
Documentation Considerations
- Topology needs to be documented, esp. The requirements of the arbiter node.
Interoperability Considerations
- OCP Virt needs to be explicitly tested on this scenario to support VM HA (live migration, restart on other node)
We will need to make sure the MCO contains the bootstrap files for the arbiter node similar to the files it contains for master nodes.
We need to make sure kubelet respects the new node role type for arbiter. This will be a simple update on the well_known_openshift_labels.go list to include the new node role.
Once the HighlyAvailableArbiter has been added to the ocp/api, we need to update the cluster-config-operator dependencies to reference the new change, so that it propagates to cluster installs in our payloads.
We need to update CEO (cluster etcd operator) to understand what an arbiter/witness node is so it can properly assign an etcd member on our less powerful node.
Update the dependencies for CEO for library-go and ocp/api to support the Arbiter additions, doing this in a separate PR to keep things clean and easier to test.
The console operand checks for valid infra control plane topology fields.
We'll need to add the HighlyAvailableArbiter flag to that check
The console operator performs a check for HighlyAvailable, we will need to add the HighlyAvailableArbiter check there so that it supports it on the same level as HA
Authentication Operator will need to be updated to take in the newly created `HighlyAvailableArbiter` topology flag, and allow the minimum of kube-apiservers running to equal 2.
We need to add the `HighlyAvailableArbiter` value to the controlPlaneTopology in ocp/api package as a TechPreview
https://github.com/openshift/api/blob/master/config/v1/types_infrastructure.go#L95-L103
Bump the openshift/api dependency to latest for the arbiter node enum.
Make sure pacemaker configured in TNF controls etcd.
Verify the following scenarios:
- Etcd agent created: start/stop/promote/demote
- Cold boot
- Graceful shutdown
For TNF we need to replace our currently running etcd after the installation with the one that is managed by pacemaker.
This allows us to keep the following benefits:
- installation can stay the way it is
- upgrade paths are not blocked
- we can keep existing operator functionality around (cert rotation, defrags etc)
- pacemaker can control etcd without any other interference
AC:
- on a (later specified) signal, we roll out a static pod revision without the etcd container
Feature Overview
Support mapping OpenShift zones to vSphere host groups, in addition to vSphere clusters.
When defining zones for vSphere administrators can map regions to vSphere datacenters and zones to vSphere clusters.
There are use cases where vSphere clusters have only one cluster construct with all their ESXi hosts but the administrators want to divide the ESXi hosts in host groups. A common example is vSphere stretched clusters, where there is only one logical vSphere cluster but the ESXi nodes are distributed across to physical sites, and grouped by site in vSphere host groups.
In order for OpenShift to be able to distribute its nodes on vSphere matching the physical grouping of hosts, OpenShift zones have to be able to map to vSphere host groups too.
Requirements{}
- Users can define OpenShift zones mapping them to host groups at installation time (day 1)
- Users can use host groups as OpenShift zones post-installation (day 2)
Epic Goal
Support mapping OpenShift zones to vSphere host groups, in addition to vSphere clusters.
When defining zones for vSphere administrators can map regions to vSphere datacenters and zones to vSphere clusters.
There are use cases where vSphere clusters have only one cluster construct with all their ESXi hosts but the administrators want to divide the ESXi hosts in host groups. A common example is vSphere stretched clusters, where there is only one logical vSphere cluster but the ESXi nodes are distributed across to physical sites, and grouped by site in vSphere host groups.
In order for OpenShift to be able to distribute its nodes on vSphere matching the physical grouping of hosts, OpenShift zones have to be able to map to vSphere host groups too.
Requirements{}
- Users can define OpenShift zones mapping them to host groups at installation time (day 1)
- Users can use host groups as OpenShift zones post-installation (day 2)
As an openshift engineer update the infrastructure and the machine api objects so it can support host vm group zonal
Acceptance criteria
- Add the appropriate fields for host vm group zonal
- Add the appropriate documentation for the above fields
- Add a new feature gate for host vm zonal
As an openshift engineer enable host vm group zonal in mao so that compute nodes properly are deployed
Acceptance Criteria:
- Modify workspace to include vmgroup
- properly configure vsphere cluster to add vm into vmgroup
As an openshift engineer enable host vm group zonal in CPMS so that control plane nodes properly are redeployed
Acceptance Criteria:
- Control plane nodes properly roll out when requried
- Control plane nodes do not roll out when not needed
{}USER STORY:{}
As someone that installs openshift on vsphere, I want to install zonal via host and vm groups so that I can use a stretched physical cluster or use a cluster as a region and hosts as zones .
{}DESCRIPTION:{}
{}Required:{}
{}Nice to have:{}
{}ACCEPTANCE CRITERIA:{}
- start validating tag naming
- validate tags exist
- validate host group exists
- update platform spec
- unit tests
- create capv deployment and failure domain manifests
- per failure domain, create vm group and vm host rule
{}ENGINEERING DETAILS:{}
Configuration steps:
- Create tag and tag categories
- Attach zonal tags to ESXi hosts - capv will complain extensively if this is not done
- Host groups {}MUST{} be created and populated prior to installation (maybe, could we get the hosts that are attached or vice versa hosts in host group are attached?)
one per zone
- vm groups wil be created by the installer one per zone
- vm / host rule will be created by the installer one per zone
https://github.com/openshift/installer/compare/master...jcpowermac:installer:test-vm-host-group
Feature Overview (aka. Goal Summary)
Overarching Goal
Move to using the upstream Cluster API (CAPI) in place of the current implementation of the Machine API for standalone Openshift.
Phase 1 & 2 covers implementing base functionality for CAPI.
Phase 2 also covers migrating MAPI resources to CAPI.
Phase 2 Goal:
- Complete the design of the Cluster API (CAPI) architecture and build the core operator logic
- attach and detach of load balancers for internal and external load balancers for control plane machines on AWS, Azure, GCP and other relevant platforms
- manage the lifecycle of Cluster API components within OpenShift standalone clusters
- E2E tests
for Phase-1, incorporating the assets from different repositories to simplify asset management.
Background, and strategic fit
- Initially CAPI did not meet the requirements for cluster/machine management that OCP had the project has moved on, and CAPI is a better fit now and also has better community involvement.
- CAPI has much better community interaction than MAPI.
- Other projects are considering using CAPI and it would be cleaner to have one solution
- Long term it will allow us to add new features more easily in one place vs. doing this in multiple places.
Acceptance Criteria
There must be no negative effect to customers/users of the MAPI, this API must continue to be accessible to them though how it is implemented "under the covers" and if that implementation leverages CAPI is open
Epic Goal
- To prepare the openshift cluster-api project for general availability
- Ensure proper testing of the CAPI installer operator
Why is this important?
- We need to be able to install and lifecycle the Cluster API ecosystem within standalone OpenShift
- We need to make sure that we can update the components via an operator
- We need to make sure that we can lifecycle the APIs via an operator
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement details and documents.
The current ClusterOperator status conditions—Available, Progressing, Upgradeable, and Degraded—are set by the corecluster controller independently of the status of other controllers.
This approach does not align with the intended purpose of these conditions, which are meant to reflect the overall status of the operator, considering all the controllers it manages.
To address this, we should introduce controller-level conditions similar to the top-level ones. These conditions would influence an aggregated top-level status, which a new controller (the ClusterOperator controller) would then consolidate into the Available, Progressing, Upgradeable, and Degraded conditions.
Moreover, when running `oc get co` against the cluster-capi-operator, only the name and version are returned. The status is not rolled up into the additional columns as expected.
Before GA, this state information should be visible from `oc get co`
Feature Overview (aka. Goal Summary)
OpenShift is traditionally a single-disk system, meaning the OpenShift installer is designed to install the OpenShift filesystem on one disk. However, new customer use cases have highlighted the need for the ability to configure additional disks during installation. Here are the specific use cases:
- Dedicated Disk for etcd:
-
- As a user, I would like to install a cluster with a dedicated disk for etcd.
- Our recommended practices for etcd suggest using a dedicated disk for optimal performance.
- Managing disk mounting through MCO can be challenging and may introduce additional issues.
- Cluster-API supports running etcd on a dedicated disk.
- Dedicated Disk for Swap Partitions:
-
- As a user, I would like to install a cluster with swap enabled on each node and utilize a dedicated disk for swap partitions.
- A dedicated disk for swap would help prevent swap activity from impacting node performance.
- Dedicated Disk for Container Runtime:
-
- As a user, I would like to install a cluster and assign a separate disk for the container runtime.
- Dedicated Disk for Image Storage:
-
- As a user, I would like to install a cluster with images stored on a dedicated disk, while keeping ephemeral storage on the node's main filesystem.
User Story:
As an OpenShift administrator, I need to be able to configure my OpenShift cluster to have additional disks on each vSphere VM so that I can use the new data disks for various OS needs.
Description:
This goal of this epic is to be able to allow the cluster administrator to install and configure after install new machines with additional disks attached to each virtual machine for various OS needs.
Required:
- Installer allows configuring additional disks for control plane and compute virtual machines
- Control Plane Machine Sets (CPMS) allows configuring control plane virtual machines with additional disks
- Machine API (MAPI) allows for configuring Machines and MachineSets with additional disks
- Cluster API (CAPI) allows for configuring Machines and MachineSets with additional disks
Nice to Have:
Acceptance Criteria:
Notes:
User Story:
As an OpenShift Engineer I need to ensure the the CPMS Operator now works with detecting any changes needed when data disks are added to the CPMS definition.
Description:
This task is to verify if any changes are needed in the CPMS Operator to handle the change data disk definitions in the CPMS.
Acceptance Criteria:
- CPMS does not roll out changes when initial install is performed.
- Adding a disk to CPMS results in control plane roll out.
- Removing a disk from CPMS results in control plane roll out.
- No errors logged as a result of data disks being present in the CPMS definition.
Notes:
Ideally we just need to make sure the operator is updated to pull in the new CRD object definitions that contain the new data disk field.
User Story:
As an OpenShift Engineer I need to ensure the the MAPI Operator.
Description:
This task is to verify if any changes are needed in the MAPI Operator to handle the change data disk definitions in the CPMS.
Acceptance Criteria:
- Adding a disk to MachineSet does not result in new machines being rolled out.
- Removing a disk from MachineSet does not result in new machines being rolled out.
- After making changes to a MachineSet related to data disks, when MachineSet is scaled down and then up, new machines contain the new data disk configurations.
- All attempts to modify existing data disk definitions in an existing Machine definition are blocked by the webhook.
Notes:
The behaviors for the data disk field should be the same as all other provider spec level fields. We want to make sure that the new fields are no different than the others. This field is not hot swap capable for running machines. A new VM must be created for this feature to work.
User Story:
As an OpenShift Engineer I need to enhance the OpenShift installer to support creating a cluster with additional disks added to control plane and compute nodes so that I can use the new data disks for various OS needs.
Description:
This task is to enhance the installer to allow configuring data disks in the install-config.yaml. This will also require setting the necessary fields in machineset and machine definitions. The important one being for CAPV to do the initial creation of disks for the configured masters.
Acceptance Criteria:
- install-config.yaml supports configuring data disks in all machinepools.
- CAPV has been updated with new multi disk support.
- CAPV machines are created that result in control plane nodes with data disks.
- MachineSet definitions for compute nodes are created correctly with data disk values from compute pool.
- CPMS definition for masters has the data disks configured correctly.
Notes:
We need to be sure that after installing a cluster, the cluster remains stable and has all correct configurations.
User Story:
As an OpenShift Engineer I need to create a new feature gate and CRD changes for vSphere multi disk so that we can gate the new function until all bugs are ironed out.
Description:
This task is to create the new feature gate to be used by all logical changes around multi disk support for vSphere. We also need to update the types for vsphere machine spec to include new array field that contains data disk definitions.
Acceptance Criteria:
- New feature gate exists for components to use.
- Initial changes to the CRD for data disks are present for components to start using.
- https://github.com/openshift/cluster-network-operator/pull/2616
- https://github.com/openshift/cluster-network-operator/pull/2591
- https://github.com/openshift/ovn-kubernetes/pull/2429
- https://github.com/openshift/ovn-kubernetes/pull/2427
- https://github.com/openshift/ovn-kubernetes/pull/2420
- https://github.com/openshift/origin/pull/29517
- https://github.com/openshift/origin/pull/29509
- https://github.com/openshift/origin/pull/29501
- https://github.com/openshift/origin/pull/29510
- https://github.com/openshift/origin/pull/29511
- https://github.com/openshift/origin/pull/29494
- https://github.com/openshift/origin/pull/29482
- https://github.com/openshift/origin/pull/29479
- https://github.com/openshift/origin/pull/29477
- https://github.com/openshift/origin/pull/29468
- https://github.com/openshift/origin/pull/29469
- https://github.com/openshift/origin/pull/29458
- https://github.com/openshift/origin/pull/29463
- https://github.com/openshift/origin/pull/29429
- https://github.com/openshift/origin/pull/29454
- https://github.com/openshift/origin/pull/29451
- https://github.com/openshift/origin/pull/29446
- https://github.com/openshift/origin/pull/29445
- https://github.com/openshift/origin/pull/29448
- https://github.com/openshift/origin/pull/29436
- https://github.com/openshift/origin/pull/29253
- https://github.com/openshift/origin/pull/29381
See the UDN Sync Meeting notes: https://docs.google.com/document/d/1wjT8lSDfG6Mj-b45_p9SlbpCVG0af-pjDlmAUnM5mOI/edit#bookmark=id.zemsagq71ou2
In our current UDN API, subnets field is mandatory always for primary role and optional for secondary role. This is because users are allowed to have a pure L2 without subnets for secondary networks. However, in the future if we want to add egress support on secondary networks, we might need subnets...
CNV has many different use cases:
- For UDPNs, we always need subnets for L2 and L3 why not make them optional and let users get default values? -> drawback is loosing visibility and this podsubnet now conflicting with other internal subnets and customer in their ignorange have the oopsy stage, we have seen this in plenty with joinsubnets already
- For UDSNs, we may or maynot have the need for IPAM, today this subnets field is optional, but then when we do need subnets we cannot set default values here so its icky.
This card tracks the design changes to the API and the code changes needed to implement this. See https://github.com/openshift/enhancements/pull/1708 for details.
We want to do Network Policies not MultiNetwork POlicies
Feature Overview (aka. Goal Summary)
Review, refine and harden the CAPI-based Installer implementation introduced in 4.16
Goals (aka. expected user outcomes)
From the implementation of the CAPI-based Installer started with OpenShift 4.16 there is some technical debt that needs to be reviewed and addressed to refine and harden this new installation architecture.
Requirements (aka. Acceptance Criteria):
Review existing implementation, refine as required and harden as possible to remove all the existing technical debt
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
| Deployment considerations | List applicable specific needs (N/A = not applicable) |
| Self-managed, managed, or both | |
| Classic (standalone cluster) | |
| Hosted control planes | |
| Multi node, Compact (three node), or Single node (SNO), or all | |
| Connected / Restricted Network | |
| Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
| Operator compatibility | |
| Backport needed (list applicable versions) | |
| UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
| Other (please specify) |
Documentation Considerations
There should not be any user-facing documentation required for this work
Epic Goal
- This epic includes tasks the team would like to tackle to improve our process, QOL, CI. It may include tasks like updating the RHEL base image and vendored assisted-service.
Why is this important?
We need a place to add tasks that are not feature oriented.
Scenarios
- ...
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement details and documents.
- ...
Dependencies (internal and external)
- ...
Previous Work (Optional):
- ...
Open questions::
- ...
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
Most of the platform subdirectories don't have OWNERS files
we should add the aliases for everything that's missing
backport to 4.16
Feature Overview (aka. Goal Summary)
This feature aims to comprehensively refactor and standardize various components across HCP, ensuring consistency, maintainability, and reliability. The overarching goal to increase customer satisfaction by increasing speed to market and saving engineering budget by reducing incidents/bugs. This will be achieved by reducing technical debt, improving code quality, and simplifying the developer experience across multiple areas, including CLI consistency, NodePool upgrade mechanisms, networking flows, and more. By addressing these areas holistically, the project aims to create a more sustainable and scalable codebase that is easier to maintain and extend.
Goals (aka. Expected User Outcomes)
- Unified Codebase: Achieve a consistent and unified codebase across different HCP components, reducing redundancy and making the code easier to understand and maintain.
- Enhanced Developer Experience: Streamline the developer workflow by reducing boilerplate code, standardizing interfaces, and improving documentation, leading to faster and safer development cycles.
- Improved Maintainability: Refactor large, complex components into smaller, modular, and more manageable pieces, making the codebase more maintainable and easier to evolve over time.
- Increased Reliability: Enhance the reliability of the platform by increasing test coverage, enforcing immutability where necessary, and ensuring that all components adhere to best practices for code quality.
- Simplified Networking and Upgrade Mechanisms: Standardize and simplify the handling of networking flows and NodePool upgrade triggers, providing a clear, consistent, and maintainable approach to these critical operations.
Requirements (aka. Acceptance Criteria)
- Standardized CLI Implementation: Ensure that the CLI is consistent across all supported platforms, with increased unit test coverage and refactored dependencies.
- Unified NodePool Upgrade Logic: Implement a common abstraction for NodePool upgrade triggers, consolidating scattered inputs and ensuring a clear, consistent upgrade process.
- Refactored Controllers: Break down large, monolithic controllers into modular, reusable components, improving maintainability and readability.
- Improved Networking Documentation and Flows: Update networking documentation to reflect the current state, and refactor network proxies for simplicity and reusability.
- Centralized Logic for Token and Userdata Generation: Abstract the logic for token and userdata generation into a single, reusable library, improving code clarity and reducing duplication.
- Enforced Immutability for Critical API Fields: Ensure that immutable fields within key APIs are enforced through proper validation mechanisms, maintaining API coherence and predictability.
- Documented and Clarified Service Publish Strategies: Provide clear documentation on supported service publish strategies, and lock down the API to prevent unsupported configurations.
Use Cases (Optional)
- Developer Onboarding: New developers can quickly understand and contribute to the HCP project due to the reduced complexity and improved documentation.
- Consistent Operations: Operators and administrators experience a more predictable and consistent platform, with reduced bugs and operational overhead due to the standardized and refactored components.
Out of Scope
- Introduction of new features or functionalities unrelated to the refactor and standardization efforts.
- Major changes to user-facing commands or APIs beyond what is necessary for standardization.
Background
Over time, the HyperShift project has grown organically, leading to areas of redundancy, inconsistency, and technical debt. This comprehensive refactor and standardization effort is a response to these challenges, aiming to improve the project's overall health and sustainability. By addressing multiple components in a coordinated way, the goal is to set a solid foundation for future growth and development.
Customer Considerations
- Minimal Disruption: Ensure that existing users experience minimal disruption during the refactor, with clear communication about any changes that might impact their workflows.
- Enhanced Stability: Customers should benefit from a more stable and reliable platform as a result of the increased test coverage and standardization efforts.
Documentation Considerations
Ensure all relevant project documentation is updated to reflect the refactored components, new abstractions, and standardized workflows.
This overarching feature is designed to unify and streamline the HCP project, delivering a more consistent, maintainable, and reliable platform for developers, operators, and users.
Goal
Refactor and modularize controllers and other components to improve maintainability, scalability, and ease of use.
User Story:
As a (user persona), I want to be able to:
- As an external dev I want to be able to add new components to the CPO easily
- As a core dev I want to feel safe when adding new components to the CPO
- As a core dev I want to add new components to the CPO with our copy/pasting big chunks of code
https://issues.redhat.com//browse/HOSTEDCP-1801 introduced a new abstraction to be used by ControlPlane components. However, when a component or a sub-resources predicate changes to false, the resources are not removed from the cluster. All resources should be deleted from the cluster.
docs: https://github.com/openshift/hypershift/blob/main/support/controlplane-component/README.md
User Story:
As a (user persona), I want to be able to:
- As an external dev I want to be able to add new components to the CPO easily
- As a core dev I want to feel safe when adding new components to the CPO
- As a core dev I want to add new components to the CPO with our copy/pasting big chunks of code
https://issues.redhat.com//browse/HOSTEDCP-1801 introduced a new abstraction to be used by ControlPlane components. We need to refactor every component to use this abstraction.
Acceptance Criteria:
Description of criteria:
All ControlPlane Components are refactored:
HCCOkube-apiserver (Mulham)kube-controller-manager (Mulham)ocm (Mulham)etcd (Mulham)oapi (Mulham)scheduler (Mulham)clusterpolicy (Mulham)CVO (Mulham)oauth (Mulham)hcp-router (Mulham)storage (Mulham)CCO (Mulham)- CNO (Jparrill)
- CSI (Jparrill)
- dnsoperator
- ignition (Ahmed)
- ingressoperator (Bryan)
- machineapprover
- nto
- olm
- pkioperator
- registryoperator (Bryan)
- snapshotcontroller
Example PR to refactor cloud-credential-operator : https://github.com/openshift/hypershift/pull/5203
docs: https://github.com/openshift/hypershift/blob/main/support/controlplane-component/README.md
Provide a PR with a CNO standard refactor
Example PR to refactor HCCO: https://github.com/openshift/hypershift/pull/4860
docs: https://github.com/openshift/hypershift/blob/main/support/controlplane-component/README.md
Provide a PR with a CSI standard refactor
Example PR to refactor HCCO: https://github.com/openshift/hypershift/pull/4860
docs: https://github.com/openshift/hypershift/blob/main/support/controlplane-component/README.md
Goal
Improve the consistency and reliability of APIs by enforcing immutability and clarifying service publish strategy support.
Why is this important?
- …
Scenarios
- ...
Acceptance Criteria
- Dev - Has a valid enhancement if necessary
- CI - MUST be running successfully with tests automated
- QE - covered in Polarion test plan and tests implemented
- Release Technical Enablement - Must have TE slides
- ...
Dependencies (internal and external)
- ...
Previous Work (Optional):
- …
Open questions:
- …
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Technical Enablement <link to Feature Enablement Presentation>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Enhancement merged: <link to meaningful PR or GitHub Issue>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
User Story:
As a (user persona), I want to be able to:
- Capability 1
- Capability 2
- Capability 3
so that I can achieve
- Outcome 1
- Outcome 2
- Outcome 3
Acceptance Criteria:
Description of criteria:
- Upstream documentation
- Point 1
- Point 2
- Point 3
(optional) Out of Scope:
Detail about what is specifically not being delivered in the story
Engineering Details:
- (optional) https://github/com/link.to.enhancement/
- (optional) https://issues.redhat.com/link.to.spike
- Engineering detail 1
- Engineering detail 2
This requires/does not require a design proposal.
This requires/does not require a feature gate.
Feature Overview (aka. Goal Summary)
An elevator pitch (value statement) that describes the Feature in a clear, concise way. Complete during New status.
<your text here>
Goals (aka. expected user outcomes)
The observable functionality that the user now has as a result of receiving this feature. Include the anticipated primary user type/persona and which existing features, if any, will be expanded. Complete during New status.
<your text here>
Requirements (aka. Acceptance Criteria):
A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.
<enter general Feature acceptance here>
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
| Deployment considerations | List applicable specific needs (N/A = not applicable) |
| Self-managed, managed, or both | |
| Classic (standalone cluster) | |
| Hosted control planes | |
| Multi node, Compact (three node), or Single node (SNO), or all | |
| Connected / Restricted Network | |
| Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
| Operator compatibility | |
| Backport needed (list applicable versions) | |
| UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
| Other (please specify) |
Use Cases (Optional):
Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.
<your text here>
Questions to Answer (Optional):
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
<your text here>
Out of Scope
High-level list of items that are out of scope. Initial completion during Refinement status.
<your text here>
Background
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
<your text here>
Customer Considerations
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
<your text here>
Documentation Considerations
Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.
<your text here>
Interoperability Considerations
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
<your text here>
Epic Goal
- ...
Why is this important?
- …
Scenarios
- ...
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement details and documents.
- ...
Dependencies (internal and external)
- ...
Previous Work (Optional):
- …
Open questions::
- …
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
PopupKebabMenu is orphaned and contains a reference to `@patternfly/react-core/deprecated`. It and related code should be removed so we can drop PF4 and adopt PF6.
Before we can adopt PatternFly 6, we need to drop PatternFly 4. We should drop 4 first so we can understand what impact if any that will have on plugins.
AC:
- Remove PF4 package. Console should not load any PF4 assets during the runtime.
- Remove PF4 support for DynamicPlugins - SharedModules + webpack configuration
https://github.com/openshift/console/blob/master/frontend/packages/console-shared/src/components/actions/menu/ActionMenuItem.tsx#L3 contains a reference to `@patternfly/react-core/deprecated`. In order to drop PF4 and adopt PF6, this reference needs to be removed.
This component was never finished and should be removed as it includes a reference to `@patternfly/react-core/deprecated`, which blocks the removal of PF4 and the adoption of PF6.
Feature Overview (aka. Goal Summary)
An elevator pitch (value statement) that describes the Feature in a clear, concise way. Complete during New status.
<your text here>
Goals (aka. expected user outcomes)
The observable functionality that the user now has as a result of receiving this feature. Include the anticipated primary user type/persona and which existing features, if any, will be expanded. Complete during New status.
<your text here>
Requirements (aka. Acceptance Criteria):
A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.
<enter general Feature acceptance here>
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
| Deployment considerations | List applicable specific needs (N/A = not applicable) |
| Self-managed, managed, or both | |
| Classic (standalone cluster) | |
| Hosted control planes | |
| Multi node, Compact (three node), or Single node (SNO), or all | |
| Connected / Restricted Network | |
| Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
| Operator compatibility | |
| Backport needed (list applicable versions) | |
| UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
| Other (please specify) |
Use Cases (Optional):
Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.
<your text here>
Questions to Answer (Optional):
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
<your text here>
Out of Scope
High-level list of items that are out of scope. Initial completion during Refinement status.
<your text here>
Background
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
<your text here>
Customer Considerations
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
<your text here>
Documentation Considerations
Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.
<your text here>
Interoperability Considerations
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
<your text here>
Epic Goal
- Base on user analytics many of customers switch back and fourth between perspectives, and average15 times per session.
- The following steps will be need:
- Surface all Dev specific Nav items in the Admin Console
- Disable the Dev perspective by default but allow admins to enable via console setting
- All quickstarts need to be updated to reflect the removal of the dev perspective
- Guided tour to show updated nav for merged perpspective
Why is this important?
- We need to alleviate this pain point and improve the overall user experience for our users.
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement details and documents.
- ...
Dependencies (internal and external)
- ...
Previous Work (Optional):
- …
Open questions::
- …
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
Description
As a developer, I do not want to maintain the code for a project already dead.
Acceptance Criteria
- Remove RHOAS plugin https://github.com/openshift/console/tree/master/frontend/packages/rhoas-plugin
- Remove RHOAS-catalog-source https://github.com/openshift/console/blob/master/frontend/packages/dev-console/integration-tests/testData/yamls/operator-installtion/RHOAS-catalog-source.yaml
- Check if there is dependencies in other package and fix it
Additional Details:
Feature Overview (aka. Goal Summary)
An elevator pitch (value statement) that describes the Feature in a clear, concise way. Complete during New status.
K8s 1.31 introduces VolumeAttributesClass as beta (code in external provisioner). We should make it available to customers as tech preview.
VolumeAttributesClass allows PVC to be modified after their creation and while attached. There is as vast number of parameters that can be updated but the most popular is to change the QoS values. Parameters that can be changed depend on the driver used.
Goals (aka. expected user outcomes)
The observable functionality that the user now has as a result of receiving this feature. Include the anticipated primary user type/persona and which existing features, if any, will be expanded. Complete during New status.
Productise VolumeAttributesClass as TP in anticipation for GA. Customer can start testing VolumeAttributesClass.
Requirements (aka. Acceptance Criteria):
A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.
- Disabled by default
- put it under TechPreviewNoUpgrade
- make sure VolumeAttributeClass object is available in beta APIs
- enable the feature in external-provisioner and external-resizer at least in AWS EBS CSI driver, check the other drivers.
- Add RBAC rules for these objects
- make sure we run its tests in one of TechPreviewNoUpgrade CI jobs (with hostpath CSI driver)
- reuse / add a job with AWS EBS CSI driver + tech preview.
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
| Deployment considerations | List applicable specific needs (N/A = not applicable) |
| Self-managed, managed, or both | both |
| Classic (standalone cluster) | yes |
| Hosted control planes | yes |
| Multi node, Compact (three node), or Single node (SNO), or all | all |
| Connected / Restricted Network | both |
| Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | all |
| Operator compatibility | N/A core storage |
| Backport needed (list applicable versions) | None |
| UI need (e.g. OpenShift Console, dynamic plugin, OCM) | TBD for TP |
| Other (please specify) | n/A |
Use Cases (Optional):
Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.
As an OCP user, I want to change parameters of my existing PVC such as the QoS attributes.
Questions to Answer (Optional):
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
<your text here>
Out of Scope
High-level list of items that are out of scope. Initial completion during Refinement status.
UI for TP
Background
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
There's been some limitations and complains on the fact that PVC attributes are sealed after their creation avoiding customers to update them. This is particularly impacting for a specific QoS is set and the volume requirements are changing.
Customer Considerations
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
Customer should not use it in production atm.
Documentation Considerations
Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.
Document VolumeAttributesClass creation and how to update PVC. Mention any limitation. Mention it's tech preview no upgrade. Add drivers support if needed.
Interoperability Considerations
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
Check which drivers support it for which parameters.
Epic Goal
Support upstream feature "VolumeAttributesClass" in OCP as Beta, i.e. test it and have docs for it.
Why is this important?
- We get this upstream feature through Kubernetes rebase. We should ensure it works well in OCP and we have docs for it.
Upstream links
- Enhancement issue: https://github.com/kubernetes/enhancements/issues/3751
- KEP: https://github.com/kubernetes/enhancements/pull/3780
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement details and documents.
- ...
Dependencies (internal and external)
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
- https://github.com/openshift/csi-operator/pull/314
- https://github.com/openshift/cluster-storage-operator/pull/550
- https://github.com/openshift/cluster-storage-operator/pull/549
- https://github.com/openshift/cluster-storage-operator/pull/545
- https://github.com/openshift/cluster-storage-operator/pull/542
Feature Overview (aka. Goal Summary)
A common concern with dealing with escalations/incidents in Managed OpenShift Hosted Control Planes is the resolution time incurred when the fix needs to be delivered in a component of the solution that ships within the OpenShift release payload. This is because OpenShift's release payloads:
- Have a hotfix process that is customer/support-exception targeted rather than fleet targeted
- Can take weeks to be available for Managed OpenShift
This feature seeks to provide mechanisms that put the upper time boundary in delivering such fixes to match the current HyperShift Operator <24h expectation
Goals (aka. expected user outcomes)
- Hosted Control Plane fixes are delivered through Konflux builds
- No additional upgrade edges
- Release specific
- Adequate, fleet representative, automated testing coverage
- Reduced human interaction
Requirements (aka. Acceptance Criteria):
A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.
- Overriding Hosted Control Plane components can be done automatically once the PRs are ready and the affected versions have been properly identified
- Managed OpenShift Hosted Clusters have their Control Planes fix applied without requiring customer intervention and without workload disruption beyond what might already be incurred because of the incident it is solving
- Fix can be promoted through integration, stage and production canary with a good degree of observability
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
| Deployment considerations | List applicable specific needs (N/A = not applicable) |
| Self-managed, managed, or both | managed (ROSA and ARO) |
| Classic (standalone cluster) | No |
| Hosted control planes | Yes |
| Multi node, Compact (three node), or Single node (SNO), or all | All supported ROSA/HCP topologies |
| Connected / Restricted Network | All supported ROSA/HCP topologies |
| Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | All supported ROSA/HCP topologies |
| Operator compatibility | CPO and Operators depending on it |
| Backport needed (list applicable versions) | TBD |
| UI need (e.g. OpenShift Console, dynamic plugin, OCM) | No |
| Other (please specify) | No |
Use Cases (Optional):
- Incident response when the engineering solution is partially or completely in the Hosted Control Plane side rather than in the HyperShift Operator
Out of Scope
- HyperShift Operator binary bundling
Background
Discussed previously during incident calls. Design discussion document
Customer Considerations
- Because the Managed Control Plane version does not change but it is overridden, customer visibility and impact should be limited as much as possible.
Documentation Considerations
SOP needs to be defined for:
- Requesting and approving the fleet wide fixes described above
- Building and delivering them
- Identifying clusters with deployed fleet wide fixes
Goal
- Have a Konflux build for every supported branch on every pull request / merge that modifies the Control Plane Operator
Why is this important?
- In order to build the Control Plane Operator images to be used for management cluster wide overrides.
- To be able to deliver managed Hosted Control Plane fixes to managed OpenShift with a similar SLO as the fixes for the HyperShift Operator.
Scenarios
- A PR that modifies the control plane in a supported branch is posted for a fix affecting managed OpenShift
Acceptance Criteria
- Dev - Konflux application and component per supported release
- Dev - SOPs for managing/troubleshooting the Konflux Application
- Dev - Release Plan that delivers to the appropriate AppSre production registry
- QE - HyperShift Operator versions that encode an override must be tested with the CPO Konflux builds that they make
Dependencies (internal and external)
- Konflux
Previous Work (Optional):
Open questions:
- Antoni Segura Puimedon How long or how many times should the CPO override be tested?
Done Checklist
- CI - CI is running, tests are automated and merged.
- DEV - Konflux App link: <link to Konflux App for CPO>
- DEV - SOP: <link to meaningful PR or GitHub Issue>
- QE - Test plan in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
The default PR posting and pushing tekton file that Konflux generates builds always. We can be more efficient with resources.
It is necessary to get the builds off of main for CPO overrides.
Feature Overview (aka. Goal Summary)
Rolling out new versions of HyperShift Operator or Hosted Control Plane components such as HyperShift's Control Plane Operator will no longer carry the possibility of triggering a Node rollout that can affect customer workloads running on those nodes
Goals (aka. expected user outcomes)
Customer Nodepool rollouts exhaustive cause list will be:
- Due to customer direct scaling up/down of the Nodepool
- Due to customer change of Hosted Cluster or Nodepool configuration that is documented to incur in a rollout
Customers will have visibility on rollouts that are pending so that they can effect a rollout of their affected nodepools at their earliest convenience
Requirements (aka. Acceptance Criteria):
- Observability:
- It must be possible to account for all Nodepools with pending rollouts
- It must be possible to identify all the Hosted Clusters with Nodepools with pending rollouts
- It must be possible for a customer to see that a Nodepool has pending rollouts
- Kubernetes expectations on resource reconciliation must be upheld
- Queued rollouts must survive HyperShift restarts
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
| Deployment considerations | List applicable specific needs (N/A = not applicable) |
| Self-managed, managed, or both | Managed (ROSA and ARO) |
| Classic (standalone cluster) | No |
| Hosted control planes | Yes |
| Multi node, Compact (three node), or Single node (SNO), or all | All supported Managed Hosted Control Plane topologies and configurations |
| Connected / Restricted Network | All supported Managed Hosted Control Plane topologies and configurations |
| Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | All supported Managed Hosted Control Plane topologies and configurations |
| Operator compatibility | N/A |
| Backport needed (list applicable versions) | All supported Managed Hosted Control Plane releases |
| UI need (e.g. OpenShift Console, dynamic plugin, OCM) | Yes. Console represtation of Nodepools/Machinepools should indicate pending rollouts and allow them to be triggered |
| Other (please specify) |
Use Cases (Optional):
- Managed OpenShift fix requires a Node rollout to be applied
Questions to Answer (Optional):
- Bryan Cox
- Should we explicitly call out the OCP versions in Backport needed (list applicable versions)?
- Antoni Segura Puimedon: Depends on what the supported OCP versions are in Managed OpenShift by the time this feature is delivered
- Is there a missing Goal of this feature, to force a rollout at a particular date? My line of thinking is what about CVE issues on the NodePool OCP version - are we giving them a warning like "hey you have a pending rollout because of a CVE; if you don't update the nodes yourself, we will on such & such date"?
- Antoni Segura Puimedon: Date based rollouts are out of scope (see its section).
- Should we explicitly call out the OCP versions in Backport needed (list applicable versions)?
- Juan Manuel Parrilla Madrid
- What’s the expectations on a regular customer nodePool upgrade? The change will be directly applied or queued following the last requested change?
- Antoni Segura Puimedon: Combinded single rollout.
- This only applies to NodePool changes or also would affect CP upgrades (thinking of OVN changes that could also affect the data plane)?
- Antoni Segura Puimedon: CP upgrades that would trigger Nodepool rollouts are in scope. OVN changes should only apply if CNO or its daemonsets are going to cause reboots
- How the customer will trigger the pending rollouts? An alert will trigger in the hosted cluster console?
- Antoni Segura Puimedon: I guess there are multiple options like scaling down and up and also adding some API to Nodepool
- I assume we will use a new status condition to reflect the current queue of pending rollouts, it’s that the case?.
- Antoni Segura Puimedon: That's a good guess. Hopefully we can represent all we want with it or we constrain ourselves to what it can express
- With "Queued rollouts must survive HyperShift restarts"... What kind of details we wanna store there (“there” should be the place to persist the changes queued), the order, the number of rollouts, the destination Hashes, more info…?
- What’s the expectations on a regular customer nodePool upgrade? The change will be directly applied or queued following the last requested change?
-
-
- Antoni Segura Puimedon: I'll leave that as an open question to refine
I'll If there are more than one change pending, we asume there will be more than one reboot?
- Antoni Segura Puimedon: I'll leave that as an open question to refine
-
Out of Scope
- Maintenance windows
- Queuing of rollouts on user actions (as that does not meet the Kubernetes reconciliation expectations and is better addressed at either the Cluster Service API level or better yet, at the customer automation side).
- Forced rollouts of pending updates on a certain date. That is something that should be handled at the Cluster Service level if there is desire to provide it.
Background
Past incidents with fixes to ignition generation resulting in rollout unexpected by the customer with workload impact
Customer Considerations
There should be an easy way to see, understand the content and trigger queued updates
Documentation Considerations
SOPs for the observability above
ROSA documentation for queued updates
Interoperability Considerations
ROSA/HCP and ARO/HCP
Goal
- Nodepools can hold off on performing a rollout of their nodes until said roll-out is triggered
Why is this important?
- Allows Managed Service to implement their policies or Node rollouts regardless if that is:
- Node maintenance windows
- Explicit user confirmation
Scenarios
- HyperShift upgrade fixes/adds reconciliation of some NodePool spec fields that result in ignition changes. Said upgrade should respect the Nodepool contract of not triggering node replacement/reboot without user/service intervention
- An important fix for the Nodes (potentially a CVE) comes up and the user wants to trigger a rollout at any given point in time to receive the benefits of the fix.
Acceptance Criteria
- Dev - Has a valid enhancement if necessary
- CI - MUST be running successfully with tests automated
- QE - covered in Polarion test plan and tests implemented
- SD QE - covered in SD promotion testing
- Release Technical Enablement - Must have TE slides
Open questions:
- What is the right UX to trigger the roll-out in these cases? Should Cluster service just scale down and up or can we offer a more powerful/convenient UX to the service operators.
Done Checklist
- CI - CI is running, tests are automated and merged.
- * Documentation and SRE enablement
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Enhancement merged: <link to meaningful PR or GitHub Issue>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
test to capture HO updates causing nodePool reboots
Feature Overview
The OpenShift IPsec implementation will be enhanced for a growing set of enterprise use cases, and for larger scale deployments.
Goals
The OpenShift IPsec implementation was originally built for purpose-driven use cases from telco NEPs, but was also be useful for a specific set of other customer use cases outside of that context. As customer adoption grew and it was adopted by some of the largest (by number of cluster nodes) deployments in the field, it became obvious that some redesign is necessary in order to continue to deliver enterprise-grade IPsec, for both East-West and North-South traffic, and for some of our most-demanding customer deployments.
Key enhancements include observability and blocked traffic across paths if IPsec encryption is not functioning properly.
Requirements
| Requirement | Notes | isMvp? |
|---|---|---|
| CI - MUST be running successfully with test automation | This is a requirement for ALL features. | YES |
| Release Technical Enablement | Provide necessary release enablement details and documents. | YES |
Questions to answer…
Out of Scope
- Configuration of external-to-cluster IPsec endpoints for N-S IPsec.
Background, and strategic fit
The OpenShift IPsec feature is fundamental to customer deployments for ensuring that all traffic between cluster nodes (East-West) and between cluster nodes and external-to-the-cluster entities that also are configured for IPsec (North-South) is encrypted by default. This encryption must scale to the largest of deployments.
Assumptions
Customer Considerations
- Customers require the option to use their own certificates or CA for IPsec.
- Customers require observability of configuration (e.g. is the IPsec tunnel up and passing traffic)
- If the IPsec tunnel is not up or otherwise functioning, traffic across the intended-to-be-encrypted network path should be blocked.
Documentation Considerations
Questions to be addressed:
- What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
- Does this feature have doc impact?
- New Content, Updates to existing content, Release Note, or No Doc Impact
- If unsure and no Technical Writer is available, please contact Content Strategy.
- What concepts do customers need to understand to be successful in [action]?
- How do we expect customers will use the feature? For what purpose(s)?
- What reference material might a customer want/need to complete [action]?
- Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
- What is the doc impact (New Content, Updates to existing content, or Release Note)?
While running IPsec e2e tests in the CI, the data plane traffic is not flowing with desired traffic type esp or udp. For example, ipsec mode external, the traffic type seems to seen as esp for EW traffic, but it's supposed to be geneve (udp) taffic.
This issue was reproducible on a local cluster after many attempts and noticed ipsec states are not cleanup on the node which is a residue from previous test run with ipsec full mode.
[peri@sdn-09 origin]$ kubectl get networks.operator.openshift.io cluster -o yaml
apiVersion: operator.openshift.io/v1
kind: Network
metadata:
creationTimestamp: "2024-05-13T18:55:57Z"
generation: 1362
name: cluster
resourceVersion: "593827"
uid: 10f804c9-da46-41ee-91d5-37aff920bee4
spec:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
defaultNetwork:
ovnKubernetesConfig:
egressIPConfig: {}
gatewayConfig:
ipv4: {}
ipv6: {}
routingViaHost: false
genevePort: 6081
ipsecConfig:
mode: External
mtu: 1400
policyAuditConfig:
destination: "null"
maxFileSize: 50
maxLogFiles: 5
rateLimit: 20
syslogFacility: local0
type: OVNKubernetes
deployKubeProxy: false
disableMultiNetwork: false
disableNetworkDiagnostics: false
logLevel: Normal
managementState: Managed
observedConfig: null
operatorLogLevel: Normal
serviceNetwork:
- 172.30.0.0/16
unsupportedConfigOverrides: null
useMultiNetworkPolicy: false
status:
conditions:
- lastTransitionTime: "2024-05-13T18:55:57Z"
status: "False"
type: ManagementStateDegraded
- lastTransitionTime: "2024-05-14T10:13:12Z"
status: "False"
type: Degraded
- lastTransitionTime: "2024-05-13T18:55:57Z"
status: "True"
type: Upgradeable
- lastTransitionTime: "2024-05-14T11:50:26Z"
status: "False"
type: Progressing
- lastTransitionTime: "2024-05-13T18:57:13Z"
status: "True"
type: Available
readyReplicas: 0
version: 4.16.0-0.nightly-2024-05-08-222442
[peri@sdn-09 origin]$ oc debug node/worker-0
Starting pod/worker-0-debug-k6nlm ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.111.23
If you don't see a command prompt, try pressing enter.
sh-5.1# chroot /host
sh-5.1# toolbox
Checking if there is a newer version of registry.redhat.io/rhel9/support-tools available...
Container 'toolbox-root' already exists. Trying to start...
(To remove the container and start with a fresh toolbox, run: sudo podman rm 'toolbox-root')
toolbox-root
Container started successfully. To exit, type 'exit'.
[root@worker-0 /]# tcpdump -i enp2s0 -c 1 -v --direction=out esp and src 192.168.111.23 and dst 192.168.111.24
dropped privs to tcpdump
tcpdump: listening on enp2s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
16:07:01.854214 IP (tos 0x0, ttl 64, id 20451, offset 0, flags [DF], proto ESP (50), length 152)
worker-0 > worker-1: ESP(spi=0x52cc9c8d,seq=0xe1c5c), length 132
1 packet captured
6 packets received by filter
0 packets dropped by kernel
[root@worker-0 /]# exit
exit
sh-5.1# ipsec whack --trafficstatus
006 #20: "ovn-1184d9-0-in-1", type=ESP, add_time=1715687134, inBytes=206148172, outBytes=0, maxBytes=2^63B, id='@1184d960-3211-45c4-a482-d7b6fe995446'
006 #19: "ovn-1184d9-0-out-1", type=ESP, add_time=1715687112, inBytes=0, outBytes=40269835, maxBytes=2^63B, id='@1184d960-3211-45c4-a482-d7b6fe995446'
006 #27: "ovn-185198-0-in-1", type=ESP, add_time=1715687419, inBytes=71406656, outBytes=0, maxBytes=2^63B, id='@185198f6-7dde-4e9b-b2aa-52439d2beef5'
006 #26: "ovn-185198-0-out-1", type=ESP, add_time=1715687401, inBytes=0, outBytes=17201159, maxBytes=2^63B, id='@185198f6-7dde-4e9b-b2aa-52439d2beef5'
006 #14: "ovn-922aca-0-in-1", type=ESP, add_time=1715687004, inBytes=116384250, outBytes=0, maxBytes=2^63B, id='@922aca42-b893-496e-bb9b-0310884f4cc1'
006 #13: "ovn-922aca-0-out-1", type=ESP, add_time=1715686986, inBytes=0, outBytes=986900228, maxBytes=2^63B, id='@922aca42-b893-496e-bb9b-0310884f4cc1'
006 #6: "ovn-f72f26-0-in-1", type=ESP, add_time=1715686855, inBytes=115781441, outBytes=98, maxBytes=2^63B, id='@f72f2622-e7dc-414e-8369-6013752ea15b'
006 #5: "ovn-f72f26-0-out-1", type=ESP, add_time=1715686833, inBytes=9320, outBytes=29002449, maxBytes=2^63B, id='@f72f2622-e7dc-414e-8369-6013752ea15b'
sh-5.1# ip xfrm state; echo ' '; ip xfrm policy
src 192.168.111.21 dst 192.168.111.23
proto esp spi 0x7f7ddcf5 reqid 16413 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x6158d9a0f4a28598500e15f81a40ef715502b37ecf979feb11bbc488479c8804598011ee 128
lastused 2024-05-14 16:07:11
anti-replay esn context:
seq-hi 0x0, seq 0x18564, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
ffffffff ffffffff ffffffff ffffffff
sel src 192.168.111.21/32 dst 192.168.111.23/32 proto udp dport 6081
src 192.168.111.23 dst 192.168.111.21
proto esp spi 0xda57e42e reqid 16413 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x810bebecef77951ae8bb9a46cf53a348a24266df8b57bf2c88d4f23244eb3875e88cc796 128
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.23/32 dst 192.168.111.21/32 proto udp sport 6081
src 192.168.111.21 dst 192.168.111.23
proto esp spi 0xf84f2fcf reqid 16417 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x0f242efb072699a0f061d4c941d1bb9d4eb7357b136db85a0165c3b3979e27b00ff20ac7 128
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.21/32 dst 192.168.111.23/32 proto udp sport 6081
src 192.168.111.23 dst 192.168.111.21
proto esp spi 0x9523c6ca reqid 16417 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0xe075d39b6e53c033f5225f8be48efe537c3ba605cee2f5f5f3bb1cf16b6c53182ecf35f7 128
lastused 2024-05-14 16:07:11
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x10fb2
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.23/32 dst 192.168.111.21/32 proto udp dport 6081
src 192.168.111.20 dst 192.168.111.23
proto esp spi 0x459d8516 reqid 16397 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0xee778e6db2ce83fa24da3b18e028451bbfcf4259513bca21db832c3023e238a6b55fdacc 128
lastused 2024-05-14 16:07:13
anti-replay esn context:
seq-hi 0x0, seq 0x3ec45, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
ffffffff ffffffff ffffffff ffffffff
sel src 192.168.111.20/32 dst 192.168.111.23/32 proto udp dport 6081
src 192.168.111.23 dst 192.168.111.20
proto esp spi 0x3142f53a reqid 16397 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x6238fea6dffdd36cbb909f6aab48425ba6e38f9d32edfa0c1e0fc6af8d4e3a5c11b5dfd1 128
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.23/32 dst 192.168.111.20/32 proto udp sport 6081
src 192.168.111.20 dst 192.168.111.23
proto esp spi 0xeda1ccb9 reqid 16401 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0xef84a90993bd71df9c97db940803ad31c6f7d2e72a367a1ec55b4798879818a6341c38b6 128
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.20/32 dst 192.168.111.23/32 proto udp sport 6081
src 192.168.111.23 dst 192.168.111.20
proto esp spi 0x02c3c0dd reqid 16401 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x858ab7326e54b6d888825118724de5f0c0ad772be2b39133c272920c2cceb2f716d02754 128
lastused 2024-05-14 16:07:13
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x26f8e
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.23/32 dst 192.168.111.20/32 proto udp dport 6081
src 192.168.111.24 dst 192.168.111.23
proto esp spi 0xc9535b47 reqid 16405 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0xd7a83ff4bd6e7704562c597810d509c3cdd4e208daabf2ec074d109748fd1647ab2eff9d 128
lastused 2024-05-14 16:07:14
anti-replay esn context:
seq-hi 0x0, seq 0x53d4c, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
ffffffff ffffffff ffffffff ffffffff
sel src 192.168.111.24/32 dst 192.168.111.23/32 proto udp dport 6081
src 192.168.111.23 dst 192.168.111.24
proto esp spi 0xb66203c8 reqid 16405 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0xc207001a7f1ed7f114b3e327308ddbddc36de5272a11fe0661d03eaecc84b6761c7ec9c4 128
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.23/32 dst 192.168.111.24/32 proto udp sport 6081
src 192.168.111.24 dst 192.168.111.23
proto esp spi 0x2e4d4deb reqid 16409 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x91e399d83aa1c2626424b502d4b8dae07d4a170f7ef39f8d1baca8e92b8a1dee210e2502 128
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.24/32 dst 192.168.111.23/32 proto udp sport 6081
src 192.168.111.23 dst 192.168.111.24
proto esp spi 0x52cc9c8d reqid 16409 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0xb605451f32f5dd7a113cae16e6f1509270c286d67265da2ad14634abccf6c90f907e5c00 128
lastused 2024-05-14 16:07:14
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0xe2735
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.23/32 dst 192.168.111.24/32 proto udp dport 6081
src 192.168.111.22 dst 192.168.111.23
proto esp spi 0x973119c3 reqid 16389 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x87d13e67b948454671fb8463ec0cd4d9c38e5e2dd7f97cbb8f88b50d4965fb1f21b36199 128
lastused 2024-05-14 16:07:14
anti-replay esn context:
seq-hi 0x0, seq 0x2af9a, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
ffffffff ffffffff ffffffff ffffffff
sel src 192.168.111.22/32 dst 192.168.111.23/32 proto udp dport 6081
src 192.168.111.23 dst 192.168.111.22
proto esp spi 0x4c3580ff reqid 16389 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x2c09750f51e86d60647a60e15606f8b312036639f8de2d7e49e733cda105b920baade029 128
lastused 2024-05-14 14:36:43
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x1
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.23/32 dst 192.168.111.22/32 proto udp sport 6081
src 192.168.111.22 dst 192.168.111.23
proto esp spi 0xa3e469dc reqid 16393 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x1d5c5c232e6fd4b72f3dad68e8a4d523cbd297f463c53602fad429d12c0211d97ae26f47 128
lastused 2024-05-14 14:18:42
anti-replay esn context:
seq-hi 0x0, seq 0xb, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
00000000 00000000 00000000 000007ff
sel src 192.168.111.22/32 dst 192.168.111.23/32 proto udp sport 6081
src 192.168.111.23 dst 192.168.111.22
proto esp spi 0xdee8476f reqid 16393 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x5895025ce5b192a7854091841c73c8e29e7e302f61becfa3feb44d071ac5c64ce54f5083 128
lastused 2024-05-14 16:07:14
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x1f1a3
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.23/32 dst 192.168.111.22/32 proto udp dport 6081
src 192.168.111.23/32 dst 192.168.111.21/32 proto udp sport 6081
dir out priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16413 mode transport
src 192.168.111.21/32 dst 192.168.111.23/32 proto udp dport 6081
dir in priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16413 mode transport
src 192.168.111.23/32 dst 192.168.111.21/32 proto udp dport 6081
dir out priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16417 mode transport
src 192.168.111.21/32 dst 192.168.111.23/32 proto udp sport 6081
dir in priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16417 mode transport
src 192.168.111.23/32 dst 192.168.111.20/32 proto udp sport 6081
dir out priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16397 mode transport
src 192.168.111.20/32 dst 192.168.111.23/32 proto udp dport 6081
dir in priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16397 mode transport
src 192.168.111.23/32 dst 192.168.111.20/32 proto udp dport 6081
dir out priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16401 mode transport
src 192.168.111.20/32 dst 192.168.111.23/32 proto udp sport 6081
dir in priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16401 mode transport
src 192.168.111.23/32 dst 192.168.111.24/32 proto udp sport 6081
dir out priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16405 mode transport
src 192.168.111.24/32 dst 192.168.111.23/32 proto udp dport 6081
dir in priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16405 mode transport
src 192.168.111.23/32 dst 192.168.111.24/32 proto udp dport 6081
dir out priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16409 mode transport
src 192.168.111.24/32 dst 192.168.111.23/32 proto udp sport 6081
dir in priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16409 mode transport
src 192.168.111.23/32 dst 192.168.111.22/32 proto udp sport 6081
dir out priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16389 mode transport
src 192.168.111.22/32 dst 192.168.111.23/32 proto udp dport 6081
dir in priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16389 mode transport
src 192.168.111.23/32 dst 192.168.111.22/32 proto udp dport 6081
dir out priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16393 mode transport
src 192.168.111.22/32 dst 192.168.111.23/32 proto udp sport 6081
dir in priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16393 mode transport
src ::/0 dst ::/0
socket out priority 0 ptype main
src ::/0 dst ::/0
socket in priority 0 ptype main
src ::/0 dst ::/0
socket out priority 0 ptype main
src ::/0 dst ::/0
socket in priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0 ptype main
src ::/0 dst ::/0 proto ipv6-icmp type 135
dir out priority 1 ptype main
src ::/0 dst ::/0 proto ipv6-icmp type 135
dir fwd priority 1 ptype main
src ::/0 dst ::/0 proto ipv6-icmp type 135
dir in priority 1 ptype main
src ::/0 dst ::/0 proto ipv6-icmp type 136
dir out priority 1 ptype main
src ::/0 dst ::/0 proto ipv6-icmp type 136
dir fwd priority 1 ptype main
src ::/0 dst ::/0 proto ipv6-icmp type 136
dir in priority 1 ptype main
sh-5.1# cat /etc/ipsec.conf
# /etc/ipsec.conf - Libreswan 4.0 configuration file
#
# see 'man ipsec.conf' and 'man pluto' for more information
#
# For example configurations and documentation, see https://libreswan.org/wiki/
config setup
# If logfile= is unset, syslog is used to send log messages too.
# Note that on busy VPN servers, the amount of logging can trigger
# syslogd (or journald) to rate limit messages.
#logfile=/var/log/pluto.log
#
# Debugging should only be used to find bugs, not configuration issues!
# "base" regular debug, "tmi" is excessive 
and "private" will log
# sensitive key material (not available in FIPS mode). The "cpu-usage"
# value logs timing information and should not be used with other
# debug options as it will defeat getting accurate timing information.
# Default is "none"
# plutodebug="base"
# plutodebug="tmi"
#plutodebug="none"
#
# Some machines use a DNS resolver on localhost with broken DNSSEC
# support. This can be tested using the command:
# dig +dnssec DNSnameOfRemoteServer
# If that fails but omitting '+dnssec' works, the system's resolver is
# broken and you might need to disable DNSSEC.
# dnssec-enable=no
#
# To enable IKE and IPsec over TCP for VPN server. Requires at least
# Linux 5.7 kernel or a kernel with TCP backport (like RHEL8 4.18.0-291)
# listen-tcp=yes
# To enable IKE and IPsec over TCP for VPN client, also specify
# tcp-remote-port=4500 in the client's conn section.
# if it exists, include system wide crypto-policy defaults
include /etc/crypto-policies/back-ends/libreswan.config
# It is best to add your IPsec connections as separate files
# in /etc/ipsec.d/
include /etc/ipsec.d/*.conf
sh-5.1# cat /etc/ipsec.d/openshift.conf
# Generated by ovs-monitor-ipsec...do not modify by hand!
config setup
uniqueids=yes
conn %default
keyingtries=%forever
type=transport
auto=route
ike=aes_gcm256-sha2_256
esp=aes_gcm256
ikev2=insist
conn ovn-f72f26-0-in-1
left=192.168.111.23
right=192.168.111.22
leftid=@cf36db5c-5c54-4329-9141-b83679b18ecc
rightid=@f72f2622-e7dc-414e-8369-6013752ea15b
leftcert="ovs_certkey_cf36db5c-5c54-4329-9141-b83679b18ecc"
leftrsasigkey=%cert
rightca=%same
leftprotoport=udp/6081
rightprotoport=udp
conn ovn-f72f26-0-out-1
left=192.168.111.23
right=192.168.111.22
leftid=@cf36db5c-5c54-4329-9141-b83679b18ecc
rightid=@f72f2622-e7dc-414e-8369-6013752ea15b
leftcert="ovs_certkey_cf36db5c-5c54-4329-9141-b83679b18ecc"
leftrsasigkey=%cert
rightca=%same
leftprotoport=udp
rightprotoport=udp/6081
conn ovn-1184d9-0-in-1
left=192.168.111.23
right=192.168.111.20
leftid=@cf36db5c-5c54-4329-9141-b83679b18ecc
rightid=@1184d960-3211-45c4-a482-d7b6fe995446
leftcert="ovs_certkey_cf36db5c-5c54-4329-9141-b83679b18ecc"
leftrsasigkey=%cert
rightca=%same
leftprotoport=udp/6081
rightprotoport=udp
conn ovn-1184d9-0-out-1
left=192.168.111.23
right=192.168.111.20
leftid=@cf36db5c-5c54-4329-9141-b83679b18ecc
rightid=@1184d960-3211-45c4-a482-d7b6fe995446
leftcert="ovs_certkey_cf36db5c-5c54-4329-9141-b83679b18ecc"
leftrsasigkey=%cert
rightca=%same
leftprotoport=udp
rightprotoport=udp/6081
conn ovn-922aca-0-in-1
left=192.168.111.23
right=192.168.111.24
leftid=@cf36db5c-5c54-4329-9141-b83679b18ecc
rightid=@922aca42-b893-496e-bb9b-0310884f4cc1
leftcert="ovs_certkey_cf36db5c-5c54-4329-9141-b83679b18ecc"
leftrsasigkey=%cert
rightca=%same
leftprotoport=udp/6081
rightprotoport=udp
conn ovn-922aca-0-out-1
left=192.168.111.23
right=192.168.111.24
leftid=@cf36db5c-5c54-4329-9141-b83679b18ecc
rightid=@922aca42-b893-496e-bb9b-0310884f4cc1
leftcert="ovs_certkey_cf36db5c-5c54-4329-9141-b83679b18ecc"
leftrsasigkey=%cert
rightca=%same
leftprotoport=udp
rightprotoport=udp/6081
conn ovn-185198-0-in-1
left=192.168.111.23
right=192.168.111.21
leftid=@cf36db5c-5c54-4329-9141-b83679b18ecc
rightid=@185198f6-7dde-4e9b-b2aa-52439d2beef5
leftcert="ovs_certkey_cf36db5c-5c54-4329-9141-b83679b18ecc"
leftrsasigkey=%cert
rightca=%same
leftprotoport=udp/6081
rightprotoport=udp
conn ovn-185198-0-out-1
left=192.168.111.23
right=192.168.111.21
leftid=@cf36db5c-5c54-4329-9141-b83679b18ecc
rightid=@185198f6-7dde-4e9b-b2aa-52439d2beef5
leftcert="ovs_certkey_cf36db5c-5c54-4329-9141-b83679b18ecc"
leftrsasigkey=%cert
rightca=%same
leftprotoport=udp
rightprotoport=udp/6081
sh-5.1#
The e2e-aws-ovn-ipsec-upgrade job is currently an optional job and always_run: false because the job not reliable and success rate is so low. This must be made as mandatory CI lane after fixing its relevant issues.
Feature Overview (aka. Goal Summary)
Ability to scale the amount of worker nodes in a Hosted Cluster in ARO/HCP to 500 nodes.
Goals (aka. expected user outcomes)
- Customers will be able to have their managed Hosted Control Planes scale up to 500 concurrent worker nodes without control plane degradation
Requirements (aka. Acceptance Criteria):
- HCP AKS topology does not use request serving nodes
- It is possible to scale down and up 1..500 nodes
- There's no downtime or API availability loss during scale up.
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
| Deployment considerations | List applicable specific needs (N/A = not applicable) |
| Self-managed, managed, or both | managed |
| Classic (standalone cluster) | No |
| Hosted control planes | Yes |
| Multi node, Compact (three node), or Single node (SNO), or all | Multi node |
| Connected / Restricted Network | Connected |
| Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | All targeted archs for ARO/HCP |
| Operator compatibility | N/A |
| Backport needed (list applicable versions) | Yes. Must support 4.17 Hosted Clusters |
| UI need (e.g. OpenShift Console, dynamic plugin, OCM) | No |
| Other (please specify) |
Use Cases (Optional):
Customer workloads require up to 500 nodes in a single hosted cluster.
Background
OCPSTRAT-1251 saw a big amount of work to enable customers to reach 500 nodes in the special ROSA/HCP topology.
Customer Considerations
500 nodes is the sum of all the maximum machines for the combined machinepools of an ARO/HCP hosted cluster
Documentation Considerations
There should be documentation/SOPs for spotting and reacting to possible noisy neighbor patterns.
Interoperability Considerations
ARO/HCP
Goal
- Scaling the hosted control planes in ARO/HCP to support hosted clusters with 500 worker nodes in total (the sum of of workers in all the Nodepools)
Why is this important?
- To support workloads with a high node count without causing Control Plane instability.
Scenarios
- Scaling up one or more NodePools to 500 worker nodes total and keep KAS availability
- Scaling down from a 500 worker nodes total count while keeping KAS SLO
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- QE - covered in Polarion test plan and tests implemented
- Perf&Scale - verify the implementation and issue recommendation
Previous Work (Optional):
Open questions:
Done Checklist
- CI - CI is running, tests are automated and merged.
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
Background
The ClusterSizingConfiguration resource that was introduced for ROSA allows setting various effects based on the size of a HostedCluster. For a service like ARO where no dedicated request serving nodes are needed, it should be sufficient to apply the effects (as annotations) on HostedClusters based on their size label. The controller/scheduler should simply reconcile HostedClusters and apply the corresponding annotations based on the size label assigned to the given HostedCluster.
User Story:
As a service provider, I want to be able to:
- Install hypershift and specify that I want a simple scheduler. Create a ClusterSizingConfiguration, and have the corresponding effects be applied to HostedClusters in that management cluster.
so that I can
- Schedule hosted cluster control plane workloads according to Hosted Cluster size.
Epic Goal
- Review design and development PRs that require feedback from NE team.
Why is this important?
- Customer requires certificates to be managed by cert-manager on configured/newly added routes.
Acceptance Criteria
- All PRs are reviewed and merged.
Dependencies (internal and external)
- CFE team dependency for addressing review suggestions.
Done Checklist
- DEV - All related PRs are merged.
Placeholder to track GA work for CFE-811
Update API godoc to document that manual intervention is required for using .spec.tls.externalCertificate. Something simple like: "The Router service account needs to be granted with read-only access to this secret, please refer to openshift docs for additional details."
Feature Overview (aka. Goal Summary)
An elevator pitch (value statement) that describes the Feature in a clear, concise way. Complete during New status.
<your text here>
Goals (aka. expected user outcomes)
The observable functionality that the user now has as a result of receiving this feature. Include the anticipated primary user type/persona and which existing features, if any, will be expanded. Complete during New status.
<your text here>
Requirements (aka. Acceptance Criteria):
A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.
<enter general Feature acceptance here>
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
| Deployment considerations | List applicable specific needs (N/A = not applicable) |
| Self-managed, managed, or both | |
| Classic (standalone cluster) | |
| Hosted control planes | |
| Multi node, Compact (three node), or Single node (SNO), or all | |
| Connected / Restricted Network | |
| Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
| Operator compatibility | |
| Backport needed (list applicable versions) | |
| UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
| Other (please specify) |
Use Cases (Optional):
Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.
<your text here>
Questions to Answer (Optional):
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
<your text here>
Out of Scope
High-level list of items that are out of scope. Initial completion during Refinement status.
<your text here>
Background
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
<your text here>
Customer Considerations
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
<your text here>
Documentation Considerations
Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.
<your text here>
Interoperability Considerations
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
<your text here>
link back to OCPSTRAT-1644 somehow
Epic Goal*
What is our purpose in implementing this? What new capability will be available to customers?
Why is this important? (mandatory)
What are the benefits to the customer or Red Hat? Does it improve security, performance, supportability, etc? Why is work a priority?
Scenarios (mandatory)
Provide details for user scenarios including actions to be performed, platform specifications, and user personas.
Dependencies (internal and external) (mandatory)
What items must be delivered by other teams/groups to enable delivery of this epic.
Contributing Teams(and contacts) (mandatory)
Our expectation is that teams would modify the list below to fit the epic. Some epics may not need all the default groups but what is included here should accurately reflect who will be involved in delivering the epic.
- Development -
- Documentation -
- QE -
- PX -
- Others -
Acceptance Criteria (optional)
Provide some (testable) examples of how we will know if we have achieved the epic goal.
Drawbacks or Risk (optional)
Reasons we should consider NOT doing this such as: limited audience for the feature, feature will be superseded by other work that is planned, resulting feature will introduce substantial administrative complexity or user confusion, etc.
Done - Checklist (mandatory)
The following points apply to all epics and are what the OpenShift team believes are the minimum set of criteria that epics should meet for us to consider them potentially shippable. We request that epic owners modify this list to reflect the work to be completed in order to produce something that is potentially shippable.
- CI Testing - Basic e2e automationTests are merged and completing successfully
- Documentation - Content development is complete.
- QE - Test scenarios are written and executed successfully.
- Technical Enablement - Slides are complete (if requested by PLM)
- Engineering Stories Merged
- All associated work items with the Epic are closed
- Epic status should be “Release Pending”
Epic Goal
- The goal of this epic is to upgrade all OpenShift and Kubernetes components that CCO uses to v1.32 which keeps it on par with rest of the OpenShift components and the underlying cluster version.
Why is this important?
- To make sure that Hive imports of other OpenShift components do not break when those rebase
- To avoid breaking other OpenShift components importing from CCO.
- To pick up upstream improvements
Acceptance Criteria
- CI - MUST be running successfully with tests automated
Dependencies (internal and external)
- Kubernetes 1.32 is released
Previous Work (Optional):
- Similar previous epic
CCO-595
Done Checklist
- CI - CI is running, tests are automated and merged.
As a developer, I want to upgrade the Kubernetes dependencies to 1.32
- to ensure compatibility with the OpenShift cluster
Epic Goal*
Drive the technical part of the Kubernetes 1.32 upgrade, including rebasing openshift/kubernetes repository and coordination across OpenShift organization to get e2e tests green for the OCP release.
Why is this important? (mandatory)
OpenShift 4.19 cannot be released without Kubernetes 1.32
Scenarios (mandatory)
Dependencies (internal and external) (mandatory)
What items must be delivered by other teams/groups to enable delivery of this epic.
Contributing Teams(and contacts) (mandatory)
Our expectation is that teams would modify the list below to fit the epic. Some epics may not need all the default groups but what is included here should accurately reflect who will be involved in delivering the epic.
- Development -
- Documentation -
- QE -
- PX -
- Others -
Acceptance Criteria (optional)
Provide some (testable) examples of how we will know if we have achieved the epic goal.
Drawbacks or Risk (optional)
Reasons we should consider NOT doing this such as: limited audience for the feature, feature will be superseded by other work that is planned, resulting feature will introduce substantial administrative complexity or user confusion, etc.
Done - Checklist (mandatory)
The following points apply to all epics and are what the OpenShift team believes are the minimum set of criteria that epics should meet for us to consider them potentially shippable. We request that epic owners modify this list to reflect the work to be completed in order to produce something that is potentially shippable.
- CI Testing - Basic e2e automationTests are merged and completing successfully
- Documentation - Content development is complete.
- QE - Test scenarios are written and executed successfully.
- Technical Enablement - Slides are complete (if requested by PLM)
- Engineering Stories Merged
- All associated work items with the Epic are closed
- Epic status should be “Release Pending”
Slack Discussion Channel - https://redhat.enterprise.slack.com/archives/C07V32J0YKF
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
Epic Goal
- Cluster Infrastructure owned components should be running on Kubernetes 1.32
- This includes
- The cluster autoscaler (+operator)
- Machine API operator
- Machine API controllers for:
- AWS
- Azure
- GCP
- vSphere
- OpenStack
- IBM
- Nutanix
- Machine API controllers for:
- Cloud Controller Manager Operator
- Cloud controller managers for:
- AWS
- Azure
- GCP
- vSphere
- OpenStack
- IBM
- Nutanix
- Cloud controller managers for:
- Cluster Machine Approver
- Cluster API Actuator Package
- Control Plane Machine Set Operator
Why is this important?
- ...
Scenarios
- ...
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement details and documents.
- ...
Dependencies (internal and external)
- ...
Previous Work (Optional):
- ...
Open questions::
- ...
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
To align with the 4.19 release, dependencies need to be updated to 1.31. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.31. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.31. This should be done by rebasing/updating as appropriate for the repository
As a customer of self managed OpenShift or an SRE managing a fleet of OpenShift clusters I should be able to determine the progress and state of an OCP upgrade and only be alerted if the cluster is unable to progress. Support a cli-status command and status-API which can be used by cluster-admin to monitor the progress. status command/API should also contain data to alert users about potential issues which can make the updates problematic.
Feature Overview (aka. Goal Summary)
Here are common update improvements from customer interactions on Update experience
- Show nodes where pod draining is taking more time.
Customers have to dig deeper often to find the nodes for further debugging.
The ask has been to bubble up this on the update progress window. - oc update status ?
From the UI we can see the progress of the update. From oc cli we can see this from "oc get cvo"
But the ask is to show more details in a human-readable format.
Know where the update has stopped. Consider adding at what run level it has stopped.
oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.12.0 True True 16s Working towards 4.12.4: 9 of 829 done (1% complete)
Documentation Considerations
Update docs for UX and CLI changes
Reference : https://docs.google.com/presentation/d/1cwxN30uno_9O8RayvcIAe8Owlds-5Wjr970sDxn7hPg/edit#slide=id.g2a2b8de8edb_0_22
Epic Goal*
Add a new command `oc adm upgrade status` command which is backed by an API. Please find the mock output of the command output attached in this card.
Why is this important? (mandatory)
- From the UI we can see the progress of the update. Using OC CLI we can see some of the information using "oc get clusterversion" but the output is not readable and it is a lot of extra information to process.
- Customer as asking us to show more details in a human-readable format as well provide an API which they can use for automation.
Scenarios (mandatory)
Provide details for user scenarios including actions to be performed, platform specifications, and user personas.
Dependencies (internal and external) (mandatory)
What items must be delivered by other teams/groups to enable delivery of this epic.
Contributing Teams(and contacts) (mandatory)
Our expectation is that teams would modify the list below to fit the epic. Some epics may not need all the default groups but what is included here should accurately reflect who will be involved in delivering the epic.
- Development -
- Documentation -
- QE -
- PX -
- Others -
Acceptance Criteria (optional)
Provide some (testable) examples of how we will know if we have achieved the epic goal.
Drawbacks or Risk (optional)
Reasons we should consider NOT doing this such as: limited audience for the feature, feature will be superseded by other work that is planned, resulting feature will introduce substantial administrative complexity or user confusion, etc.
Done - Checklist (mandatory)
The following points apply to all epics and are what the OpenShift team believes are the minimum set of criteria that epics should meet for us to consider them potentially shippable. We request that epic owners modify this list to reflect the work to be completed in order to produce something that is potentially shippable.
- CI Testing - Tests are merged and completing successfully
- Documentation - Content development is complete.
- QE - Test scenarios are written and executed successfully.
- Technical Enablement - Slides are complete (if requested by PLM)
- Other
On the call to discuss oc adm upgrade status roadmap to server side-implementation (notes) we agreed on basic architectural direction and we can starting moving in that direction:
- status API will be backed by a new controller
- new controller will be a separate binary but delivered in the CVO image (=release payload) to avoid needing new ClusterOperator
- new operator will maintain a singleton resource of a new UpgradeStatus CRD - this is the interface to the consumers
Let's start building this controller; we can implement the controller perform the functionality currently present in the client, and just expose it through an API. I am not sure how to deal with the fact that we won't have the API merged until it merges into o/api, which is not soon. Maybe we can implement the controller over a temporary fork of o/api and rely on manually inserting the CRD into the cluster when we test the functionality? Not sure.
We need to avoid committing to implementation details and investing effort into things that may change though.
Definition of Done
CVO repository has a new controller (a new cluster-version-operator cobra subcommand sounds like a good option; an alternative would a completely new binary included in CVO image)- The payload contains manifests (SA, RBAC, Deployment) to deploy the new controller when DevPreviewNoUpgrade feature set is enabled (but not TechPreview)
- The controller uses properly scoped minimal necessary RBAC through a dedicated SA
- The controller will react on ClusterVersion changes in the cluster through an informer
- The controller will maintain a single ClusterVersion status insight as specified by the Update Health API Draft
- The controller does not need to maintain all fields precisely: it can use placeholders or even ignore fields that need more complicated logic over more resources (estimated finish, completion, assessment)
- The controller will publish the serialized CV status insight (in yaml or json) through a ConfigMap (this is a provisionary measure until we can get the necessary API and client-go changes merged) under a key that identifies the kube resource ("cv-version")
- The controller only includes the necessary types code from o/api PR together with the necessary generated code (like deepcopy). These local types will need to be replaced with the types eventually merged into o/api and vendored to o/cluster-version-operator
Testing notes
This card only brings a skeleton of the desired functionality to the DevPreviewNoUpgrade feature set. Its purpose is mainly to enable further development by putting the necessary bits in place so that we can start developing more functionality. There's not much point in automating testing of any of the functionality in this card, but it should be useful to start getting familiar with how the new controller is deployed and what are its concepts.
For seeing the new controller in action:
1. Launch a cluster that includes both the code and manifests. As of Nov 11, #1107 is not yet merged so you need to use launch 4.18,openshift/cluster-version-operator#1107 aws,no-spot
2. Enable the DevPreviewNoUpgrade feature set. CVO will restart and will deploy all functionality gated by this feature set, including the USC. It can take a bit of time, ~10-15m should be enough though.
3. Eventually, you should be able to see the new openshift-update-status-controller Namespace created in the cluster
4. You should be able to see a update-status-controller Deployment in that namespace
5. That Deployment should have one replica running and being ready. It should not crashloop or anything like that. You can inspect its logs for obvious failures and such. At this point, its log should, near its end, say something like "the ConfigMap does not exist so doing nothing"
6. Create the ConfigMap that mimics the future API (make sure to create it in the openshift-update-status-controller namespace): oc create configmap -n openshift-update-status-controller status-api-cm-prototype
7. The controller should immediately-ish insert a usc-cv-version key into the ConfigMap. Its content is a YAML-serialized ClusterVersion status insight (see design doc). As of OTA-1269 the content is not that important, but the (1) reference to the CV (2) versions field should be correct.
8. The status insight should have a condition of Updating type. It should be False at this time (the cluster is not updating).
9. Start upgrading the cluster (it's a cluster bot cluster with ephemeral 4.18 version so you'll need to use --to-image=pullspec and probably force it
10. While updating, you should be able to observe the controller activity in the log (it logs some diffs), but also the content of the status insight in the ConfigMap changing. The versions field should change appropriately (and startedAt too), and the Updating condition should become True.
11. Eventually the update should finish and the Updating condition should flip to False again.
Some of these will turn into automated testcases, but it does not make sense to implement that automation while we're using the ConfigMap instead of the API.
After OTA-960 is fixed, ClusterVersion/version and oc adm upgrade can be used to monitor the process of migrating a cluster to multi-arch.
$ oc adm upgrade info: An upgrade is in progress. Working towards 4.18.0-ec.3: 761 of 890 done (85% complete), waiting on machine-config Upgradeable=False Reason: PoolUpdating Message: Cluster operator machine-config should not be upgraded between minor versions: One or more machine config pools are updating, please see `oc get mcp` for further details Upstream: https://api.integration.openshift.com/api/upgrades_info/graph Channel: candidate-4.18 (available channels: candidate-4.18) No updates available. You may still upgrade to a specific release image with --to-image or wait for new updates to be available.
But oc adm upgrade status reports COMPLETION 100% while the migration/upgrade is still ongoing.
$ OC_ENABLE_CMD_UPGRADE_STATUS=true oc adm upgrade status Unable to fetch alerts, ignoring alerts in 'Update Health': failed to get alerts from Thanos: no token is currently in use for this session = Control Plane = Assessment: Completed Target Version: 4.18.0-ec.3 (from 4.18.0-ec.3) Completion: 100% (33 operators updated, 0 updating, 0 waiting) Duration: 15m Operator Status: 33 Healthy Control Plane Nodes NAME ASSESSMENT PHASE VERSION EST MESSAGE ip-10-0-95-224.us-east-2.compute.internal Unavailable Updated 4.18.0-ec.3 - Node is unavailable ip-10-0-33-81.us-east-2.compute.internal Completed Updated 4.18.0-ec.3 - ip-10-0-45-170.us-east-2.compute.internal Completed Updated 4.18.0-ec.3 - = Worker Upgrade = WORKER POOL ASSESSMENT COMPLETION STATUS worker Completed 100% 3 Total, 2 Available, 0 Progressing, 0 Outdated, 0 Draining, 0 Excluded, 0 Degraded Worker Pool Nodes: worker NAME ASSESSMENT PHASE VERSION EST MESSAGE ip-10-0-72-40.us-east-2.compute.internal Unavailable Updated 4.18.0-ec.3 - Node is unavailable ip-10-0-17-117.us-east-2.compute.internal Completed Updated 4.18.0-ec.3 - ip-10-0-22-179.us-east-2.compute.internal Completed Updated 4.18.0-ec.3 - = Update Health = SINCE LEVEL IMPACT MESSAGE - Warning Update Speed Node ip-10-0-95-224.us-east-2.compute.internal is unavailable - Warning Update Speed Node ip-10-0-72-40.us-east-2.compute.internal is unavailable Run with --details=health for additional description and links to related online documentation $ oc get clusterversion version NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.18.0-ec.3 True True 14m Working towards 4.18.0-ec.3: 761 of 890 done (85% complete), waiting on machine-config $ oc get co machine-config NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE machine-config 4.18.0-ec.3 True True False 63m Working towards 4.18.0-ec.3
The reason is that PROGRESSING=True is not detected for co/machine-config as the status command checks only operator.Status.Versions[name=="operator"] and it needs to check ClusterOperator.Status.Versions[name=="operator-image"] as well.
For grooming:
It will be challenging for the status command to check the operator image's pull spec because it does not know the expected value. CVO knows it because CVO holds the manifests (containing the expected value) from the multi-arch payload.
One "hacky" workaround is that the status command gets the pull spec from the MCO deployment:
oc get deployment -n openshift-machine-config-operator machine-config-operator -o json | jq -r '.spec.template.spec.containers[]|select(.name=="machine-config-operator")|.image' quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787a505ca594b0a727549353c503dec9233a9d3c2dcd6b64e3de5f998892a1d5
Note this co/machine-config -> deployment/machine-config-operator trick may not be feasible if we want to extend it to all cluster operators. But it should work as a hacky workaround to check only MCO.
We may claim that the status command is not designed for monitoring the multi-arch migration and suggest to use oc adm upgrade instead. In that case, we can close this card as Obsolete/Won'tDo.
manifests.zip
has the mockData/manifests for the status cmd that are taken during the migration.
oc#1920 started the work for the status command to recognize the migration and we need to extend the work to cover (the comments from Petr's review):
- "Target Version: 4.18.0-ec.3 (from 4.18.0-ec.3)": confusing. We should tell "multi-arch" migration somehow. Or even better: from the current arch to multi-arch, for example "Target Version: 4.18.0-ec.3 multi (from x86_64)" if we could get the origin arch from CV or somewhere else.
- We have spec.desiredUpdate.architecture since forever, and can use that being Multi as a partial hint.
MULTIARCH-4559is adding tech-preview status properties around architecture in 4.18, but tech-preview, so may not be worth bothering with in oc code. Two history entries with the same version string but different digests is probably a reliable-enough heuristic, coupled with the spec-side hint.
- We have spec.desiredUpdate.architecture since forever, and can use that being Multi as a partial hint.
- "Duration: 6m55s (Est. Time Remaining: 1h4m)": We will see if we could find a simple way to hand this special case. I do not understand "the 97% completion will be reached so fast." as I am not familiar with the algorithm. But it seems acceptable to Petr that we show N/A for the migration.
- I think I get "the 97% completion will be reached so fast." now as only MCO has the operator-image pull spec. Other COs claim the completeness immaturely. With that said, "N/A" sounds like the most possible way for now.
- Node status like "All control plane nodes successfully updated to 4.18.0-ec.3" for control planes and "ip-10-0-17-117.us-east-2.compute.internal Completed". It is technically hard to detect the transaction during migration as MCO annotates only the version. This may become a separate card if it is too big to finish with the current one.
- "targetImagePullSpec := getMCOImagePullSpec(mcoDeployment)" should be computed just once. Now it is in the each iteration of the for loop. We should also comment about why we do it with this hacky way.
Implement a new Informer controller in the Update Status Controller to watch Node resources in the cluster and maintain an update status insight for each. The informer will need to interact with additional resources such as MachineConfigPools and MachineConfigs, e.g. to discover the OCP version tied to config that is being reconciled on the Node, but should not attempt to maintain the MachineConfigPool status insights. Generally the node status insight should carry enough data for any client to be able to render a line that the oc adm upgrade status currently shows:
NAME ASSESSMENT PHASE VERSION EST MESSAGE
build0-gstfj-ci-prowjobs-worker-b-9lztv Degraded Draining 4.16.0-ec.2 ? failed to drain node: <node> after 1 hour. Please see machine-config-controller logs for more informatio
build0-gstfj-ci-prowjobs-worker-d-ddnxd Unavailable Pending ? ? Machine Config Daemon is processing the node
build0-gstfj-ci-tests-worker-b-d9vz2 Unavailable Pending ? ? Not ready
build0-gstfj-ci-tests-worker-c-jq5rk Unavailable Updated 4.16.0-ec.3 - Node is marked unschedulable
The basic expectations for Node status insights are described in the design docs but the current source of truth for the data structure is the NodeStatusInsight structure from https://github.com/openshift/api/pull/2012 .
Definition of Done
- During the upgrade, the status api contains a Node status insight for each Node in the cluster
- Do not bother with the status insight lifecycle (when a Node is removed from the cluster, the status insight should technically disappear, but do not address that in this card, suitable lifecycle mechanism for this does not exist yet and OTA-1418 will address it)
- Overall the functionality should match what oc adm upgrade status client-based checks
- The NodeStatusInsight should have correctly populated: name, resource, poolResource, scopeType, version, estToComplete and message fields, following the existing logic from oc adm upgrade status
- Health insights are out of scope
- Status insights for MCPs are out of scope
- The Updating condition has a similar meaning and interpretation like in the other insights.
- When its status is False, it will contain a reason which needs to be interpreted. Three known reasons are Pending, Updated and Paused:
- Pending: Node will eventually be updated but has not started yet
- Updated: Node already underwent the update.
- Paused: Node is running an outdated version but something is pausing the process (like parent MCP .spec.paused field)
- When Updating=True, there are also three known reasons: Draining, Updating and Rebooting.
- Draining: MCO drains the node so it can be updated and rebooted
- Updating: MCO applies the new config and prepares the node to be rebooted into the new OS version
- Rebooting: MCO is rebooting the node, after which it (hopefully) becomes ready again
- When its status is False, it will contain a reason which needs to be interpreted. Three known reasons are Pending, Updated and Paused:
- The Degraded and Unavailable condition logic should match the existing assessment logic from oc adm upgrade status
Extend the Control Plane Informer in the Update Status Controller so it watches ClusterOperator resources in the cluster and maintains an update status insight for each.
The actual API structure for an update status insights needs to be taken from whatever state https://github.com/openshift/api/pull/2012 is at the moment. The story does not include the actual API form nor how it is exposed by the cluster (depending on the state of API approval, the USC may still expose the API as a ConfigMap or an actual custom resource), it includes just the logic to watch ClusterOperator resources and producing a matching set of cluster operator status insights.
The basic expectations for cluster operator status insights are described in the design docs
Definition of Done
- During the control plane upgrade, the status api contains a ClusterOperator status insight for each platform (hopefully we can use the same selection logic like we have in the prototype) ClusterOperator in the cluster
- Outside of control plane update, the cluster operator status insights are not present
- Updating condition:
- If a CO is updating right now (Progressing=True && does not have a target operator version), then Updating=True and a suitable reason
- Otherwise, if it has target version, Updating=False and Reason=Updated
- Otherwise, if it does not have target version, Updating=False and Reason=Pending
- Healthy condition
- Corresponds to the existing checks in the client prototype, taking into account the thresholds (Healthy=True if the "bad" condition is not holding long enough yet, but we may invent a special Reason for this case)
- This card does *not* involve creating Health Insights when there are unhealthy operators
- This card does *not* involve updating a ClusterVersion status insight from the CO-related data (such as completeness percentage or duration estimate)
Spun out of https://issues.redhat.com/browse/MCO-668
This aims to capture the work required to rotate the MCS-ignition CA + cert.
Original description copied from MCO-668:
Today in OCP there is a TLS certificate generated by the installer , which is called "root-ca" but is really "the MCS CA".
A key derived from this is injected into the pointer Ignition configuration under the "security.tls.certificateAuthorities" section, and this is how the client verifies it's talking to the expected server.
If this key expires (and by default the CA has a 10 year lifetime), newly scaled up nodes will fail in Ignition (and fail to join the cluster).
The MCO should take over management of this cert, and the corresponding user-data secret field, to implement rotation.
Reading:
- There is a section in the customer facing documentation that touches on this: https://docs.openshift.com/container-platform/4.13/security/certificate_types_descriptions/machine-config-operator-certificates.html
- There's a section in the customer facing documentation for this: https://docs.openshift.com/container-platform/4.13/security/certificate_types_descriptions/machine-config-operator-certificates.html that needs updating for clarification.
- There's a pending PR to openshift/api: https://github.com/openshift/api/pull/1484/files
- Also see old (related) bug: https://issues.redhat.com/browse/OCPBUGS-9890
- This is also separate to https://issues.redhat.com/browse/MCO-499 which describes the management of kubelet certs
The machinesets in the machine-api namespace reference a user-data secret (per pool and can be customized) which stores the initial ignition stub configuration pointing to the MCS, and the TLS cert. This today doesn't get updated after creation.
The MCO now has the ability to manage some fields of the machineset object as part of the managed bootimage work. We should extend that to also sync in the updated user-data secrets for the ignition tls cert.
The MCC should be able to parse both install-time-generated machinesets as well as user-created ones, so as to not break compatibility. One way users are using this today is to use a custom secret + machineset to do non-MCO compatible ignition fields, for example, to partition disks for different device types for nodes in the same pool. Extra care should be taken not to break this use case
The CA/cert generated by the installer is not currently managed and also does not preserve the signing key; so the cert controller we are adding in the MCO(leveraged from library-go), throws away everything and starts fresh. Normally this happens fairly quickly so both the MCS and the -user-data secrets are updated together. However, in certain cases(such as agent based installations) where a bootstrap node joins the cluster late, it will have the old CA from installer, and unfortunately the MCS will have a TLS cert signed by the new CA - resulting in invalid TLS cert errors.
To account for such cases, we have to ensure the first CA embedded in any machine is matching the format expected by the cert controller. To do this, we'll have to do the following in the installer:
Have the bootstrap MCC generate the CA/TLS cert using the cert controller, and populate them into the right places(this card)Make changes in the installer to remove the creation of the CA/cert, since the bootstrap MCC will now handle this(https://issues.redhat.com/browse/MCO-1458)- Template out all root-ca artifacts in the format expected by the library-go cert controller. This would involve adding certain annotations on the artifacts(with respect to validity of the cert and some other ownership metadata)
- The root CA signing key is currently discarded by the installer, so this will have to be a new template in the installer.
Feature Overview (aka. Goal Summary)
We need to maintain our dependencies across all the libraries we use in order to stay in compliance.
Goals (aka. expected user outcomes)
The observable functionality that the user now has as a result of receiving this feature. Include the anticipated primary user type/persona and which existing features, if any, will be expanded. Complete during New status.
<your text here>
Requirements (aka. Acceptance Criteria):
A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.
<enter general Feature acceptance here>
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
| Deployment considerations | List applicable specific needs (N/A = not applicable) |
| Self-managed, managed, or both | |
| Classic (standalone cluster) | |
| Hosted control planes | |
| Multi node, Compact (three node), or Single node (SNO), or all | |
| Connected / Restricted Network | |
| Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
| Operator compatibility | |
| Backport needed (list applicable versions) | |
| UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
| Other (please specify) |
Use Cases (Optional):
Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.
<your text here>
Questions to Answer (Optional):
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
<your text here>
Out of Scope
High-level list of items that are out of scope. Initial completion during Refinement status.
<your text here>
Background
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
<your text here>
Customer Considerations
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
<your text here>
Documentation Considerations
Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.
<your text here>
Interoperability Considerations
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
<your text here>
Epic Goal
- Migrate all components to functional components
- Remove all HOC patterns
- Break the file down into smaller files
- Improve type definitions
- Improve naming for better self-documentation
- Address any React anti-patterns like nested components, or mirroring props in state.
- Address issues with handling binary data
- Add unit tests to these components
Acceptance Criteria
- Refactor secret forms
- Adding unit tests to these components.
- Fix https://issues.redhat.com/browse/OCPBUGS-32401
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
The SourceSecretForm component needs to be refactored to address several tech debt issues: * Rename to AuthSecretForm
- Refactor into a function component
- Remove i18n withTranslation HOC pattern
- Improve type definitions
The BasicAuthSubform component needs to be refactored to address several tech debt issues: * Rename to BasicAuthSecretForm
- Refactor into a function component
- Remove i18n withTranslation HOC pattern
- Improve type definitions
An epic we can duplicate for each release to ensure we have a place to catch things we ought to be doing regularly but can tend to fall by the wayside.
Currently console is using TypeScript 4, which is preventing us from upgrading to NodeJS-22. Due to that we need to update TypeScript 5 (not necessarily latest version).
AC:
- Update TypeScript to version 5
- Update ES build target to ES-2021
Note: In case of higher complexity we should be splitting the story into multiple stories, per console package.
As a developer I want to make sure we are running the latest version of webpack in order to take advantage of the latest benefits and also keep current so that future updating is a painless as possible.
We are currently on v4.47.0.
Changelog: https://webpack.js.org/blog/2020-10-10-webpack-5-release/
By updating to version 5 we will need to update following pkgs as well:
- html-webpack-plugin
- webpack-bundle-analyzer
- copy-webpack-plugin
- fork-ts-checker-webpack-plugin
AC: Update webpack to version 5 and determine what should be the ideal minor version.
Feature Overview (aka. Goal Summary)
An elevator pitch (value statement) that describes the Feature in a clear, concise way. Complete during New status.
Some Kubernetes clusters do not have direct Internet access and rely solely on proxies for communication, so OLM v1 needs to support proxies to enable this communication.
Goals (aka. expected user outcomes)
The observable functionality that the user now has as a result of receiving this feature. Include the anticipated primary user type/persona and which existing features, if any, will be expanded. Complete during New status.
Some Kubernetes clusters do not have direct Internet access and rely solely on proxies for communication. This may be done for isolation, testing or to enhance security and minimise vulnerabilities. This is a fully supported configuration in OpenShift, with origin tests designed to validate functionality in proxy-based environments. Supporting proxies is essential to ensure your solution operates reliably within these secure and compliant setups.
To address this need, we have two key challenges to solve:
- Enable catalogd and the operator-controller to work with OpenShift proxy configurations.
- Implement a solution to pass the proxy configuration for the solutions managed by OLM so that they work well with the proxy.
Requirements (aka. Acceptance Criteria):
A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.
- Enable catalogd and the operator-controller to work with OpenShift proxy configurations.
- Implement a solution to pass the proxy configuration for the solutions managed by OLM so that they work well with the proxy.
- Trusted CA support
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
| Deployment considerations | List applicable specific needs (N/A = not applicable) |
| Self-managed, managed, or both | |
| Classic (standalone cluster) | |
| Hosted control planes | |
| Multi node, Compact (three node), or Single node (SNO), or all | |
| Connected / Restricted Network | Restricted Network |
| Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
| Operator compatibility | |
| Backport needed (list applicable versions) | 4.18 |
| UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
| Other (please specify) |
Use Cases (Optional):
Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.
<your text here>
Questions to Answer (Optional):
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
<your text here>
Out of Scope
High-level list of items that are out of scope. Initial completion during Refinement status.
- Proxy support for operator-controller when communicating with catalogd.
- Adding code to support proxies upstream (the environment variable mechanism can be used).
- Support for CSV-defined proxies.
- oc-mirror support (which should already be there)
Background
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
OpenShift’s centralized proxy control via the proxies.config.openshift.io (a.k.a. proxy.config.openshift.io) resource makes managing proxies across a cluster easier. At the same time, vanilla Kubernetes requires a manual and decentralized proxy configuration, making it more complex to manage, especially in large clusters. There is no native Kubernetes solution that can adequately address the need for centralized proxy management.
Kubernetes lacks a built-in unified API like OpenShift’s proxies.config.openshift.io, which can streamline proxy configuration and management across any Kubernetes vendor. Consequently, Kubernetes requires more manual work to ensure the proxy configuration is consistent across the cluster, and this complexity increases with the scale of the environment. As such, vanilla Kubernetes does not provide a solution that can natively address proxy configuration across all clusters and vendors without relying on external tools or complex manual processes (such as that devised by OpenShift).
Customer Considerations
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
<your text here>
Documentation Considerations
Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.
<your text here>
Interoperability Considerations
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
<your text here>
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
Epic Goal
- Support OpenShift Cluster Proxy
See RFC for more details
Why is this important?
Scenarios
- ...
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement details and documents.
- ...
Dependencies (internal and external)
- ...
Previous Work (Optional):
- …
Open questions::
- …
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
It looks like OLMv1 doesn't handle proxies correctly, aws-ovn-proxy job is permafailing https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-aws-ovn-proxy/1861444783696777216
I suspect it's on the OLM operator side, are you looking at the cluster-wide proxy object and wiring it into your containers if set?
Feature Overview (aka. Goal Summary)
The ability in OpenShift to create trust and directly consume access tokens issued by external OIDC Authentication Providers using an authentication approach similar to upstream Kubernetes.
BYO Identity will help facilitate CLI only workflows and capabilities of the Authentication Provider (such as Keycloak, Dex, Azure AD) similar to upstream Kubernetes.
Goals (aka. expected user outcomes)
Ability in OpenShift to provide a direct, pluggable Authentication workflow such that the OpenShift/K8s API server can consume access tokens issued by external OIDC identity providers. Kubernetes provides this integration as described here. Customer/Users can then configure their IDPs to support the OIDC protocols and workflows they desire such as Client credential flow.
OpenShift OAuth server is still available as default option, with the ability to tune in the external OIDC provider as a Day-2 configuration.
Requirements (aka. Acceptance Criteria):
- The customer should be able to tie into RBAC functionality, similar to how it is closely aligned with OpenShift OAuth
Use Cases (Optional):
- As a customer, I would like to integrate my OIDC Identity Provider directly with the OpenShift API server.
- As a customer in multi-cluster cloud environment, I have both K8s and non-K8s clusters using my IDP and hence I need seamless authentication directly to the OpenShift/K8sAPI using my Identity Provider
Questions to Answer (Optional):
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
Out of Scope
High-level list of items that are out of scope. Initial completion during Refinement status.
Background
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
Customer Considerations
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
Documentation Considerations
Provide information that needs to be considered and planned so that documentation will meet customer needs. Initial completion during Refinement status.
Interoperability Considerations
Which other projects and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
Epic Goal
The ability to provide a direct authentication workflow such that OpenShift can consume bearer tokens issued by external OIDC identity providers, replacing the built-in OAuth stack by deactivating/removing its components as necessary.
Why is this important? (mandatory)
OpenShift has its own built-in OAuth server which can be used to obtain OAuth access tokens for authentication to the API. The server can be configured with an external identity provider (including support for OIDC), however it is still the built-in server that issues tokens, and thus authentication is limited to the capabilities of the oauth-server.
Scenarios (mandatory)
- As a customer, I want to integrate my OIDC Identity Provider directly with OpenShift so that I can fully use its capabilities in machine-to-machine workflows.
*As a customer in a hybrid cloud environment, I want to seamlessly use my OIDC Identity Provider across all of my fleet.
Dependencies (internal and external) (mandatory)
- Support in the console/console-operator (already completed)
- Support in the OpenShift CLI `oc` (already completed)
Contributing Teams(and contacts) (mandatory)
- Development - OCP Auth
- Documentation - OCP Auth
- QE - OCP Auth
- PX -
- Others -
Acceptance Criteria (optional)
- external OIDC provider can be configured to be used directly via the kube-apiserver to issue tokens
- built-in oauth stack no longer operational in the cluster; respective APIs, resources and components deactivated
- changing back to the built-in oauth stack possible
Drawbacks or Risk (optional)
- Enabling an external OIDC provider to an OCP cluster will result in the oauth-apiserver being removed from the system; this inherently means that the two API Services it is serving (v1.oauth.openshift.io, v1.user.openshift.io) will be gone from the cluster, and therefore any related data will be lost. It is the user's responsibility to create backups of any required data.
- Configuring an external OIDC identity provider for authentication by definition means that any security updates or patches must be managed independently from the cluster itself, i.e. cluster updates will not resolve security issues relevant to the provider itself; the provider will have to be updated separately. Additionally, new functionality or features on the provider's side might need integration work in OpenShift (depending on their nature).
Done - Checklist (mandatory)
- CI Testing - Basic e2e automationTests are merged and completing successfully
- Documentation - Content development is complete.
- QE - Test scenarios are written and executed successfully.
- Technical Enablement - Slides are complete (if requested by PLM)
- Engineering Stories Merged
- All associated work items with the Epic are closed
- Epic status should be “Release Pending”
Description of problem:
This is a bug found during pre-merge test of 4.18 epic AUTH-528 PRs and filed for better tracking per existing "OpenShift - Testing Before PR Merges - Left-Shift Testing" google doc workflow.
co/console degraded with AuthStatusHandlerDegraded after OCP BYO external oidc is configured and then removed (i.e. reverted back to OAuth IDP).
Version-Release number of selected component (if applicable):
Cluster-bot build which is built at 2024-11-25 09:39 CST (UTC+800) build 4.18,openshift/cluster-authentication-operator#713,openshift/cluster-authentication-operator#740,openshift/cluster-kube-apiserver-operator#1760,openshift/console-operator#940
How reproducible:
Always (tried twice, both hit it)
Steps to Reproduce:
1. Launch a TechPreviewNoUpgrade standalone OCP cluster with above build. Configure htpasswd IDP. Test users can login successfully.
2. Configure BYO external OIDC in this OCP cluster using Microsoft Entra ID. KAS and console pods can roll out successfully. oc login and console login to Microsoft Entra ID can succeed.
3. Remove BYO external OIDC configuration, i.e. go back to original htpasswd OAuth IDP:
[xxia@2024-11-25 21:10:17 CST my]$ oc patch authentication.config/cluster --type=merge -p='
spec:
type: ""
oidcProviders: null
'
authentication.config.openshift.io/cluster patched
[xxia@2024-11-25 21:15:24 CST my]$ oc get authentication.config cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Authentication
metadata:
annotations:
include.release.openshift.io/ibm-cloud-managed: "true"
include.release.openshift.io/self-managed-high-availability: "true"
release.openshift.io/create-only: "true"
creationTimestamp: "2024-11-25T04:11:59Z"
generation: 5
name: cluster
ownerReferences:
- apiVersion: config.openshift.io/v1
kind: ClusterVersion
name: version
uid: e814f1dc-0b51-4b87-8f04-6bd99594bf47
resourceVersion: "284724"
uid: 2de77b67-7de4-4883-8ceb-f1020b277210
spec:
oauthMetadata:
name: ""
serviceAccountIssuer: ""
type: ""
webhookTokenAuthenticator:
kubeConfig:
name: webhook-authentication-integrated-oauth
status:
integratedOAuthMetadata:
name: oauth-openshift
oidcClients:
- componentName: cli
componentNamespace: openshift-console
- componentName: console
componentNamespace: openshift-console
conditions:
- lastTransitionTime: "2024-11-25T13:10:23Z"
message: ""
reason: OIDCConfigAvailable
status: "False"
type: Degraded
- lastTransitionTime: "2024-11-25T13:10:23Z"
message: ""
reason: OIDCConfigAvailable
status: "False"
type: Progressing
- lastTransitionTime: "2024-11-25T13:10:23Z"
message: ""
reason: OIDCConfigAvailable
status: "True"
type: Available
currentOIDCClients:
- clientID: 95fbae1d-69a7-4206-86bd-00ea9e0bb778
issuerURL: https://login.microsoftonline.com/6047c7e9-b2ad-488d-a54e-dc3f6be6a7ee/v2.0
oidcProviderName: microsoft-entra-id
KAS and console pods indeed can roll out successfully; and now oc login and console login indeed can succeed using the htpasswd user and password:
[xxia@2024-11-25 21:49:32 CST my]$ oc login -u testuser-1 -p xxxxxx
Login successful.
...
But co/console degraded, which is weird:
[xxia@2024-11-25 21:56:07 CST my]$ oc get co | grep -v 'True *False *False'
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
console 4.18.0-0.test-2024-11-25-020414-ci-ln-71cvsj2-latest True False True 9h AuthStatusHandlerDegraded: Authentication.config.openshift.io "cluster" is invalid: [status.oidcClients[1].currentOIDCClients[0].issuerURL: Invalid value: "": oidcClients[1].currentOIDCClients[0].issuerURL in body should match '^https:\/\/[^\s]', status.oidcClients[1].currentOIDCClients[0].oidcProviderName: Invalid value: "": oidcClients[1].currentOIDCClients[0].oidcProviderName in body should be at least 1 chars long]
Actual results:
co/console degraded, as above.
Expected results:
co/console is normal.
Additional info:
Upstream K8s deprecated PodSecurityPolicy and replaced it with a new built-in admission controller that enforces the Pod Security Standards (See here for the motivations for deprecation).] There is an OpenShift-specific dedicated pod admission system called Security Context Constraints. Our aim is to keep the Security Context Constraints pod admission system while also allowing users to have access to the Kubernetes Pod Security Admission.
With OpenShift 4.11, we are turned on the Pod Security Admission with global "privileged" enforcement. Additionally we set the "restricted" profile for warnings and audit. This configuration made it possible for users to opt-in their namespaces to Pod Security Admission with the per-namespace labels. We also introduced a new mechanism that automatically synchronizes the Pod Security Admission "warn" and "audit" labels.
With OpenShift 4.19, we intend to move the global configuration to enforce the "restricted" pod security profile globally. With this change, the label synchronization mechanism will also switch into a mode where it synchronizes the "enforce" Pod Security Admission label rather than the "audit" and "warn".
Epic Goal
Get Pod Security admission to be run in "restricted" mode globally by default alongside with SCC admission.
When creating a custom SCC, it is possible to assign a priority that is higher than existing SCCs. This means that any SA with access to all SCCs might use the higher priority custom SCC, and this might mutate a workload in an unexpected/unintended way.
To protect platform workloads from such an effect (which, combined with PSa, might result in rejecting the workload once we start enforcing the "restricted" profile) we must pin the required SCC to all workloads in platform namespaces (openshift-, kube-, default).
Each workload should pin the SCC with the least-privilege, except workloads in runlevel 0 namespaces that should pin the "privileged" SCC (SCC admission is not enabled on these namespaces, but we should pin an SCC for tracking purposes).
The following tables track progress.
Progress summary
| # namespaces | 4.19 | 4.18 | 4.17 | 4.16 | 4.15 | 4.14 |
|---|---|---|---|---|---|---|
| monitored | 82 | 82 | 82 | 82 | 82 | 82 |
| fix needed | 68 | 68 | 68 | 68 | 68 | 68 |
| fixed | 39 | 39 | 35 | 32 | 39 | 1 |
| remaining | 29 | 29 | 33 | 36 | 29 | 67 |
| ~ remaining non-runlevel | 8 | 8 | 12 | 15 | 8 | 46 |
| ~ remaining runlevel (low-prio) | 21 | 21 | 21 | 21 | 21 | 21 |
| ~ untested | 3 | 2 | 2 | 2 | 82 | 82 |
Progress breakdown
| # | namespace | 4.19 | 4.18 | 4.17 | 4.16 | 4.15 | 4.14 |
|---|---|---|---|---|---|---|---|
| 1 | oc debug node pods | |
|
#1763 |
#1816 |
#1818 |
|
| 2 | openshift-apiserver-operator | |
|
|
#573 |
#581 |
|
| 3 | openshift-authentication | |
|
|
#656 |
#675 |
|
| 4 | openshift-authentication-operator | |
|
|
#656 |
#675 |
|
| 5 | openshift-catalogd | |
|
|
#50 |
#58 |
|
| 6 | openshift-cloud-credential-operator | |
|
|
#681 |
#736 |
|
| 7 | openshift-cloud-network-config-controller | |
#2282 |
#2490 |
#2496 | ||
| 8 | openshift-cluster-csi-drivers | #118 #5310 #135 | #524 #131 #306 #265 #75 | #170 #459 | #484 |
||
| 9 | openshift-cluster-node-tuning-operator | |
|
|
#968 |
#1117 |
|
| 10 | openshift-cluster-olm-operator | |
|
|
#54 |
n/a | n/a |
| 11 | openshift-cluster-samples-operator | |
|
|
#535 |
#548 |
|
| 12 | openshift-cluster-storage-operator | |
#516 |
#459 #196 | #484 #211 |
||
| 13 | openshift-cluster-version | #1038 | #1068 |
||||
| 14 | openshift-config-operator | |
|
|
#410 |
#420 |
|
| 15 | openshift-console | |
|
#871 |
#908 |
#924 |
|
| 16 | openshift-console-operator | |
|
#871 |
#908 |
#924 |
|
| 17 | openshift-controller-manager | |
|
|
#336 |
#361 |
|
| 18 | openshift-controller-manager-operator | |
|
|
#336 |
#361 |
|
| 19 | openshift-e2e-loki | |
#56579 |
#56579 |
#56579 |
#56579 |
|
| 20 | openshift-image-registry | #1008 | #1067 |
||||
| 21 | openshift-ingress | #1032 | |||||
| 22 | openshift-ingress-canary | #1031 | |||||
| 23 | openshift-ingress-operator | #1031 | |||||
| 24 | openshift-insights | #1033 |
#1041 |
#1049 |
#915 |
#967 |
|
| 25 | openshift-kni-infra | |
#4504 |
#4542 |
#4539 |
#4540 |
|
| 26 | openshift-kube-storage-version-migrator | |
|
|
#107 |
#112 |
|
| 27 | openshift-kube-storage-version-migrator-operator | |
|
|
#107 |
#112 |
|
| 28 | openshift-machine-api | #1308 #1317 | #1311 | #407 | #315 #282 #1220 #73 #50 #433 | #332 #326 #1288 #81 #57 #443 |
|
| 29 | openshift-machine-config-operator | |
#4636 |
#4219 | #4384 | #4393 |
|
| 30 | openshift-manila-csi-driver | |
|
#234 |
#235 |
#236 |
|
| 31 | openshift-marketplace | |
#578 |
|
#561 |
#570 |
|
| 32 | openshift-metallb-system | |
#238 |
#240 |
#241 | ||
| 33 | openshift-monitoring | #2298 #366 | #2498 | #2335 | #2420 |
||
| 34 | openshift-network-console | |
#2545 |
||||
| 35 | openshift-network-diagnostics | |
#2282 |
#2490 |
#2496 | ||
| 36 | openshift-network-node-identity | |
#2282 |
#2490 |
#2496 | ||
| 37 | openshift-nutanix-infra | |
#4504 |
|
#4539 |
#4540 |
|
| 38 | openshift-oauth-apiserver | |
|
|
#656 |
#675 |
|
| 39 | openshift-openstack-infra | |
#4504 |
#4539 |
#4540 |
||
| 40 | openshift-operator-controller | |
|
|
#100 |
#120 |
|
| 41 | openshift-operator-lifecycle-manager | |
|
|
#703 |
#828 |
|
| 42 | openshift-route-controller-manager | |
|
|
#336 |
#361 |
|
| 43 | openshift-service-ca | |
|
|
#235 |
#243 |
|
| 44 | openshift-service-ca-operator | |
|
|
#235 |
#243 |
|
| 45 | openshift-sriov-network-operator | |
|
#995 |
#999 |
#1003 |
|
| 46 | openshift-user-workload-monitoring | |
|
|
#2335 |
#2420 |
|
| 47 | openshift-vsphere-infra | |
#4504 |
#4542 |
#4539 |
#4540 |
|
| 48 | (runlevel) kube-system | ||||||
| 49 | (runlevel) openshift-cloud-controller-manager | ||||||
| 50 | (runlevel) openshift-cloud-controller-manager-operator | ||||||
| 51 | (runlevel) openshift-cluster-api | ||||||
| 52 | (runlevel) openshift-cluster-machine-approver | ||||||
| 53 | (runlevel) openshift-dns | ||||||
| 54 | (runlevel) openshift-dns-operator | ||||||
| 55 | (runlevel) openshift-etcd | ||||||
| 56 | (runlevel) openshift-etcd-operator | ||||||
| 57 | (runlevel) openshift-kube-apiserver | ||||||
| 58 | (runlevel) openshift-kube-apiserver-operator | ||||||
| 59 | (runlevel) openshift-kube-controller-manager | ||||||
| 60 | (runlevel) openshift-kube-controller-manager-operator | ||||||
| 61 | (runlevel) openshift-kube-proxy | ||||||
| 62 | (runlevel) openshift-kube-scheduler | ||||||
| 63 | (runlevel) openshift-kube-scheduler-operator | ||||||
| 64 | (runlevel) openshift-multus | ||||||
| 65 | (runlevel) openshift-network-operator | ||||||
| 66 | (runlevel) openshift-ovn-kubernetes | ||||||
| 67 | (runlevel) openshift-sdn | ||||||
| 68 | (runlevel) openshift-storage |
Feature Overview (aka. Goal Summary)
Implement Migration core for MAPI to CAPI for AWS
- This feature covers the design and implementation of converting from using the Machine API (MAPI) to Cluster API (CAPI) for AWS
- This Design investigates possible solutions for AWS
- Once AWS shim/sync layer is implemented use the architecture for other clouds in phase-2 & phase 3
Acceptance Criteria
When customers use CAPI, There must be no negative effect to switching over to using CAPI . Seamless migration of Machine resources. the fields in MAPI/CAPI should reconcile from both CRDs.
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
Epic Goal
- Create the core/common tooling needed to enable the migration designed in
OCPCLOUD-1578 - To allow providers to individually migrate from MAPI to CAPI
- Implementation plan in https://docs.google.com/document/d/1IZPmcJujKPdoBZKt66i3eGcJWb1RDIgk1ywt13h6T-w/edit
Why is this important?
- We need to build out the core so that development of the migration for individual providers can then happen in parallel
Scenarios
- ...
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement details and documents.
- ...
Dependencies (internal and external)
- ...
Previous Work (Optional):
- ...
Open questions::
- ...
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
As QE have tried to test upstream CAPI pausing, we've hit a few issues with running the migration controller. & cluster capi operator on a real cluster vs envtest.
This card captures the work required to iron out these kinks, and get things running (i.e not crashing).
I also think we want an e2e or some sort of automated testing to ensure we don't break things again.
Goal: Stop the CAPI operator crashing on startup in a real cluster.
Non goals: get the entire conversion flow running from CAPI -> MAPI and MAPI -> CAPI. We still need significant feature work before we're here.
- https://github.com/openshift/cluster-capi-operator/pull/244
- https://github.com/openshift/cluster-capi-operator/pull/243
- https://github.com/openshift/cluster-capi-operator/pull/242
- https://github.com/openshift/cluster-capi-operator/pull/238
- https://github.com/openshift/cluster-machine-approver/pull/262
Background
For the MachineSet controller, we need to implement a forward conversion, converting the MachineAPI MachineSet to ClusterAPI.
This will involve creating the CAPI MachineSet if it does not exist, and managing the Infrastructure templates.
This card covers the case where MAPI is currently authoritative.
Behaviours
- Create Cluster API mirror if not present
- CAPI mirror should be paused on create
- Names of mirror should be 1:1 with original
- Manage InfraTemplate creation by converting MAPI providerSpec
- InfraTemplate naming should be based on hash to be able to deduplicate
- InfraTemplate naming should be based on parent resources
- InfraTemplate should have ownerReference to CAPI MachineSet
- If template has changed, remove ownerReference from old template. If no other ownerReferences, remove template.
- Should be identifiable as created by the sync controller (annotated?)
- Ensure CAPI MachineSet spec and status overwritten with conversion from MAPI
- Ensure Labels/Annotations copied from MAPI to CAPI
- On failures
- Set Synchronized condition to False and report error on MAPI resource
- On success
- Set Synchronized condition to True on MAPI resource
- Set status.synchronizedGeneration to match the auth resource generation
Steps
- Implement MAPI to CAPI conversion by leveraging library for conversion and applying above MachineSet level rules
Stakeholders
- Cluster Infra
Definition of Done
- When a MAPI MachineSet exists, a CAPI MachineSet is created and kept up to date if there are changes
- Docs
- <Add docs requirements for this card>
- Testing
- <Explain testing that will be added>
Background
To enable CAPI MachineSets to still mirror MAPI MachineSets accurately, and to enable MAPI MachineSets to be implemented by CAPI MachineSets in the future, we need to implement a way to convert CAPI Machines back into MAPI Machines.
These steps assume that the CAPI Machine is authoritative, or, that there is no MAPI Machines.
Behaviours
- If no Machine exists in MAPI
- But the CAPI Machine is owned, and that owner exists in MAPI
- Create a MAPI Machine to mirror the CAPI Machine
- MAPI Machines should set authority to CAPI on create
- But the CAPI Machine is owned, and that owner exists in MAPI
- If a MAPI Machine exists
- Convert infrastructure template from InfraMachine to providerSpec
- Update spec and status fields of MAPI Machine to reflect CAPI Machine
- On failures
- Set Synchronized condition to False and report error on MAPI resource
- On success
- Set Synchronized condition to True on MAPI resource
- Set status.synchronizedGeneration to match the auth resource generation
Steps
- Implement conversion based on the behaviours outlined above using the CAPI to MAPI conversion library
Stakeholders
- Cluster Infra
Definition of Done
- When a CAPI MachineSet scales up and is mirrored in MAPI, the CAPI Machine gets mirrored into MAPI
- Docs
- <Add docs requirements for this card>
- Testing
- <Explain testing that will be added>
Epic Goal
This is the epic tracking the work to collect a list of TLS artifacts (certificates, keys and CA bundles).
This list will contain a set of required an optional metadata. Required metadata examples are ownership (name of Jira component) and ability to auto-regenerate certificate after it has expired while offline. In most cases metadata can be set via annotations to secret/configmap containing the TLS artifact.
Components not meeting the required metadata will fail CI - i.e. when a pull request makes a component create a new secret, the secret is expected to have all necessary metadata present to pass CI.
This PR will enforce it WIP API-1789: make TLS registry tests required
Description of problem:
In order to make TLS registry tests required we need to make sure all OpenShift variants are using the same metadata for kube-apiserver certs. Hypershift uses several certs stored in the secret without accompanying metadata (namely component ownership).
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1.
2.
3.
Actual results:
Expected results:
Additional info:
Feature Overview (aka. Goal Summary)
As a cluster administrator, I want to use Karpenter on an OpenShift cluster running in AWS to scale nodes instead of Cluster Autoscalar(CAS). I want to automatically manage heterogeneous compute resources in my OpenShift cluster without the additional manual task of managing node pools. Additional features I want are:
- Reducing cloud costs through instance selection and scaling/descaling
- Support GPUs, spot instances, mixed compute types and other compute types.
- Automatic node lifecycle management and upgrades
This feature covers the work done to integrate upstream Karpenter 1.x with ROSA HCP. This eliminates the need for manual node pool management while ensuring cost-effective compute selection for workloads. Red Hat manages the node lifecycle and upgrades.
The feature will be rolled out with ROSA (AWS) since it has more mature Karpenter ecosystem, followed by ARO (Azure) implementation(check OCPSTRAT-1498)
Goals (aka. expected user outcomes)
- Run Karpenter in management cluster and disable CAS
- Automate node provisioning in workload cluster
- automate lifecycle management in workload cluster
- Reduce cost in heterogenous compute workloads
- Additional features karpenter https://docs.google.com/spreadsheets/d/1TjoLVVK5gbN29Sd4hLX5yS5wSYCTSHFmZGIcU3jgpCw/edit?gid=0#gid=0
Requirements (aka. Acceptance Criteria):
As a cluster-admin or SRE I should be able to configure Karpenter with OCP on AWS. Both cli and UI should enable users to configure Karpenter and disable CAS.
- Run Karpenter in management cluster and disable CAS
- OCM API
- Enable/Disable Cluster autoscaler
- Enable/disable AutoNode feature
- New ARN role configuration for Karpenter
- Optional: New managed policy or integration with existing nodepool role permissions
- Expose NodeClass/Nodepool resources to users.
- secure node provisioning and management, machine approval system for Karpenter instances
- HCP Karpenter cleanup/deletion support
- ROSA CAPI fields to enable/disable/configure Karpenter
- Write end-to-end tests for karpenter running on ROSA HCP
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
| Deployment considerations | List applicable specific needs (N/A = not applicable) |
| Self-managed, managed, or both | managed ROSA HCP |
| Classic (standalone cluster) | |
| Hosted control planes | yes |
| Multi node, Compact (three node), or Single node (SNO), or all | MNO |
| Connected / Restricted Network | Connected |
| Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | x86_x64, ARM (aarch64) |
| Operator compatibility | |
| Backport needed (list applicable versions) | No |
| UI need (e.g. OpenShift Console, dynamic plugin, OCM) | yes - console |
| Other (please specify) | rosa-cli |
Use Cases (Optional):
Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.
<your text here>
Questions to Answer (Optional):
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
<your text here>
Out of Scope
High-level list of items that are out of scope. Initial completion during Refinement status.
- Supporting this feature in Standalone OCP/self-hosted HCP/ROSA classic
- Creating a multi-provider cost/pricing operator compatible with CAPI is beyond the scope of this Feature. That may take more time.
Background
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
- Karpenter.sh is an open-source node provisioning project built for Kubernetes. It is designed to simplify Kubernetes infrastructure by automatically launching and terminating nodes based on the needs of your workloads. Karpenter can help you to reduce costs, improve performance, and simplify operations.
- Karpenter works by observing the unscheduled pods in your cluster and launching new nodes to accommodate them. Karpenter can also terminate nodes that are no longer needed, which can help you save money on infrastructure costs.
- Karpenter architecture has a Karpenter-core and Karpenter-provider as components.
The core has AWS code which does the resource calculation to reduce the cost by re-provisioning new nodes.
Customer Considerations
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
<your text here>
Documentation Considerations
Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.
- Migration guides from using CAS to Karpenter
- Performance testing to compare CAS vs Karpenter on ROSA HCP
- API documentation for NodePool and EC2NodeClass configuration
Interoperability Considerations
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
<your text here>
Goal
- Have a CI pipeline that runs HO with --tech-preview
- Add e2e test for autoNode via karpenter that validates: all manifest are created as expected management and guest side. Karpenter is able to autoprovision and remove compute.
Why is this important?
- …
Scenarios
- ...
Acceptance Criteria
- Dev - Has a valid enhancement if necessary
- CI - MUST be running successfully with tests automated
- QE - covered in Polarion test plan and tests implemented
- Release Technical Enablement - Must have TE slides
- ...
Dependencies (internal and external)
- ...
Previous Work (Optional):
- …
Open questions:
- …
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Technical Enablement <link to Feature Enablement Presentation>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Enhancement merged: <link to meaningful PR or GitHub Issue>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
Goal
- As a service provider I the cluster admin to only manipulate fields of the nodeclass API that won't impact the service ability to operate, e.g. userdata and ami can't be changed.
- As a service provider I want to be the solely authoritative source of truth to set input that impacts the ability to operate AutoNode.
Why is this important?
- The way we implement this will have UX implications for cluster admin which has direct impact on customer satisfaction.
Scenarios
We decided to start by using validating admission policies to implement ownership of ec2NodeClass fields. So we can restrict fields crud to a particular service account.
This has some caveats:
- If a field that the service own is required in the API, we need to let the cluster admin to set it on creation even though we'll clobber it by reconciling a controller. To mitigate this we might want to change the upstream CEL validations of the ec2NodeClass API
- The raw userdata is exposed to the cluster admin via ec2NodeClass.spec.userdata
- Since we enforce the values for userdata and ami via controller reconciliation there's potential room for race conditions
If using validating policies for this proves to be satisfactory we'll need to consider alternatives, e.g:
- Having an additional dedicated CRD for openshiftnodeclass that translates into the ec2NodeClass and completely prevent the cluster admin from interacting with the latter via vap.
- having our own class similar to eks so we can fully manage the operational input in the backend.
- # ...
Acceptance Criteria
- Dev - Has a valid enhancement if necessary
- CI - MUST be running successfully with tests automated
- QE - covered in Polarion test plan and tests implemented
- Release Technical Enablement - Must have TE slides
- ...
Dependencies (internal and external)
- ...
Previous Work (Optional):
- …
Open questions:
- …
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Technical Enablement <link to Feature Enablement Presentation>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Enhancement merged: <link to meaningful PR or GitHub Issue>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
Goal
- Validate the implementation details are prod ready before removing the feature gate
Why is this important?
- …
Scenarios
- Revisit Default ec2Class
- Revisit karpenter IAM policy
- Consider vendoring karpenter instead of using unstructured types
- Consider manage the userdata programatically instead of using NodePool API
- Revisit rbac and watchers
- Last time I checked, karpenter was failing to allocate OCP workloads because of a conflicting label.
https://github.com/kubernetes-sigs/karpenter/issues/1046#issuecomment-2578291462
https://github.com/kubernetes-sigs/karpenter/pull/1908
- Remove feature gate
- ...
Acceptance Criteria
- Dev - Has a valid enhancement if necessary
- CI - MUST be running successfully with tests automated
- QE - covered in Polarion test plan and tests implemented
- Release Technical Enablement - Must have TE slides
- ...
Dependencies (internal and external)
- ...
Previous Work (Optional):
- …
Open questions:
- …
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Technical Enablement <link to Feature Enablement Presentation>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Enhancement merged: <link to meaningful PR or GitHub Issue>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
Move from karpenter nodepool to programatic generate userdata
Goal
- Instances created by karpenter can automatically become Nodes
Why is this important?
- Reduce operational burden.
Scenarios
For CAPI/MAPI driven machine management the cluster-machine-approver uses the machine.status.ips to match the CSRs. In karpenter there's no Machine resources
We'll need to implement something similar. Some ideas:
- Explore using the nodeClaim resource info like status.providerID to match the CSRs
- Store the requesting IP when the ec2 instances query ignition and follow similar comparison criteria than machine approver to match CSRs
- Query AWS to get info and compare info to match CSRs
- ...
Acceptance Criteria
- Dev - Has a valid enhancement if necessary
- CI - MUST be running successfully with tests automated
- QE - covered in Polarion test plan and tests implemented
- Release Technical Enablement - Must have TE slides
- ...
Dependencies (internal and external)
- ...
Previous Work (Optional):
- …
Open questions:
- …
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Technical Enablement <link to Feature Enablement Presentation>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Enhancement merged: <link to meaningful PR or GitHub Issue>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
User Story:
As a (user persona), I want to be able to:
- Instances created by karpenter can automatically become Nodes
so that I can achieve
- Reduce operational burden.
Acceptance Criteria:
Description of criteria:
- For CAPI/MAPI driven machine management the cluster-machine-approver uses the machine.status.ips to match the CSRs. In karpenter there's no Machine resources
We'll need to implement something similar. Some ideas:
– Explore using the nodeClaim resource info like status.providerID to match the CSRs
– Store the requesting IP when the ec2 instances query ignition and follow similar comparison criteria than machine approver to match CSRs
– Query AWS to get info and compare info to match CSRs
(optional) Out of Scope:
Detail about what is specifically not being delivered in the story
Engineering Details:
- (optional) https://github/com/link.to.enhancement/
- (optional) https://issues.redhat.com/link.to.spike
- Engineering detail 1
- Engineering detail 2
This requires/does not require a design proposal.
This requires/does not require a feature gate.
Goal
- Codify and enable usage of a prototype for HCP working with karpetner management side.
Why is this important?
- A first usable version is critical to democratize knowledge and develop internal feedback.
Acceptance Criteria
- Deploying a cluster with --auto-node results in karpenter running management side, the CRDs and a default ec2NodeClass installed within the guest cluster
- ...
Dependencies (internal and external)
- ...
Previous Work (Optional):
- …
Open questions:
- …
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Technical Enablement <link to Feature Enablement Presentation>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Enhancement merged: <link to meaningful PR or GitHub Issue>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
Goal Summary
This feature aims to make sure that the HyperShift operator and the control-plane it deploys uses Managed Service Identities (MSI) and have access to scoped credentials (also via access to AKS's image gallery potentially). Additionally, for operators deployed in customers account (system components), they would be scoped with Azure workload identities.
Problem
Today Azure installation requires manually created service principal which involves relations, permission granting, credential setting, credential storage, credentials rotation, credentials clean up, and service principal deletion. This is not only mundane and time-consuming but also less secure and risks access to resources by adversaries due to lack of credential rotation.
Goal
Employ Azure managed credentials which drastically reduce the steps required to just managed identity creation, permission granting, and resource deletion.
Ideally, this should be a HyperShift-native functionality. I.e., HyperShift should use managed identities for the control plane, the kubelet, and any add-on that needs access to Azure resources.
Goal
- Each controller within HyperShift must define the permissions required in Azure similar to what is defined for AWS
Why is this important?
- The ARO HCP service will have Azure built in roles these are the equivalent of AWS managed policies.
Scenarios
- ...
Acceptance Criteria
- Dev - Has a valid enhancement if necessary
- CI - MUST be running successfully with tests automated
- QE - covered in Polarion test plan and tests implemented
- Release Technical Enablement - Must have TE slides
- Each HyperShift controller has a defined set for Azure permissions
Dependencies (internal and external)
- ...
Previous Work (Optional):
- …
Open questions:
- …
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Technical Enablement <link to Feature Enablement Presentation>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Enhancement merged: <link to meaningful PR or GitHub Issue>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
Currently, we were using contributor role for all control plane identities on ARO HCP. We should go ahead and restrict the identities we know there are existing ARO roles for already.
Epic Goal
The Cluster Ingress Operator can authenticate with Service Principal backed by a certificate stored in an Azure Key Vault. The Secrets CSI driver will be used to mount the certificate as a volume on the image registry deployment in a hosted control plane.
Why is this important?
- This is needed to enable authentication with Service Principal with backing certificates for ARO HCP.
Acceptance Criteria
- Cluster Ingress Operator is able to authenticate with Azure in ARO HCP using Service Principal with a backing certificate.
- Updated documentation
- ARO HCP CI coverage
Dependencies (internal and external)
Azure SDK
Previous Work (Optional):
Open questions:
Which degree of coverage should run on AKS e2e vs on existing e2es
Done Checklist
CI - Existing CI is running, tests are automated and merged.
CI - AKS CI is running, tests are automated and merged.
DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
DEV - Downstream build attached to advisory: <link to errata>
QE - Test plans in Polarion: <link or reference to Polarion>
QE - Automated tests merged: <link or reference to automated tests>
DOC - Downstream documentation merged: <link to meaningful PR>
Epic Goal
The Cluster Network Operator can authenticate with Service Principal backed by a certificate stored in an Azure Key Vault. The Secrets CSI driver will be used to mount the certificate as a volume on the image registry deployment in a hosted control plane.
Why is this important?
- This is needed to enable authentication with Service Principal with backing certificates for ARO HCP.
Acceptance Criteria
- Cluster Network Operator is able to authenticate with Azure in ARO HCP using Service Principal with a backing certificate.
- Updated documentation
- ARO HCP CI coverage
Dependencies (internal and external)
Azure SDK
Previous Work (Optional):
Open questions:
Which degree of coverage should run on AKS e2e vs on existing e2es
Done Checklist
CI - Existing CI is running, tests are automated and merged.
CI - AKS CI is running, tests are automated and merged.
DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
DEV - Downstream build attached to advisory: <link to errata>
QE - Test plans in Polarion: <link or reference to Polarion>
QE - Automated tests merged: <link or reference to automated tests>
DOC - Downstream documentation merged: <link to meaningful PR>
Feature Overview
- This feature is to track automation in ODC, related packages, upgrades and some tech debts
Goals
- Improve automation for Pipelines dynamic plugins
- Improve automation for OpenShift Developer console
- Engineering tech debt for ODC
Requirements
- This Section:* A list of specific needs or objectives that a Feature must deliver to satisfy the Feature.. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts. If a non MVP requirement slips, it does not shift the feature.
| Requirement | Notes | isMvp? |
|---|---|---|
| CI - MUST be running successfully with test automation | This is a requirement for ALL features. | YES |
| Release Technical Enablement | Provide necessary release enablement details and documents. | YES |
(Optional) Use Cases
This Section:
- Main success scenarios - high-level user stories
- Alternate flow/scenarios - high-level user stories
- ...
Questions to answer...
- ...
Out of Scope
- ...
Background, and strategic fit
This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.
Assumptions
- ...
Customer Considerations
- ...
Documentation Considerations
Questions to be addressed:
- What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
- Does this feature have doc impact?
- New Content, Updates to existing content, Release Note, or No Doc Impact
- If unsure and no Technical Writer is available, please contact Content Strategy.
- What concepts do customers need to understand to be successful in [action]?
- How do we expect customers will use the feature? For what purpose(s)?
- What reference material might a customer want/need to complete [action]?
- Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
- What is the doc impact (New Content, Updates to existing content, or Release Note)?
Problem:
This epic covers the scope of automation-related stories in ODC
Goal:
Why is it important?
Automation enhancements for ODC
Use cases:
- <case>
Acceptance criteria:
- Automation enhancements to improve test execution time and efficiency
- Improving test execution to get more tests run on CI
Dependencies (External/Internal):
Design Artifacts:
Exploration:
Note:
BU Priority Overview
To date our work within the telecommunications radio access network space has focused primarily on x86-based solutions. Industry trends around sustainability, and more specific discussions with partners and customers, indicate a desire to progress towards ARM-based solutions with a view to production deployments in roughly a 2025 timeframe. This would mean being able to support one or more RAN partners DU applications on ARM-based servers.
Goals
- Introduce ARM CPUs for RAN DU scenario (SNO deployment) with a feature parity to Intel Ice Lake/SPR-EE/SPR-SP w/o QAT for DU with:
- STD-kernel (RT-Kernel is not supported by RHEL)
- SR-IOV and DPDK over SR-IOV
- PTP (OC, BC). Partner asked for LLS-C3, according to Fujitsu - ptp4l and phy2sys to work with NVIDIA Aerial SDK
- Characterize ARM-based RAN DU solution performance and power metrics (unless performance parameters are specified by partners, we should propose them, see Open questions)
- Productize ARM-based RAN DU solution by 2024 (partner’s expectation).
State of the Business
Depending on source 75-85% of service provider network power consumption is attributable to the RAN sites, with data centers making up the remainder. This means that in the face of increased downward pressure on both TCO and carbon footprint (the former for company performance reasons, the later for regulatory reasons) it is an attractive place to make substantial improvements using economies of scale.
There are currently three main obvious thrusts to how to go about this:
- Introducing tools that improve overall observability w.r.t. power utilization of the network.
- Improvement of existing RAN architectures via smarter orchestration of workloads, fine tuning hardware utilization on a per site basis in response to network usage, etc.
- Introducing alternative architectures which have been designed from the ground up with lower power utilization as a goal.
This BU priority focuses on the third of these approaches.
BoM
- Details per partner in TELCOSTRAT-99 and TELCOSTRAT-210 ARM requirements for DU
- SuperMicro ARS-111GL-NHR
- NVidia GraceHopper CPU
- 2 units of BlueField-3 DPU (in NIC mode)
- Single NUMA
- GPU - 1 NVidia GraceHopper GPU (GH200)
Out of scope
- …
Open questions:
- What are the latency KPIs? Do we need a RT-kernel to meet them?
- What page size is expected?
- What are the performance/throughout requirements?
Reference Documents:
- Softbank AI-RAN PoC Architecture
- Fujitsu vRAN RHOCP Integration
- Red Hat Ecosystem Catalog
- NVIDIA Grace Hopper - Red Hat Ecosystem Catalog (RHEL 9.2 +)
- NVIDIA BlueField-3 200GbE (2x200) - Red Hat Ecosystem Catalog (RHEL 9.4 +)
Epic Goal
The PerformanceProfile currently allows the user to select either the standard kernel (by default) or the realtime kernel, using the realTimeKernel field. However, for some use cases (e.g. Nvidia based ARM server) a kernel with 64k page size is required. This is supported through the MachineConfig kernelType, which currently supports the following options:
- default (standard kernel with 4k pages)
- realtime (realtime kernel with 4k pages)
- 64k-pages (standard kernel with 64k pages)
At some point it is likely that 64k page support will be added to the realtime kernel, which would likely mean another "realtime-64k-pages" option (or similar) would be added.
The purpose of this epic is to allow the 64k-pages (standard kernel with 64k pages) option to be selected in the PerformanceProfile and make it easy to support new kernelTypes added to the MachineConfig. There is a workaround for this today, by applying an additional MachineConfig CR, which overrides the kernelType, but this is awkward for the user.
One option to support this in the PerformanceProfile would be to deprecate the existing realTimeKernel option and replace it with a new kernelType option. The kernelType option would support the same values as the MachineConfig kernelType (i.e. default, realtime, 64k-pages). The old option could be supported for backwards compatibility - attempting to use both options at the same time would be treated as an error. Another option would be to add a new kernelPageSize option (with values like default or 64k) and then internally map that to the MachineConfig kernelType (after validation that the combination of kernel type and page size was allowed).
This will require updates to the customer documentation and to the performance-profile-creator to support the new option.
This will also might require updates to the workloadhints kernel related sections.
Why is this important?
- This makes it easier for the user to select the kernel page size.
Scenarios
- User wants to use standard kernel with default page size (4k).
- User wants to use standard kernel with 64k page size.
- User wants to use realtime kernel with default page size (4k).
- Future: user wants to use realtime kernel with 64k page size.
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement details and documents.
- ...
Dependencies (internal and external)
- N/A
Previous Work (Optional):
- N/A
Open questions::
- Need to decide the best option for specifying the kernel type and kernel page size in the PerformanceProfile.
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
Acceptance criteria:
- Introduce a new kernelPageSize field to the performance profile API:
for x86/amd64, the only valid value is 4K, while for aarch64, the valid values are 4K, 64K. - Adding this to performance-profile.md doc.
- A smart validation should be integrated into the validation webhook:
64k should be used only on aarch64 nodes and is currently supported with a no real-time kernel. Moreover, we should check for invalid inputs on both arch systems. - The default value is 4K if none is specified.
- Modifying the kernel Type selection process, adding the new kernel type 64k-pages alongside default and realtime.
- Verification on an ARM cluster.
