Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1367129 - 'oc label pod' fails
Summary: 'oc label pod' fails
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Command Line Interface
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: ---
Assignee: Fabiano Franz
QA Contact: Xingxing Xia
Depends On:
TreeView+ depends on / blocked
Reported: 2016-08-15 15:55 UTC by Scott Dodson
Modified: 2016-08-16 20:37 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2016-08-16 20:12:05 UTC

Attachments (Terms of Use)

Description Scott Dodson 2016-08-15 15:55:45 UTC
I'm working on automated installation and when I detect that all the metrics components aren't running and we have a failed deployer pod I attempt to re-run the deployer with MODE=refresh. When I do that I get the following error.

OCP 3.3 environment.

[root@ose3-master ~]# oc logs -n openshift-infra metrics-deployer-78dp2
+ deployer_mode=refresh
+ image_version=3.3.0
+ master_url=https://kubernetes.default.svc:443
+ [[ 3 == \/ ]]
++ parse_bool false REDEPLOY
++ local v=false
++ '[' false '!=' true -a false '!=' false ']'
++ echo false
+ redeploy=false
+ '[' false == true ']'
+ mode=refresh
+ '[' refresh = redeploy ']'
++ parse_bool false IGNORE_PREFLIGHT
++ local v=false
++ '[' false '!=' true -a false '!=' false ']'
++ echo false
+ ignore_preflight=false
+ cassandra_nodes=1
++ parse_bool true USE_PERSISTENT_STORAGE
++ local v=true
++ '[' true '!=' true -a true '!=' false ']'
++ echo true
+ use_persistent_storage=true
+ cassandra_pv_size=10Gi
+ metric_duration=7
+ user_write_access=false
+ heapster_node_id=nodename
+ metric_resolution=10s
+ project=openshift-infra
+ master_ca=/var/run/secrets/
+ token_file=/var/run/secrets/
+ dir=/etc/deploy/_output
+ secret_dir=/secret
+ rm -rf /etc/deploy/_output
+ mkdir -p /etc/deploy/_output
+ chmod 700 /etc/deploy/_output
+ mkdir -p /secret
+ chmod 700 /secret
chmod: changing permissions of '/secret': Read-only file system
+ :
+ hawkular_metrics_alias=hawkular-metrics
+ hawkular_cassandra_alias=hawkular-cassandra
++ date +%s
+ openshift admin ca create-signer-cert --key=/etc/deploy/_output/ca.key --cert=/etc/deploy/_output/ca.crt --serial=/etc/deploy/_output/ca.serial.txt --name=metrics-signer@1471275687
+ '[' -n 1 ']'
+ oc config set-cluster master --api-version=v1 --certificate-authority=/var/run/secrets/ --server=https://kubernetes.default.svc:443
cluster "master" set.
++ cat /var/run/secrets/
+ oc config set-credentials account --token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtaW5mcmEiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoibWV0cml
user "account" set.
+ oc config set-context current --cluster=master --user=account --namespace=openshift-infra
context "current" set.
+ oc config use-context current
switched to context "current".
+ old_kc=/etc/deploy/.kubeconfig
+ KUBECONFIG=/etc/deploy/_output/kube.conf
+ '[' -z 1 ']'
+ oc config set-cluster deployer-master --api-version=v1 --certificate-authority=/var/run/secrets/ --server=https://kubernetes.default.svc:443
cluster "deployer-master" set.
++ cat /var/run/secrets/
+ oc config set-credentials deployer-account --token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtaW5mcmEiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIj
user "deployer-account" set.
+ oc config set-context deployer-context --cluster=deployer-master --user=deployer-account --namespace=openshift-infra
context "deployer-context" set.
+ '[' -n 1 ']'
+ oc config use-context deployer-context
switched to context "deployer-context".
+ case $deployer_mode in
+ '[' false '!=' true ']'
+ validate_preflight
+ set -e
+ set +x

validate_master_accessible: ok
validate_hostname: The HAWKULAR_METRICS_HOSTNAME value is deemed acceptable.
Deleting any previous deployment (leaving route and PVCs)
POD_NAME metrics-deployer-78dp2
The Pod "metrics-deployer-78dp2" is invalid.
spec: Forbidden: pod updates may not change fields other than `containers[*].image` or `spec.activeDeadlineSeconds`

Comment 1 Scott Dodson 2016-08-15 17:28:59 UTC
[root@ose3-master ~]# oc version
oc v3.3.0.19
kubernetes v1.3.0+507d3a7
features: Basic-Auth GSSAPI Kerberos SPNEGO

openshift v3.3.0.19
kubernetes v1.3.0+507d3a7

Comment 4 Matt Wringe 2016-08-16 14:20:11 UTC
What is failing is the 'oc label pod' command.

Can you please verify that you can run this command against another pod in your system?

eg "oc label pod ${POD_NAME} foo=bar"

If this command fails then we need to notify the team responsible for the oc commands.

Comment 5 Matt Wringe 2016-08-16 14:22:08 UTC
Actually, I am just going to assign this to the CLI team. I can't see this being something wrong with the metric pods.

Comment 6 Scott Dodson 2016-08-16 20:12:05 UTC
Happened with oc v3.3.0.12 versus v3.3.0.21 API. Closing as not a bug as we don't assure compat between  pre-release builds.

Comment 7 Fabiano Franz 2016-08-16 20:37:45 UTC
Thanks Scott, just for the record then although I didn't use the specified IMAGE_PREFIX and IMAGE_VERSION, I couldn't reproduce the issue.

$ oc new-app -f metrics-deployer.yaml -p -p USE_PERSISTENT_STORAGE=false -p MODE=refresh -p CASSANDRA_NODES=1 -p CASSANDRA_PV_SIZE=10 -p USER_WRITE_ACCESS=true

$ oc logs metrics-deployer-fg2pg -f
validate_master_accessible: ok
validate_hostname: The HAWKULAR_METRICS_HOSTNAME value is deemed acceptable.
validate_deployer_secret: ok
Deleting any previous deployment (leaving route and PVCs)
POD_NAME metrics-deployer-fg2pg
pod "metrics-deployer-fg2pg" labeled

In your logs I see the following line:

'The Pod "metrics-deployer-78dp2" is invalid.'

Might be related to the issue. 'oc' won't be able to label the pod if it's invalid.

Note You need to log in before you can comment on or make changes to this bug.