Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1367129 - 'oc label pod' fails
Summary: 'oc label pod' fails
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Command Line Interface
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Fabiano Franz
QA Contact: Xingxing Xia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-15 15:55 UTC by Scott Dodson
Modified: 2016-08-16 20:37 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-16 20:12:05 UTC


Attachments (Terms of Use)

Description Scott Dodson 2016-08-15 15:55:45 UTC
I'm working on automated installation and when I detect that all the metrics components aren't running and we have a failed deployer pod I attempt to re-run the deployer with MODE=refresh. When I do that I get the following error.

OCP 3.3 environment.

[root@ose3-master ~]# oc logs -n openshift-infra metrics-deployer-78dp2
+ deployer_mode=refresh
+ image_prefix=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/
+ image_version=3.3.0
+ master_url=https://kubernetes.default.svc:443
+ [[ 3 == \/ ]]
++ parse_bool false REDEPLOY
++ local v=false
++ '[' false '!=' true -a false '!=' false ']'
++ echo false
+ redeploy=false
+ '[' false == true ']'
+ mode=refresh
+ '[' refresh = redeploy ']'
++ parse_bool false IGNORE_PREFLIGHT
++ local v=false
++ '[' false '!=' true -a false '!=' false ']'
++ echo false
+ ignore_preflight=false
+ cassandra_nodes=1
++ parse_bool true USE_PERSISTENT_STORAGE
++ local v=true
++ '[' true '!=' true -a true '!=' false ']'
++ echo true
+ use_persistent_storage=true
+ cassandra_pv_size=10Gi
+ metric_duration=7
+ user_write_access=false
+ heapster_node_id=nodename
+ metric_resolution=10s
+ project=openshift-infra
+ master_ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+ token_file=/var/run/secrets/kubernetes.io/serviceaccount/token
+ dir=/etc/deploy/_output
+ secret_dir=/secret
+ rm -rf /etc/deploy/_output
+ mkdir -p /etc/deploy/_output
+ chmod 700 /etc/deploy/_output
+ mkdir -p /secret
+ chmod 700 /secret
chmod: changing permissions of '/secret': Read-only file system
+ :
+ hawkular_metrics_hostname=hawkular-metrics.apps.example.com
+ hawkular_metrics_alias=hawkular-metrics
+ hawkular_cassandra_alias=hawkular-cassandra
++ date +%s
+ openshift admin ca create-signer-cert --key=/etc/deploy/_output/ca.key --cert=/etc/deploy/_output/ca.crt --serial=/etc/deploy/_output/ca.serial.txt --name=metrics-signer@1471275687
+ '[' -n 1 ']'
+ oc config set-cluster master --api-version=v1 --certificate-authority=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --server=https://kubernetes.default.svc:443
cluster "master" set.
++ cat /var/run/secrets/kubernetes.io/serviceaccount/token
+ oc config set-credentials account --token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtaW5mcmEiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoibWV0cml
jcy1kZXBsb3llci10b2tlbi11anVseSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJtZXRyaWNzLWRlcGxveWVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNzFjOWQ1MTItNjBjZS0xMWU2LWJhZWYtNTI1NDAwOWNjMTlhIiwic3ViIjoic3lzdGVtOnNlcnZ
pY2VhY2NvdW50Om9wZW5zaGlmdC1pbmZyYTptZXRyaWNzLWRlcGxveWVyIn0.wWVIy4aDKXM6IlpBZFrzmli-tOagyEsM0Em9kqSaTSXT0VUMLV7laWiDkSJ7LXWaK9nJAyQevk0b5zzgHj68fXGu5CR7U5L_EBKo3dp-lPQsxvb10nCBsHeL2EjW686lviDyMD4toJeluGlUYXK_E_LIM4WxiRvJGUhJOuzVR_D5WszpeEipFQfBTn849Vb25cj2jR7DUBdd8fZB9Z3
TI8V1csMdL4adIEKqfxdOmeYg8nQ1bS01l1v7OD3fVebwj-yPvpUdrNhcV8hPw-uUBUUuzsRpzULM4EGlWWbF21Amf-hpe4tUGr0kDlr0qxPdmM6xo4LaJyObPTXlmPAKpA
user "account" set.
+ oc config set-context current --cluster=master --user=account --namespace=openshift-infra
context "current" set.
+ oc config use-context current
switched to context "current".
+ old_kc=/etc/deploy/.kubeconfig
+ KUBECONFIG=/etc/deploy/_output/kube.conf
+ '[' -z 1 ']'
+ oc config set-cluster deployer-master --api-version=v1 --certificate-authority=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --server=https://kubernetes.default.svc:443
cluster "deployer-master" set.
++ cat /var/run/secrets/kubernetes.io/serviceaccount/token
+ oc config set-credentials deployer-account --token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtaW5mcmEiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIj
oibWV0cmljcy1kZXBsb3llci10b2tlbi11anVseSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJtZXRyaWNzLWRlcGxveWVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNzFjOWQ1MTItNjBjZS0xMWU2LWJhZWYtNTI1NDAwOWNjMTlhIiwic3ViIjoic3lzdG
VtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1pbmZyYTptZXRyaWNzLWRlcGxveWVyIn0.wWVIy4aDKXM6IlpBZFrzmli-tOagyEsM0Em9kqSaTSXT0VUMLV7laWiDkSJ7LXWaK9nJAyQevk0b5zzgHj68fXGu5CR7U5L_EBKo3dp-lPQsxvb10nCBsHeL2EjW686lviDyMD4toJeluGlUYXK_E_LIM4WxiRvJGUhJOuzVR_D5WszpeEipFQfBTn849Vb25cj2jR7DUB
dd8fZB9Z3TI8V1csMdL4adIEKqfxdOmeYg8nQ1bS01l1v7OD3fVebwj-yPvpUdrNhcV8hPw-uUBUUuzsRpzULM4EGlWWbF21Amf-hpe4tUGr0kDlr0qxPdmM6xo4LaJyObPTXlmPAKpA
user "deployer-account" set.
+ oc config set-context deployer-context --cluster=deployer-master --user=deployer-account --namespace=openshift-infra
context "deployer-context" set.
+ '[' -n 1 ']'
+ oc config use-context deployer-context
switched to context "deployer-context".
+ case $deployer_mode in
+ '[' false '!=' true ']'
+ validate_preflight
+ set -e
+ set +x

PREFLIGHT CHECK SUCCEEDED
validate_master_accessible: ok
validate_hostname: The HAWKULAR_METRICS_HOSTNAME value is deemed acceptable.
validate_deployer_secret:
Deleting any previous deployment (leaving route and PVCs)
POD_NAME metrics-deployer-78dp2
The Pod "metrics-deployer-78dp2" is invalid.
spec: Forbidden: pod updates may not change fields other than `containers[*].image` or `spec.activeDeadlineSeconds`

Comment 1 Scott Dodson 2016-08-15 17:28:59 UTC
[root@ose3-master ~]# oc version
oc v3.3.0.19
kubernetes v1.3.0+507d3a7
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ose3-master.example.com:8443
openshift v3.3.0.19
kubernetes v1.3.0+507d3a7

Comment 4 Matt Wringe 2016-08-16 14:20:11 UTC
What is failing is the 'oc label pod' command.

Can you please verify that you can run this command against another pod in your system?

eg "oc label pod ${POD_NAME} foo=bar"

If this command fails then we need to notify the team responsible for the oc commands.

Comment 5 Matt Wringe 2016-08-16 14:22:08 UTC
Actually, I am just going to assign this to the CLI team. I can't see this being something wrong with the metric pods.

Comment 6 Scott Dodson 2016-08-16 20:12:05 UTC
Happened with oc v3.3.0.12 versus v3.3.0.21 API. Closing as not a bug as we don't assure compat between  pre-release builds.

Comment 7 Fabiano Franz 2016-08-16 20:37:45 UTC
Thanks Scott, just for the record then although I didn't use the specified IMAGE_PREFIX and IMAGE_VERSION, I couldn't reproduce the issue.

-----
$ oc new-app -f metrics-deployer.yaml -p HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.apps.127.0.0.1.xip.io -p USE_PERSISTENT_STORAGE=false -p MODE=refresh -p CASSANDRA_NODES=1 -p CASSANDRA_PV_SIZE=10 -p USER_WRITE_ACCESS=true

$ oc logs metrics-deployer-fg2pg -f
(...)
PREFLIGHT CHECK SUCCEEDED
validate_master_accessible: ok
validate_hostname: The HAWKULAR_METRICS_HOSTNAME value is deemed acceptable.
validate_deployer_secret: ok
Deleting any previous deployment (leaving route and PVCs)
POD_NAME metrics-deployer-fg2pg
pod "metrics-deployer-fg2pg" labeled
(...)
-----

In your logs I see the following line:

'The Pod "metrics-deployer-78dp2" is invalid.'

Might be related to the issue. 'oc' won't be able to label the pod if it's invalid.


Note You need to log in before you can comment on or make changes to this bug.