| *** zhangfei has joined #openstack-helm | 00:39 | |
| *** julim has joined #openstack-helm | 00:49 | |
| *** zhangfei has quit IRC | 00:53 | |
| *** zhangfei has joined #openstack-helm | 01:10 | |
| *** gagehugo has joined #openstack-helm | 01:22 | |
| *** gagehugo has quit IRC | 01:31 | |
| *** gagehugo has joined #openstack-helm | 01:36 | |
| *** roman_g has quit IRC | 02:02 | |
| *** yamamoto has quit IRC | 02:04 | |
| *** yamamoto has joined #openstack-helm | 02:04 | |
| openstackgerrit | Clare cheng proposed openstack/openstack-helm master: update version to latest https://review.openstack.org/570830 | 02:07 |
|---|---|---|
| *** SlickNik has quit IRC | 02:44 | |
| *** spsurya has quit IRC | 02:44 | |
| *** portdirect has quit IRC | 02:44 | |
| *** SlickNik has joined #openstack-helm | 02:45 | |
| *** spsurya has joined #openstack-helm | 02:45 | |
| *** portdirect has joined #openstack-helm | 02:45 | |
| *** barjavel.freenode.net sets mode: +o portdirect | 02:45 | |
| *** yamamoto has quit IRC | 02:46 | |
| *** openstackgerrit has quit IRC | 02:46 | |
| *** pcarver has quit IRC | 02:46 | |
| *** kgz has quit IRC | 02:46 | |
| *** anticw has quit IRC | 02:46 | |
| *** yamamoto has joined #openstack-helm | 02:48 | |
| *** openstackgerrit has joined #openstack-helm | 02:48 | |
| *** pcarver has joined #openstack-helm | 02:48 | |
| *** kgz has joined #openstack-helm | 02:48 | |
| *** anticw has joined #openstack-helm | 02:48 | |
| *** Guest157` has quit IRC | 02:50 | |
| *** flaviosr_ has quit IRC | 02:50 | |
| *** zhangfei has quit IRC | 02:50 | |
| *** sar_ has quit IRC | 02:50 | |
| *** cfriesen has quit IRC | 02:50 | |
| *** gagehugo has quit IRC | 02:50 | |
| *** ianychoi_ has quit IRC | 02:50 | |
| *** gmmaha has quit IRC | 02:50 | |
| *** cloudnull has quit IRC | 02:50 | |
| *** Guest157` has joined #openstack-helm | 02:50 | |
| *** flaviosr_ has joined #openstack-helm | 02:50 | |
| *** gagehugo has joined #openstack-helm | 02:51 | |
| *** ianychoi_ has joined #openstack-helm | 02:51 | |
| *** gmmaha has joined #openstack-helm | 02:51 | |
| *** cloudnull has joined #openstack-helm | 02:51 | |
| *** gmmaha has quit IRC | 02:51 | |
| *** gmmaha has joined #openstack-helm | 02:51 | |
| *** zhangfei has joined #openstack-helm | 02:51 | |
| *** sar_ has joined #openstack-helm | 02:51 | |
| *** cfriesen has joined #openstack-helm | 02:51 | |
| *** mdih has joined #openstack-helm | 04:36 | |
| *** sar_ has quit IRC | 05:47 | |
| *** sar_ has joined #openstack-helm | 06:19 | |
| tdoc | Hi, I'm trying to deploy a multinode setup on top of our own k8s, but run into some issue with the rabbitmq charts. The pod fails to start and when looking at the container log, it seems the rabbitmq-server is failing to do a reverse DNS lookup on its hostname. | 06:20 |
| tdoc | What I've noticed is that it works fine if I remove the readiness/livelyness probes from the statefulset. | 06:21 |
| tdoc | This suggests some kind of chicken-egg problem. It looks like kubernetes is not adding the DNS record while the pod is not ready, but the pod can't start because it needs this DNS record. | 06:22 |
| tdoc | Has anyone seen something like that? | 06:22 |
| portdirect | tdoc: this should work via the 'discovery' service | 06:23 |
| portdirect | Are you disabling pod name resolution in the DNS service? | 06:24 |
| tdoc | No, I don't think we do... At least other DNS records do work fine... | 06:25 |
| tdoc | Intially the pod complains that it can't lookup rabbitmq-rabbitmq-0.rabbitmq-dsv-7b1733.openstack.svc.cluster.local | 06:26 |
| portdirect | Hmm | 06:26 |
| portdirect | This should be ok: http://logs.openstack.org/28/569328/3/check/openstack-helm-multinode-ubuntu/6e2722b/primary/helm/rabbitmq.txt.gz | 06:26 |
| portdirect | http://logs.openstack.org/28/569328/3/check/openstack-helm-multinode-ubuntu/6e2722b/primary/pod-logs/openstack/rabbitmq-rabbitmq-0/rabbitmq.txt.gz | 06:27 |
| tdoc | Yeah, it's strange. It suggests that somehow our kubernetes works differently with regards to DNS, but I'm not sure how to track that down. | 06:29 |
| portdirect | What are you using? | 06:30 |
| portdirect | Ie k8s version and dns implementation | 06:30 |
| tdoc | k8s is v1.10.2 | 06:31 |
| portdirect | All our upstream gates use 1.10.x and kube-dns, though coredns should be ok | 06:31 |
| tdoc | We use coredns. | 06:32 |
| tdoc | So maybe there is a subtle difference there? | 06:33 |
| portdirect | Poss try adding a toleration for unready endpoints here: https://github.com/openstack/openstack-helm/blob/master/rabbitmq/templates/service-discovery.yaml | 06:33 |
| portdirect | But it shouldnt be required :/ | 06:33 |
| portdirect | Poss some lag in your env for dns to come up | 06:34 |
| tdoc | Could be I guess, I'll look into that toleration. And maybe see if we could test with kube-dns. Not really familiar with how to change that, but I can find out. | 06:36 |
| *** cNilesh has joined #openstack-helm | 06:36 | |
| *** radek__ has joined #openstack-helm | 06:39 | |
| tdoc | hmm, I tried adding publishNotReadyAddresses: true to service-discovery.yaml, but I don't even see it in the definition once I deploy it. Not sure what I'm doing wrong here. | 07:05 |
| *** roman_g has joined #openstack-helm | 07:30 | |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: Implement nginx sidecar https://review.openstack.org/555386 | 07:34 |
| *** ianychoi_ is now known as ianychoi | 08:40 | |
| tdoc | @portdirect: I tried with kube-dns just now, and with publishNotReadyAddresses enabled on the service. But still the same problem. | 09:15 |
| *** zhangfei has quit IRC | 09:20 | |
| *** zhangfei has joined #openstack-helm | 09:31 | |
| tdoc | hmm, publishNotReadyAddresses is broken, apparently, with service.alpha.kubernetes.io/tolerate-unready-endpoints it works | 10:00 |
| tdoc | Still doesn't explain why I need to add that though. | 10:03 |
| *** yamamoto has quit IRC | 10:10 | |
| *** goutham has joined #openstack-helm | 10:44 | |
| goutham | Hi all im landing into this issue https://hastebin.com/nakiqucufu.sql while deploying promenade anyone has idea on this issue ?? | 10:45 |
| goutham | this is because of the ceph directory division into 3 sub directories any idea on how to fix this ?? | 10:50 |
| *** yamamoto has joined #openstack-helm | 11:11 | |
| *** yamamoto has quit IRC | 11:16 | |
| *** caoyuan has joined #openstack-helm | 11:26 | |
| *** caoyuan has quit IRC | 11:30 | |
| *** zhangfei has quit IRC | 11:31 | |
| *** yamamoto has joined #openstack-helm | 11:36 | |
| *** yamamoto has quit IRC | 11:47 | |
| *** yamamoto has joined #openstack-helm | 11:48 | |
| *** sar_ has quit IRC | 12:28 | |
| *** sar_ has joined #openstack-helm | 12:41 | |
| *** mihdih_ has joined #openstack-helm | 12:58 | |
| *** mdih has quit IRC | 12:59 | |
| *** sthussey has joined #openstack-helm | 13:30 | |
| openstackgerrit | Merged openstack/openstack-helm-addons master: Basic addons gate https://review.openstack.org/568991 | 13:38 |
| *** piowac has joined #openstack-helm | 13:41 | |
| *** piowac has left #openstack-helm | 13:41 | |
| *** goutham has quit IRC | 13:54 | |
| *** piotrrr has joined #openstack-helm | 13:59 | |
| *** jistr is now known as jistr|mtg | 14:36 | |
| *** felipemonteiro has joined #openstack-helm | 14:51 | |
| *** jistr|mtg is now known as jistr | 14:52 | |
| mattmceuen | GM all - team meeting in 8min, please add anything you'd like to discuss to here: https://etherpad.openstack.org/p/openstack-helm-meeting-2018-05-29 | 14:52 |
| mattmceuen | @gautham, can you try sharing that error in the #airshipit chat room? | 14:53 |
| mattmceuen | Oh looks like you did already | 14:53 |
| *** felipemonteiro_ has joined #openstack-helm | 14:54 | |
| tdoc | i'm having some DNS related issues with rabbitmq, is that something for the meeting or not? (new here) | 14:54 |
| mattmceuen | hey tdoc - we can bring it up in Roundtable in the meeting for sure. It's a good general chat item but we also have a light agenda today :) | 14:55 |
| mattmceuen | I see you brought it up already, that is good. portdirect is on vacation and I just got IRCCloud working again after the long weekend, so we're a bit behind on IRC | 14:56 |
| tdoc | ok, will raise it then | 14:56 |
| tdoc | sure, no problem.. | 14:58 |
| *** felipemonteiro has quit IRC | 14:58 | |
| tdoc | as mentioned above, I need to add service.alpha.kubernetes.io/tolerate-unready-endpoints to make it work | 14:58 |
| tdoc | so at this point, I'm not sure if I should file a bug or not, since the problem doesn't seem to be happing in the gating system | 14:59 |
| *** felipemonteiro_ is now known as felipemonteiro | 14:59 | |
| mattmceuen | I'd say go ahead and file a bug to track it - if it ends up not being a problem at least we can get a paper trail on it | 15:00 |
| mattmceuen | In case someone runs into the same issue in the future | 15:00 |
| tdoc | ok, makes sense. | 15:00 |
| *** pp___ has joined #openstack-helm | 15:00 | |
| mattmceuen | Meeting time | 15:00 |
| *** Guest157` is now known as d|k | 15:01 | |
| *** d|k is now known as Guest57469 | 15:02 | |
| *** Guest57469 is now known as d|k` | 15:02 | |
| *** MarkBaker has joined #openstack-helm | 15:23 | |
| *** felipemonteiro has quit IRC | 15:23 | |
| *** felipemonteiro has joined #openstack-helm | 15:24 | |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm-infra master: Ingress TLS for keystone https://review.openstack.org/568132 | 15:34 |
| *** goutham has joined #openstack-helm | 15:37 | |
| *** goutham has left #openstack-helm | 15:41 | |
| *** pp___ has quit IRC | 15:47 | |
| *** MarkBaker has quit IRC | 15:50 | |
| *** schwicht has joined #openstack-helm | 15:56 | |
| *** d|k` is now known as d|k | 16:00 | |
| tdoc | I'll see if I can open an issue in launchpad for that DNS thing in the morning. (Day is about to end over here.) | 16:01 |
| *** cNilesh has quit IRC | 16:32 | |
| *** mihdih_ has quit IRC | 16:51 | |
| openstackgerrit | Chris Wedgwood proposed openstack/openstack-helm-addons master: artifactory: Update to 5.11.1 https://review.openstack.org/570526 | 17:22 |
| openstackgerrit | Chris Wedgwood proposed openstack/openstack-helm-addons master: artifactory: Update to 5.11.1 https://review.openstack.org/570526 | 17:25 |
| *** gmmaha has quit IRC | 17:35 | |
| *** gmmaha has joined #openstack-helm | 17:37 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-addons master: Fix minor chart issues due to osh changes https://review.openstack.org/567672 | 17:37 |
| *** schwicht has quit IRC | 17:39 | |
| *** felipemonteiro_ has joined #openstack-helm | 18:40 | |
| *** felipemonteiro has quit IRC | 18:44 | |
| *** cfriesen_ has joined #openstack-helm | 18:47 | |
| *** felipemonteiro_ has quit IRC | 18:51 | |
| *** felipemonteiro_ has joined #openstack-helm | 18:51 | |
| *** cfriesen has quit IRC | 18:52 | |
| *** mihdih_ has joined #openstack-helm | 19:06 | |
| *** radek__ has quit IRC | 19:10 | |
| *** gmmaha has quit IRC | 19:13 | |
| *** goldenfri has joined #openstack-helm | 19:19 | |
| *** cfriesen_ is now known as cfriesen | 19:20 | |
| *** marshallmargenau has joined #openstack-helm | 19:34 | |
| *** gmmaha has joined #openstack-helm | 19:35 | |
| *** felipemonteiro_ has quit IRC | 19:49 | |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm-infra master: Ingress TLS for keystone https://review.openstack.org/568132 | 20:23 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: Ingress TLS for keystone https://review.openstack.org/566350 | 20:28 |
| openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP Test/Fix Ceph deployment with diff namespace name https://review.openstack.org/568888 | 21:44 |
| rwellum | mattmceuen: added quite a bit of debugging info to: https://etherpad.openstack.org/p/openstack-helm-troubleshooting FYI | 21:54 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: Add PURE backend config to cinder and nova charts https://review.openstack.org/569165 | 22:25 |
| mattmceuen | thanks rwellum, that's good stuff - if we throw some markup at it that's a lot of the doc right there! | 23:06 |
| *** mihdih_ has quit IRC | 23:38 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!