| bryan_att | anticw: these servers are fresh installed before every test run using MAAS and Ubuntu 16.04. The OSH script is the first thing that runs on them | 00:28 |
|---|---|---|
| openstackgerrit | Dae Seong Kim proposed openstack/openstack-helm master: Add Tempest script in helm test framework https://review.openstack.org/476733 | 00:33 |
| *** randomhack has joined #openstack-helm | 00:33 | |
| *** renmak_ has quit IRC | 00:33 | |
| *** renmak__ has quit IRC | 00:33 | |
| *** renmak__ has joined #openstack-helm | 00:41 | |
| *** renmak_ has joined #openstack-helm | 00:41 | |
| *** renmak__ has quit IRC | 00:42 | |
| *** renmak_ has quit IRC | 00:42 | |
| *** gouthamr has quit IRC | 00:48 | |
| *** randomhack has quit IRC | 00:53 | |
| *** larainema has quit IRC | 00:58 | |
| *** randomhack has joined #openstack-helm | 01:46 | |
| *** randomhack has quit IRC | 02:03 | |
| *** larainema has joined #openstack-helm | 02:55 | |
| *** mmehan has quit IRC | 03:18 | |
| japestinho | Hi portdirect, here is the log from kube-contoller-manager and I see it keeps repeating | 03:24 |
| japestinho | https://www.irccloud.com/pastebin/5hf5FrsB/ | 03:24 |
| *** renmak_ has joined #openstack-helm | 03:45 | |
| *** renmak__ has joined #openstack-helm | 03:45 | |
| *** felipemonteiro has quit IRC | 03:55 | |
| *** renmak_ has quit IRC | 03:59 | |
| *** renmak__ has quit IRC | 03:59 | |
| openstackgerrit | Stacey Fletcher proposed openstack/openstack-helm master: WIP: Refactor Basic Launch https://review.openstack.org/491903 | 04:12 |
| *** renmak__ has joined #openstack-helm | 04:55 | |
| *** renmak_ has joined #openstack-helm | 04:55 | |
| *** randomhack has joined #openstack-helm | 05:00 | |
| *** renmak__ has quit IRC | 05:00 | |
| *** renmak__ has joined #openstack-helm | 05:01 | |
| *** randomhack has quit IRC | 05:04 | |
| *** renmak_ has quit IRC | 05:04 | |
| *** renmak_ has joined #openstack-helm | 05:04 | |
| *** spsurya has joined #openstack-helm | 05:05 | |
| *** renmak__ has quit IRC | 06:36 | |
| *** renmak_ has quit IRC | 06:38 | |
| openstackgerrit | Mateusz Blaszkowski proposed openstack/openstack-helm-addons master: Elasticsearch: configuring log rotation https://review.openstack.org/492013 | 07:51 |
| *** zioproto has quit IRC | 08:47 | |
| *** rwellum has quit IRC | 08:47 | |
| *** larainema has quit IRC | 08:47 | |
| *** julim has quit IRC | 08:47 | |
| *** mariusv has quit IRC | 08:47 | |
| *** cloudnull has quit IRC | 08:47 | |
| *** sghosh has quit IRC | 08:47 | |
| *** dims has quit IRC | 08:47 | |
| *** lamt has quit IRC | 08:47 | |
| *** alraddarla_ has quit IRC | 08:47 | |
| *** csuttles has quit IRC | 08:47 | |
| *** dulek has quit IRC | 08:47 | |
| *** osh-chatbot has quit IRC | 08:47 | |
| *** cheetopet has quit IRC | 08:47 | |
| *** anticw has quit IRC | 08:47 | |
| *** bradjones has quit IRC | 08:47 | |
| *** redondo-mk has quit IRC | 08:47 | |
| *** bryan_att has quit IRC | 08:47 | |
| *** hogepodge has quit IRC | 08:47 | |
| *** SamYaple has quit IRC | 08:47 | |
| *** MarkBaker has quit IRC | 08:47 | |
| *** nkp349 has quit IRC | 08:47 | |
| *** mcnanci has quit IRC | 08:47 | |
| *** serverascode has quit IRC | 08:47 | |
| *** srwilkers_ has quit IRC | 08:47 | |
| *** RuiChen has quit IRC | 08:47 | |
| *** cargonza has quit IRC | 08:47 | |
| *** srwilkers has quit IRC | 08:47 | |
| *** v1k0d3n has quit IRC | 08:47 | |
| *** leifmadsen has quit IRC | 08:47 | |
| *** aimeeu has quit IRC | 08:47 | |
| *** portdirect has quit IRC | 08:47 | |
| *** danpawlik has quit IRC | 08:47 | |
| *** andreaf has quit IRC | 08:47 | |
| *** evrardjp has quit IRC | 08:47 | |
| *** alanmeadows has quit IRC | 08:47 | |
| *** gagehugo has quit IRC | 08:47 | |
| *** dansmith has quit IRC | 08:47 | |
| *** kragniz has quit IRC | 08:47 | |
| *** spsurya has quit IRC | 08:47 | |
| *** jistr has quit IRC | 08:47 | |
| *** openstackgerrit has quit IRC | 08:47 | |
| *** xek has quit IRC | 08:47 | |
| *** japestinho has quit IRC | 08:47 | |
| *** jayahn has quit IRC | 08:47 | |
| *** openstack has joined #openstack-helm | 13:53 | |
| *** felipemonteiro has quit IRC | 13:56 | |
| *** marst has joined #openstack-helm | 14:09 | |
| *** felipemonteiro has joined #openstack-helm | 14:20 | |
| portdirect | hey peeps: would really appreciate some fb on this https://review.openstack.org/481234, I think its now good to go. | 14:34 |
| portdirect | http://logs.openstack.org/34/481234/40/check/gate-openstack-helm-multi-armada-ubuntu-xenial-3-node-nv/2414b5f/ | 14:34 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Armada OpenStack deployment yaml https://review.openstack.org/481234 | 14:40 |
| *** gouthamr has joined #openstack-helm | 14:47 | |
| alraddarla | maybe if you'd stop pushing new changes to it portdirect :P | 14:48 |
| portdirect | fair - just saw an obvious optimisation | 14:49 |
| portdirect | https://review.openstack.org/#/c/481234/40..41/tools/gate/armada_launch.sh was a but yucky before. | 14:49 |
| alraddarla | makes sense i was just messing with ya | 14:51 |
| *** felipemonteiro has quit IRC | 14:58 | |
| alraddarla | portdirect, are there any docs to support this? | 14:58 |
| openstackgerrit | Kaspars Skels proposed openstack/openstack-helm master: DNM: Test gerrit trigger for glance https://review.openstack.org/492170 | 15:14 |
| v1k0d3n | nice job portdirect | 15:17 |
| v1k0d3n | glad to see armada starting to make it in the gate | 15:17 |
| v1k0d3n | i'm going to start handing out some work in our group re: armada and the gating scripts so we can start gating internally as well. in fact, been talking with sk about this as well, and looking to collaborate soon. | 15:17 |
| v1k0d3n | i will have one of our guys check out that ps...just to learn and pick it up a bit. it's a lot for someone new to wrap their head around, but we have a couple of developers that should be able to come up to speed pretty quickly. | 15:18 |
| *** spsurya has joined #openstack-helm | 15:39 | |
| srwilkers_ | Awesome v1k0d3n :) would be nice to get more eyes on it and the current gate scripts. More people with gate-foo, the better for us all :) | 15:43 |
| *** randomhack has quit IRC | 15:43 | |
| *** randomhack has joined #openstack-helm | 15:48 | |
| *** randomha1k has joined #openstack-helm | 15:58 | |
| osh-chatbot | <v1k0d3n> totally agree @srwilkers :slightly_smiling_face: | 15:59 |
| *** randomhack has quit IRC | 16:00 | |
| bryan_att | anyone - I posted a number of messages yesterday ^^^ with info on issues I am experiencing. Any help is appreciated. | 16:20 |
| *** aric49 has joined #openstack-helm | 16:20 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Use RBD external provisioner https://review.openstack.org/490983 | 16:25 |
| *** randomha1k has quit IRC | 16:29 | |
| *** renmak__ has quit IRC | 17:02 | |
| *** renmak_ has quit IRC | 17:02 | |
| openstackgerrit | Darla Ahlert proposed openstack/openstack-helm master: [WIP] Add Rally Chart https://review.openstack.org/491015 | 17:03 |
| *** renmak_ has joined #openstack-helm | 17:22 | |
| *** renmak__ has joined #openstack-helm | 17:22 | |
| bryan_att | portdirect: srwilkers_ v1k0d3n any input on the items above ^^^ is appreciated. I'm finding that for some reason the worker nodes are unable to connect to the DNS service on the master node. Why does resolve.conf get updated with "nameserver 10.96.0.10" ? | 17:25 |
| *** randomhack has joined #openstack-helm | 17:25 | |
| osh-chatbot | <v1k0d3n> nameserver 10.96.0.10 is updated in your /etc/hosts, so the physical nodes know how to connect to a records inside of the kuberentes cluster. | 17:25 |
| osh-chatbot | <v1k0d3n> this is a requirement for ceph | 17:26 |
| osh-chatbot | <v1k0d3n> as being used in OSH | 17:26 |
| bryan_att | ubuntu@opnfv02:~$ kubectl describe po/ceph-mon-keyring-generator-nzlvt -n ceph | 17:26 |
| bryan_att | Name:ceph-mon-keyring-generator-nzlvt | 17:26 |
| bryan_att | Namespace:ceph | 17:26 |
| bryan_att | Node:opnfv03/204.178.3.197 | 17:26 |
| bryan_att | Start Time:Wed, 09 Aug 2017 16:17:47 +0000 | 17:26 |
| *** bryan_att has quit IRC | 17:26 | |
| *** bryan_att has joined #openstack-helm | 17:27 | |
| bryan_att | v1k0d3n: (paste did not work right for some reason...) but anyway that DNS service is inaccessible at some point from the worker nodes. I need help to understand why and correct it. | 17:28 |
| osh-chatbot | <v1k0d3n> define “unreachable”…. | 17:28 |
| bryan_att | meaning nslookups fail | 17:29 |
| bryan_att | https://www.irccloud.com/pastebin/rLQg4vyr/ | 17:29 |
| bryan_att | that is why the ceph containers can't be started - the images can't be pulled | 17:30 |
| bryan_att | this is very repeatable | 17:30 |
| osh-chatbot | <v1k0d3n> so @bryan_att, you should have multiple nameservers. the scripts afaik are only placing 10.96.0.10 as nameserver 01. do you not have other nameservers you use for your hosts? | 17:32 |
| *** randomhack has quit IRC | 17:32 | |
| bryan_att | the scripts removed the other nameservers. see the /etc/resolv.conf posted. prior to OSH gate scripts there were other nameservers there. | 17:33 |
| osh-chatbot | <v1k0d3n> so i have to be honest at this point. i understand that OSH devs don’t want to troubleshoot kubernetes issues. why are we recommending that users use gate scripts to deploy kubernertes and openstack-helm? | 17:34 |
| osh-chatbot | <v1k0d3n> the goal in the beginning…and to be clear, i haven’t seen any public updates to this…is to allow users to bring a kube cluster. | 17:35 |
| osh-chatbot | <v1k0d3n> and then deploy osh on top of it. | 17:35 |
| osh-chatbot | <v1k0d3n> this is why i think a project like sonobouy could help the OSH team. to just say “we need at least these conformance tests to pass…this rbac (provided by OSH) and this should be ok. | 17:36 |
| osh-chatbot | <v1k0d3n> in some cases, i get it…sdn may play a role. | 17:36 |
| osh-chatbot | <raymaika> @v1k0d3n I think that's the issue - bryan_att is using the gate scripts to deploy, and is seeing these problems | 17:36 |
| bryan_att | v1k0d3n: one of my goals is to replicate the gate scripts in a multi-node env. the envs currently being tested for OSH may be working but they represent a narrow beaten path, and once you try this in any other env you run into issues such as I am finding. | 17:37 |
| osh-chatbot | <v1k0d3n> agreed. is using the gate scripts to deploy on bare metal really the right path here? | 17:37 |
| bryan_att | if the scripts are to have broader value they should adapt to the real multi-node env that is being used, e.g. for OPNFV CI/CD of the Armada project once it gets started. | 17:38 |
| osh-chatbot | <v1k0d3n> prob need portdirect srwilkers or alanmeadows to speak on this. | 17:38 |
| osh-chatbot | <v1k0d3n> @bryan_att what made you use the gate scripts for bare metal? | 17:38 |
| bryan_att | I don't want to replicate 90% of the scripts just so they work in a different env. That does not make sense. | 17:39 |
| bryan_att | v1k0d3n: because there i snot other scripted process atm. | 17:39 |
| bryan_att | no other | 17:39 |
| osh-chatbot | <v1k0d3n> oh, is this what you’re trying to do for OPNFV? | 17:40 |
| bryan_att | v1k0d3n: it's a start - to test if we are really ready to pick OSH up as a project for OPNFV | 17:40 |
| osh-chatbot | <v1k0d3n> ah, gotcha… | 17:40 |
| osh-chatbot | <v1k0d3n> @bryan_att we are working on some scripts atm for what i think you’re trying to do. | 17:41 |
| osh-chatbot | <v1k0d3n> but i would really check out the most recent ps that @portdirect included for armada. | 17:41 |
| osh-chatbot | <v1k0d3n> this imo is the ultimate deployment tool/option for what you’re looking for. | 17:41 |
| *** schwicht has quit IRC | 17:42 | |
| bryan_att | v1k0d3n: I will,once it's been merged. but it gets complicated to cherry pick patches to test against. | 17:42 |
| osh-chatbot | <v1k0d3n> how so? | 17:43 |
| bryan_att | though I still don't see why (1) we have multi-node scripts that do not match the RST multi-node guide; (2) why we need multiple deployment scripts | 17:43 |
| osh-chatbot | <v1k0d3n> it needs reviews anyway…portdirect was even asking for this. | 17:43 |
| osh-chatbot | <v1k0d3n> your fb would be really critical in this case :slightly_smiling_face: | 17:43 |
| osh-chatbot | <v1k0d3n> probably have the highest value, since you’re working directly with OPNFV | 17:43 |
| bryan_att | I wll take a look but no guarantee that I have expertise to comment; I'm better as an end user who can give feedback as to whether it works | 17:44 |
| bryan_att | and what unforseen gotchas occur, as I have been struggling with now for >3 weeks | 17:44 |
| portdirect | with the sole exception of setting up the k8s cluster, the scripts and multinode should be identical from a deployment perspective. | 17:48 |
| portdirect | I would again strongly advise following it so you can identify where it looks like things are going astray - it looks like calico is not happy in your env. | 17:48 |
| bryan_att | portdirect: following what? | 17:49 |
| portdirect | http://openstack-helm.readthedocs.io/en/latest/install/multinode.html | 17:49 |
| osh-chatbot | <v1k0d3n> ^^^ YES to that. | 17:50 |
| osh-chatbot | <v1k0d3n> @bryan_att same thing we use here too. we’ve seen OSH stabilize a lot in the last couple of weeks. | 17:50 |
| portdirect | it essentially just describes the steps in here: https://github.com/openstack/openstack-helm/blob/master/tools/gate/basic_launch.sh#L38-L108 | 17:51 |
| bryan_att | portdirect: afaict that process differs substantially from the gate scripts, at least the scripts are fairly complex, and mapping the two is complicated by very limited in-script documentation... and I don't want to manually replicate the scripts as that is non-repeatable | 17:51 |
| osh-chatbot | <v1k0d3n> that can be scripted if you want some auto-foo-magically-deliciousness | 17:51 |
| portdirect | and the k8s setup to support it | 17:51 |
| bryan_att | and I have to repeat this a dozen times a day | 17:51 |
| osh-chatbot | <v1k0d3n> although, @portdirect i would really suggest that users just deploy some form of kubernetes and have a list of conference tests that OSH requires. | 17:52 |
| osh-chatbot | <v1k0d3n> this is the purpose of: https://github.com/heptio/sonobuoy | 17:52 |
| *** randomhack has joined #openstack-helm | 17:52 | |
| osh-chatbot | <v1k0d3n> i completely understand telling everyone to use kubeadm. it is _the_ choice for kubernetes going forward…but users will still want some tool beyond it; it’s just a building block. | 17:53 |
| bryan_att | i still don't understand why if you have a multi-node gate script why it should not apply as a generic deployment script. otherwise your gate is a snowflake - beyond that path all sorts of things can break that will not be caught in your CI/CD | 17:53 |
| osh-chatbot | <v1k0d3n> so to that point, when tools like KOPS, Apprenda, Kargo/Kubespray, etc start using kubeadm is that building block…OSH shouldn’t care…it should only care if it passes the conformance tests required by OSH. | 17:53 |
| bryan_att | but if that's not the intent, I guess I will have to wait for some other deployment tool to be developed - I certainly don't have the time/expertise to develop it myself. | 17:55 |
| v1k0d3n | bryan_att: told you we have something. Send me a PM I guess if you want. | 17:58 |
| portdirect | bryan_att: the gate script is working for several people - it looks like calico is your problem here | 18:00 |
| portdirect | I've asked a few times to get a look at your setup to try and diagnose issues - but we have reached the point where without access its really stabbing in the dark | 18:00 |
| *** lrensing has quit IRC | 18:02 | |
| *** schwicht has joined #openstack-helm | 18:09 | |
| *** felipemonteiro has joined #openstack-helm | 18:10 | |
| *** felipemonteiro_ has joined #openstack-helm | 18:12 | |
| *** felipemonteiro has quit IRC | 18:16 | |
| bryan_att | portdirect: well I've given specific information as to what is occuring, I would think that some of the symptoms wold be recognized or a potential cause identified by calico experts. The node/network config is quite standard and simple. | 18:17 |
| bryan_att | Dell PowerEdge R720; 4 NICs (IPMI, PXE, Private, Public), all untagged, static IP assignment (PXE: 10.5.61.0/24, Private: 10.5.62.0/24, Public 204.178.3.0/24), no bridges/bonds, no proxy, Xenial installed via MAAS over PXE net, static route added for Public GW post-install. | 18:22 |
| bryan_att | pretty much a simple OOTB config; I have varied the nodes/roles in a 3-node deploy (avoiding the even nodes issue), and used the various NICs/subnets for the node IPs (all in the same subnet), all with exactly the same result. so there's something fundamentally amiss with the calico setup or how the nodes under k8s use it. | 18:25 |
| *** felipemonteiro_ has quit IRC | 19:01 | |
| *** lrensing has joined #openstack-helm | 19:37 | |
| openstackgerrit | Merged openstack/openstack-helm master: Use RBD external provisioner https://review.openstack.org/490983 | 20:26 |
| openstackgerrit | Merged openstack/openstack-helm master: Armada OpenStack deployment yaml https://review.openstack.org/481234 | 20:29 |
| v1k0d3n | bryan_att ^^^ | 20:31 |
| v1k0d3n | armada stuff | 20:31 |
| bryan_att | sorry on a call I'll check thanks | 20:31 |
| *** renmak_ has quit IRC | 20:41 | |
| *** renmak__ has quit IRC | 20:41 | |
| *** renmak_ has joined #openstack-helm | 20:42 | |
| *** renmak__ has joined #openstack-helm | 20:43 | |
| *** marst_ has joined #openstack-helm | 20:44 | |
| *** julim has quit IRC | 20:46 | |
| *** julim has joined #openstack-helm | 20:46 | |
| *** marst has quit IRC | 20:47 | |
| openstackgerrit | Kaspars Skels proposed openstack/openstack-helm master: DNM: Test gerrit trigger for glance https://review.openstack.org/492170 | 20:49 |
| *** randomhack has quit IRC | 20:49 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Gate: Update scripts to remove replication and unrequired ceph setup https://review.openstack.org/492288 | 20:51 |
| *** spsurya has quit IRC | 20:58 | |
| openstackgerrit | Darla Ahlert proposed openstack/openstack-helm master: [WIP] Add Rally Chart https://review.openstack.org/491015 | 21:02 |
| *** alraddarla has quit IRC | 21:04 | |
| openstackgerrit | Kaspars Skels proposed openstack/openstack-helm master: DNM: Test gerrit trigger for glance https://review.openstack.org/492170 | 21:10 |
| openstackgerrit | Kaspars Skels proposed openstack/openstack-helm master: DNM: Test gerrit trigger for glance https://review.openstack.org/492170 | 21:13 |
| openstackgerrit | Larry Rensing proposed openstack/openstack-helm-addons master: Gnocchi chart https://review.openstack.org/472348 | 21:13 |
| openstackgerrit | Stacey Fletcher proposed openstack/openstack-helm master: WIP: Refactor Basic Launch https://review.openstack.org/491903 | 21:16 |
| openstackgerrit | Kaspars Skels proposed openstack/openstack-helm master: DNM: Test gerrit trigger for glance https://review.openstack.org/492170 | 21:20 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Gate: Update scripts to remove replication and unrequired ceph setup https://review.openstack.org/492288 | 21:22 |
| openstackgerrit | Kaspars Skels proposed openstack/openstack-helm master: DNM: Test gerrit trigger for glance https://review.openstack.org/492170 | 21:25 |
| openstackgerrit | Kaspars Skels proposed openstack/openstack-helm master: DNM: Test gerrit trigger for glance https://review.openstack.org/492170 | 21:28 |
| openstackgerrit | Kaspars Skels proposed openstack/openstack-helm master: DNM: Test gerrit trigger for glance https://review.openstack.org/492170 | 21:42 |
| openstackgerrit | Kaspars Skels proposed openstack/openstack-helm master: DNM: Test gerrit trigger for glance https://review.openstack.org/492170 | 21:43 |
| *** schwicht has quit IRC | 21:49 | |
| *** lrensing has quit IRC | 21:49 | |
| *** gouthamr has quit IRC | 21:52 | |
| *** schwicht has joined #openstack-helm | 21:54 | |
| *** aric49 has quit IRC | 21:56 | |
| *** schwicht has quit IRC | 22:05 | |
| *** julim has quit IRC | 22:11 | |
| *** schwicht has joined #openstack-helm | 22:26 | |
| *** schwicht has quit IRC | 22:52 | |
| openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Update Documentation - How does Openstack Helm stand up Openstack complete service(s) https://review.openstack.org/492324 | 23:39 |
| *** jaypipes has quit IRC | 23:44 | |
| *** marst_ has quit IRC | 23:47 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!