Thursday, 2016-12-15

*** dhill_ has quit IRC00:04
*** dhill_ has joined #tripleo00:05
*** dsneddon has quit IRC00:06
*** ooolpbot has joined #tripleo00:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]00:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged]00:10
*** morazi has quit IRC00:14
*** ipsecguy has quit IRC00:35
*** ipsecguy has joined #tripleo00:36
*** cdearborn has joined #tripleo00:37
*** limao has joined #tripleo00:54
*** fragatina has joined #tripleo00:55
*** ooolpbot has joined #tripleo01:10
*** ooolpbot has quit IRC01:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]01:10
openstackLaunchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged]01:10
*** yamahata_ has joined #tripleo01:24
*** dr_gogeta86 has quit IRC01:35
*** bana_k has quit IRC01:38
openstackgerritMerged openstack-infra/tripleo-ci: Default libvirt_cpu_mode to none when using qemu
*** tzumainn has quit IRC01:45
*** pleia2 has left #tripleo01:59
*** jkilpatr has quit IRC02:08
*** ooolpbot has joined #tripleo02:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]02:10
*** ooolpbot has quit IRC02:10
openstackLaunchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged]02:10
*** ansmith has joined #tripleo02:17
*** jeckersb_gone is now known as jeckersb02:20
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Remove swapfile from undercloud
*** cwolferh has quit IRC02:23
*** fzdarsky_ has joined #tripleo02:25
*** Goneri has quit IRC02:26
*** fzdarsky|afk has quit IRC02:29
*** ctayal has quit IRC02:32
*** jeckersb is now known as jeckersb_gone02:43
*** ansmith has quit IRC02:51
*** bkopilov has quit IRC03:07
*** ooolpbot has joined #tripleo03:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]03:10
openstackLaunchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged]03:10
*** cwolferh has joined #tripleo03:14
*** egafford has quit IRC03:26
openstackgerritLars Kellogg-Stedman proposed openstack/puppet-tripleo: add support for collectd
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates: add collectd composable service
*** cwolferh has quit IRC03:42
*** nyechiel has joined #tripleo03:47
*** ctayal has joined #tripleo04:04
*** links has joined #tripleo04:06
*** ccamacho has quit IRC04:09
*** ooolpbot has joined #tripleo04:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]04:10
openstackLaunchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged]04:10
*** dmacpher is now known as dmacpher-afk04:12
*** bkopilov has joined #tripleo04:25
*** cwolferh has joined #tripleo04:29
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates: add collectd composable service
*** saneax-_-|AFK is now known as saneax04:46
openstackgerritJason E. Rist proposed openstack/tripleo-ui: Adds ids namespaced with `plans_` to artifacts on plans page
openstackgerritJason E. Rist proposed openstack/tripleo-ui: Adds an id for the Login button for test automation
*** ctayal has quit IRC04:58
*** ooolpbot has joined #tripleo05:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]05:10
openstackLaunchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged]05:10
*** saneax is now known as saneax-_-|AFK05:12
*** dhill_ has quit IRC05:14
*** saneax-_-|AFK is now known as saneax05:17
*** nyechiel has quit IRC05:24
*** ctayal has joined #tripleo05:27
*** dmacpher-afk is now known as dmacpher05:28
*** oshvartz has quit IRC05:39
*** ctayal_ has joined #tripleo05:53
*** fragatina has quit IRC05:55
*** ctayal has quit IRC05:57
*** masco has joined #tripleo05:57
openstackgerritMerged openstack/tripleo-heat-templates: Deployed server: switch to apply-config hook
*** udesale has joined #tripleo06:00
*** ooolpbot has joined #tripleo06:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]06:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged]06:10
*** abregman has joined #tripleo06:10
*** coolsvap has joined #tripleo06:10
*** jaosorior has joined #tripleo06:16
*** ctayal_ has quit IRC06:20
openstackgerritIan Wienand proposed openstack/diskimage-builder: Recreate initramfs within loopback image
*** ealcaniz has joined #tripleo06:25
*** ctayal has joined #tripleo06:28
*** lmiccini has joined #tripleo06:29
*** ctayal_ has joined #tripleo06:39
*** ctayal has quit IRC06:41
*** rajinir has quit IRC06:46
*** ctayal_ has quit IRC06:48
*** pgadiya has joined #tripleo06:49
openstackgerritJason E. Rist proposed openstack/tripleo-ui: Adds ids namespaced with `plans_` to artifacts on plans page
*** yprokule has joined #tripleo06:59
*** iranzo has joined #tripleo07:00
*** oshvartz has joined #tripleo07:01
*** ctayal has joined #tripleo07:09
*** ooolpbot has joined #tripleo07:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]07:10
openstackLaunchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged]07:10
*** rcernin has joined #tripleo07:13
*** ctayal has quit IRC07:16
*** ealcaniz has quit IRC07:16
*** yamahata_ has quit IRC07:19
*** ctayal has joined #tripleo07:21
*** ramishra has quit IRC07:24
*** ramishra has joined #tripleo07:26
*** jaosorior has quit IRC07:37
*** jaosorior has joined #tripleo07:37
*** milan has joined #tripleo07:44
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo: Composable HA
*** bana_k has joined #tripleo07:48
jaosoriord0ugal: ping07:52
*** jaosorior has quit IRC07:55
*** abehl has joined #tripleo07:56
*** Vijayendra has quit IRC07:57
*** abehl has quit IRC07:58
*** abehl has joined #tripleo07:58
*** gfidente has joined #tripleo07:58
*** gfidente has quit IRC07:58
*** gfidente has joined #tripleo07:58
*** bana_k has quit IRC08:02
*** anton has quit IRC08:05
*** jlinkes has joined #tripleo08:09
*** ooolpbot has joined #tripleo08:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]08:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged]08:10
*** apetrich has quit IRC08:11
*** b00tcat has quit IRC08:12
*** b00tcat has joined #tripleo08:15
*** ccamacho has joined #tripleo08:15
*** jbadiapa has quit IRC08:16
*** shardy_afk is now known as shardy08:17
openstackgerritMerged openstack/tripleo-common: Add FreeIPA enrollment environment generator
*** dciabrin has joined #tripleo08:18
*** aufi has joined #tripleo08:18
*** anton has joined #tripleo08:19
*** florianf has joined #tripleo08:20
*** ctayal has quit IRC08:25
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Use transport_url for rabbitmq connection parameters in heat
openstackgerritFlavio Percoco proposed openstack/tripleo-heat-templates: WIP: Containerized Glance
openstackgerritFlavio Percoco proposed openstack/tripleo-heat-templates: WIP: Allow to disable glance registry in tripleo
*** jaosorior has joined #tripleo08:30
*** chem has quit IRC08:32
*** fragatina has joined #tripleo08:33
*** mcornea has joined #tripleo08:35
*** leanderthal has joined #tripleo08:37
jaosoriorbandini: hey, could you check this out ?08:37
*** pcaruana has joined #tripleo08:37
*** dsariel has quit IRC08:37
*** fragatina has quit IRC08:37
*** ohamada has joined #tripleo08:40
bandinijaosorior: ack, yeah it was on my list for today08:41
*** imranh has quit IRC08:42
*** jpena|off is now known as jpena08:43
*** imranh has joined #tripleo08:43
*** ealcaniz has joined #tripleo08:44
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here
*** zoli|gone is now known as zoli|wfh08:48
*** amoralej|off is now known as amoralej08:51
*** milan has quit IRC08:53
*** jpich has joined #tripleo08:55
*** jpena is now known as jpena|off08:55
*** yprokule_ has joined #tripleo08:55
*** jpena|off is now known as jpena08:56
*** ohamada has quit IRC08:56
*** ohamada has joined #tripleo08:56
*** chem has joined #tripleo08:57
*** yprokule has quit IRC08:57
*** yprokule_ is now known as yprokule08:57
*** chem has quit IRC08:59
*** chem has joined #tripleo08:59
*** panda|zZ is now known as panda09:01
openstackgerritMerged openstack/tripleo-ui: Add dependencies for UI internationalisation
jaosoriorshardy: are you around?09:02
*** athomas has joined #tripleo09:04
openstackgerritMerged openstack/tripleo-ui: Adds basic internationalization support
*** ealcaniz has quit IRC09:06
*** kodokuu has joined #tripleo09:07
skramajashardy: did you get a chance to look at the proposal -
*** ooolpbot has joined #tripleo09:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]09:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged]09:10
*** tremble has joined #tripleo09:12
*** tremble has quit IRC09:12
*** tremble has joined #tripleo09:12
*** flepied has quit IRC09:16
shardyjaosorior: Hi, yup I'm around09:16
shardyskramaja: sorry, I missed that - why do we need a different method to make these changes instead of puppet?09:18
shardyI'm not in favor of introducing a new tool which also modifies the system config - we've already got puppet managing some kernel configuration in t-h-t09:18
shardythe issue we can't really solve without help from ironic is how to avoid a reboot09:19
shardyskramaja: is it possible to configure the DPDK interfaces with os-net-config, but not bring them up until the reboot happens?09:19
skramajakernel arg changes for dpdk is mandatory before os-net-config binds it.. so the current way of doing is setting the kernel args in first-boot.yaml.. and rebooting..09:19
skramajashardy: os-net-config does a ifup - so basically we should avoid ifup for DPDK09:20
shardyskramaja: Yeah, so I guess we have two options - modify os-net-config to allow configuring but not ifup-ing the interfaces09:21
shardyor add a configuration step before os-net-config runs09:21
shardywe don't need a new tool to do the second though, it's already possible via t-h-t09:21
shardyskramaja: did you see that we now run os-net-config via a script?09:21
skramajashardy: you meant the 20-os-net-config script?09:22
shardyskramaja: no, we don't use that anymore, we run it via a script in t-h-t09:22
shardywhich makes it much easier to modify or run an alternative script09:22
shardynot sure if that helps tho, as you'd have to reboot during the script09:23
shardyskramaja: to clarify, what is the objection to the current firstboot script approach?09:23
*** apetrich has joined #tripleo09:23
shardythat seems to be a good time in the deployment cycle to do the reboot, e.g as early as possible09:23
d0ugalWhere are we with CI today? is there anything I can do to help?09:23
shardyso it doesn't seem so bad, given that the ironic ramdisk won't do this for us atm09:23
skramajashardy: there are other scenarios like "tune-d" which also needs reboot..09:24
owalshskramaja: can't we configure tuned in the firstboot script?09:24
skramajashardy: if we put everything on first-boot, which we are doing now as stopgap,09:24
shardyskramaja: Ok, but what's the reason that's viewed as a stopgap?09:25
*** ealcaniz has joined #tripleo09:25
shardyis it because there's no error path when the script doesn't work?09:25
shardyowalsh: yeah exactly, that would work OK I think09:25
skramajashardy: yes and no clear steps sepration..09:26
skramajaits like dumping everything into firstboot..09:26
shardyskramaja: well, it's not "everything", it's some specific actions which need to happen, uh, on firstboot09:26
owalshskramaja: I agree it feels a bit wrong...09:27
skramajashardy: :)09:27
owalshskramaja: I'm looking at dropping a yaml file and running a python script09:27
owalshskramaja: ... sometihng I can unit test etc...09:28
shardyLike, I get that it's not *ideal* - the perfect solution would be for the deploy ramdisk to configure everything before the image even boots09:28
shardybut we know the ironic folks don't want that09:28
shardyso we're stuck with doing something early in the first boot, then rebooting09:28
shardycloud-init is a not-bad way to do that IMO09:28
shardywe could reinvent another way to do the same config slightly later, but I'm just not clear why09:28
*** dtantsur|afk is now known as dtantsur09:29
owalshnumber of people telling me the reboot is very costly09:29
skramajaok shardy09:29
shardyowalsh: Yeah, but there's nothing we can do about that unless we convice the ironice community to make these changes during deployment of the image09:29
shardyor, perhaps provide some hooks in the ipa ramdisk that allow it09:30
shardyor, we could run a patched version of the ipa ramdisk09:30
shardynone of the reboot problems are solved by introducing a new agent that runs the same script we run via cloud-init tho09:30
shardyskramaja: there are some other ways to move the config out of cloud-init09:31
* skramaja listening09:31
shardyif you want to run this via t-h-t, we could have a conditional deployment, which reboots only when some previous deployment did something09:31
shardyusing heat conditionals09:31
shardyor a provider template that defaults to OS::Heat::None09:31
shardyskramaja: let me write a quick prototype, give me a few mins09:32
*** jbadiapa has joined #tripleo09:32
skramajathanks shardy09:32
skramajaowalsh: what do you mean by - "I'm looking at dropping a yaml file and running a python script", not clear..09:33
*** akrivoka has joined #tripleo09:33
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci: Postci tests trial and error
owalshskramaja: 3rd example here
owalshskramaja: set yaml in boot_config file, then run a python script that read the yaml and applies config09:36
owalshskramaja: better than templating params in scripts IMO :-)09:37
skramajagot it owalsh..09:37
*** shinobu__ has joined #tripleo09:38
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Remove dead mocks and dead code
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Remove the keystone_pki cert generation from tripleoclient
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Use the overcloudrc generated in a Mistral action
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Remove the unused get_config_value function
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Simplify Fetching the passwords from Mistral
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: [WIP] Respond to multiple messages from workflow executions
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Remove the unused delete image function
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Remove the unused constant RESOURCE_REGISTRY_NAME
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Remove two unused Exception subclasses
*** fzdarsky_ is now known as fzdarsky09:42
*** kodokuu has quit IRC09:43
*** kodokuu has joined #tripleo09:43
*** lucas-afk is now known as lucasagomes09:49
openstackgerritDougal Matthews proposed openstack/tripleo-common: Remove the unused constant OBJECT_META_KEY_PREFIX
*** dsariel has joined #tripleo09:52
*** flepied has joined #tripleo09:55
openstackgerritDougal Matthews proposed openstack/tripleo-common: Remove unused exception ContainerAlreadyExistsError
openstackgerritDougal Matthews proposed openstack/tripleo-common: Remove unused exception HeatValidationFailedError
* d0ugal looks for other stuff to delete09:56
*** milan has joined #tripleo09:58
*** paramite has joined #tripleo10:03
*** yprokule_ has joined #tripleo10:03
skramajashardy: another case with first-boot maintanence is, if there is a cluster with DPDK and SR-IOV or other feature, which needs different kernel args per role, it will be hard to maintain in first-boot script..10:04
*** yprokule has quit IRC10:04
*** yprokule_ is now known as yprokule10:04
skramajashardy:  as of now we are comparing host name  and taking decision in the first-boot script.10:05
*** ooolpbot has joined #tripleo10:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]10:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged]10:10
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Add pre-network hook and example showing config-then-reboot
shardyskramaja: ack, see if ^^ would suit you better10:12
shardyit enables per-role pre-network config, including an example that does a reboot10:12
shardynot yet tested, feedback welcome10:12
skramajalooking at it shardy..10:12
skramajaowalsh: ^10:13
*** hewbrocca_afk is now known as hewbrocca10:13
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart-extras: Add overcloud delete role
*** openstackgerrit has quit IRC10:18
* owalsh looking10:19
Ngwhat's the rough workflow for adding a new action in tripleo-common and having it show up in mistral? Do I have to rebuild or reboot the undercloud, or can I poke things so it notices the new stuff?10:19
shardyNg: you have to install a new tripleo-common with the action, then run mistral-db-manage populate10:20
Ngshardy: aha, that is the magic command I hadn't spotted. thanks!10:20
shardyNg: that populate normally happens via the "openstack undercloud install"10:20
shardyso you can re-run that if you like but it takes a while10:20
Ngshardy: I'm using quickstart, so my undercloud image comes pre-built10:21
Ng(I think it can build images, but I've never looked at that part)10:21
d0ugalNg: I don't follow this exactly, but this is one way to do everything needed.
shardyNg: you can still run openstack undercloud install10:21
d0ugalWe should probably update that just to show the needed commands, rather than the opinionated install method10:21
shardyit's designed to be idempotent, so you can run it as many times as you like (in theory)10:21
Ngthanks :)10:21
*** nyechiel has joined #tripleo10:23
therved0ugal, Isn't it too early for spring cleanup? :)10:24
d0ugaltherve: hah, I think this is long overdue from spring this year ;)10:24
d0ugaltherve: also I recently discovered
d0ugalIt gives lots of false positives, but gives a good starting point.10:26
therveThat's pretty cool, thanks for the pointer10:26
*** openstackgerrit has joined #tripleo10:28
openstackgerritAndrey Shestakov proposed openstack/diskimage-builder: Fix dhcp-all-interfaces for ubuntu-minimal xenial
apetrichmarios, about your comment here I was thinking about using CONTAINER_NAME_PATTERN do you think it is clear enough?10:28
mariosapetrich: sure don't mean to bikeshed on that just something more explicit that sounds fine jpich ^^10:29
d0ugalapetrich: I'd go with "plan" rather than "container"10:29
d0ugalI know we use container in various places when we mean plan, but really we should move away from that IMO10:29
apetrichd0ugal, aye10:29
jpichI'd prefer to use plan as well10:30
*** nmathew has joined #tripleo10:31
*** fultonj has quit IRC10:31
shardyNg: Oh, I just remembered another useful thing when working with actions is entry_point_inspector10:31
shardythe centos package appears broken due to deps re pyparsing, but if you pip install you can do10:31
*** fultonj has joined #tripleo10:32
shardyepi group show mistral.actions | grep tripleo10:32
shardyI found it useful to ensure my actions were updated to what I expected before testing10:32
d0ugalshardy: neat, I didn't know about that one.10:32
shardyAnd if you're hacking on workflows you might want to cut/paste from that script10:33
shardyI had a little script which refreshed the actions and workflows which made life a little easier10:33
therveshardy, Is it better than the mistral list actions CLI?10:34
shardytherve: well it's useful to figure out if a missing action wasn't correctly installed, or if it silently failed during the db populate10:34
d0ugaltherve: epi? I guess it is just quicker - you can verify before you run the populate command and restart mistral.10:35
shardytherve: I experienced both problems at various times10:35
d0ugalI keep meaning to improve the populate command, it should handle the faulures but it could give us more useful feedback.10:35
apetrichPLAN_NAME_PATTERN gets the line just under 79 chars10:35
therveshardy, Makes sense, thanks10:35
shardyd0ugal: when I was testing it just failed but continued10:35
shardymaybe that's fixed now, I think we discussed it before :)10:36
jaosoriorshardy: hey, so for the TLS everywhere work, certmonger requires the nodes' hostnames and services principals in order to request the appropriate certificates, this is already happening and it's good. However, these entries need to exist in the CA beforehand; what I was doing is to pre-create all the entries with a script, and once having that (and an OTP that the CA gives you), trigger the overcloud10:36
jaosoriordeployment with the OTP, so stuff would work... The issue is that pre-creating these entries is very tedious as deployments can get quite big and we need an entry per subjectAltName that will be used in the certs (so we also need to populate the per-network hostnames with their respective service principals). Now, for another use-case (having identities per nove host) aside from this one we came up with a10:36
jaosoriorsolution based on Nova's vendordata plugin structure. This "plugin" is a service that basically detects when a server is created, gets the host name, and creates the entry in the CA, and removes this entry when the server is deleted. Now, for the TripleO case, we do need the subjectAltNames, which we could pass through nova metadata. Basically we could pass the relevant service name with the name of the10:36
jaosoriornetwork it's listening on, and the vendordata service could create the relevant entry from this data. Would you be fine if I introduce a hook for the nova metadata in the TripleO servers that reads an entry from the service profiles, and from that generates said list?10:36
*** derekh has joined #tripleo10:36
d0ugalshardy: I do sort-of remember this. Maybe I even opened a bug. I shall check.10:36
shardypastebomb ;)10:36
jaosoriorwrote it on the spot, no pasting :P10:36
*** tobias_fiberdata has quit IRC10:36
jaosoriorweechat just separated it10:36
jaosoriorsorry for the long text though10:37
d0ugaljaosorior: IIRC, 512 is the maximum char limit in IRC :/10:37
*** limao has quit IRC10:37
shardyjaosorior: so, to clarify, the question is can we add a list of things to the nova server metadata map, for consumption by the Nova vendordata plugin?10:39
shardyso we'd merge some additional data in with ServerMetadata here?10:40
shardySounds OK to me10:40
*** b00tcat has quit IRC10:41
openstackgerritAdriano Petrich proposed openstack/tripleo-common: Validate the plan name before running
jaosoriorI'll come up with a POC in a bit to show it better10:41
skramajathanks shardy for the prototye.. i will start working on it to incorporate kernel args and tuned changes.. will poke you, if hit with any hurdels..10:41
*** bvandenh has joined #tripleo10:41
*** b00tcat has joined #tripleo10:43
*** tobias_fiberdata has joined #tripleo10:50
*** tosky has joined #tripleo10:52
*** jlinkes has quit IRC10:53
*** chem has quit IRC10:54
*** chem has joined #tripleo10:54
*** Vijayendra has joined #tripleo10:55
* owalsh doing the same, thanks shardy10:55
openstackgerritHonza Pokorny proposed openstack/tripleo-ui: Remove SharedWorker in favor of cookies
*** jlinkes has joined #tripleo11:08
*** ooolpbot has joined #tripleo11:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]11:10
openstackLaunchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged]11:10
*** ealcaniz has quit IRC11:11
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Add pre-network hook and example showing config-then-reboot
shardyowalsh, skramaja: np - let me know if it works OK - added the stop of occ in ^^11:13
b00tcatEmilienM: I remember you commented a while ago on my patches that we needed the puppet packages in RDO, and I'm not that :-) but out of curiosity, what if we hosted them in the MidoNet repos? would that also be an option?11:19
*** ealcaniz has joined #tripleo11:20
*** cdearborn has quit IRC11:20
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Disable the ceilo swift pipeline for multinode
shardyb00tcat: I think the main reason they would be better in RDO is we then get automatic delorean builds, which can then be consumed in CI11:32
*** dtantsur is now known as dtantsur|brb11:32
shardyb00tcat: if there's any future plan to get midonet integration tested in CI (would be a good idea?), I expect that will be useful11:32
*** panda is now known as panda|lunch11:33
b00tcatshardy: right, thanks :-) I'm atm trying to get stuff packaged at RDO11:33
b00tcatshardy: although I must admit that I still didn't really understand when the RDO packages get pulled, as I thought they'd be in the base image already11:34
shardyb00tcat: once a repo is packaged and in the RDO repos, it gets built nearly every commit11:35
shardyb00tcat: however, we've recently removed all vendor packages from the base image, ref
b00tcatshardy: thanks11:37
shardythe reason is none of them are tested in CI, and they broke us several times11:37
b00tcatshardy: that makes sense :D11:37
*** shardy is now known as shardy_lunch11:39
*** adarazs is now known as adarazs_lunch11:39
*** pkovar has joined #tripleo11:41
*** rbowen has joined #tripleo11:43
*** apetrich has quit IRC11:44
jistrweshay: i think i managed to pin down the "running `virsh list` destroys the dev env" issue --
openstackLaunchpad bug 1650238 in tripleo-quickstart "Systemd freezing execution in VMs on virsh list" [Low,New]11:44
hewbroccajistr: are you using fedora as your virthost?11:45
jistrhewbrocca: no, quickstart will refuse running on fedora11:45
hewbroccapmyers was having some problems with that, had to switch to centos11:45
hewbroccaahh, OK11:45
jistrcentos 711:45
hewbroccaadarazs_lunch: around ^^^ ?11:45
jistri think it's likely that the root cause itself is in libvirt/qemu/kvm11:45
jistrnot sure what can we do about it in quickstart11:46
hewbroccahuh yeah11:46
jistr(perhaps qemu:///system instead of qemu:///session, but that might have some other drawbacks)11:46
hewbroccalibvirt should not segfault, no matter what args you pass it11:46
hewbroccawhat's "-i" do to sudo?11:46
openstackgerritJulie Pichon proposed openstack-infra/tripleo-ci: Add jpich to Planet
jistrhewbrocca: runs a login shell and executes the command there IIRC11:47
jistras opposed to executing the target command directly11:47
*** panda|lunch is now known as panda11:48
jistrtrown: FYI when you're around ^^11:48
jistrwell at least we know what commands *not* to run :)11:48
* jistr restarts OOOQ and relocates11:48
hewbroccajistr: oh, I gotcha11:50
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Remove white space between print () in
*** bkopilov has quit IRC11:50
hewbroccawhy on earth would that make any difference?11:50
toskyuhm, the environment variables overwritten11:51
EmilienMtripleo ocata-2 has been released11:54
*** jpena is now known as jpena|lunch11:54
*** fzdarsky is now known as fzdarsky|lunch11:55
*** jkilpatr has joined #tripleo11:57
*** ealcaniz has quit IRC11:59
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Adds a step0 for pre upgrade-init checks
EmilienMderekh: have you seen bnemec's thought on ?12:02
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]12:02
derekhEmilienM: nope, looking12:02
derekhEmilienM: ya, I came to a similar conclusion this morning, and ran a ci job with a tcpdump of the traffic to gearman
*** rbowen has quit IRC12:04
*** david-lyle has quit IRC12:05
*** david-lyle has joined #tripleo12:05
*** rbowen has joined #tripleo12:06
*** Baggypants12000 has joined #tripleo12:07
openstackgerritNuman Siddique proposed openstack/tripleo-heat-templates: OVN plugin configuration fixes
EmilienMderekh: do you have an idea? I see nothing special in the tcpdump12:07
mariosshardy_lunch: matbu ccamacho hi guys I left a comment at grateful for any thoughts if you get a chance - we still on for our call in 2.5 hours (ish)?12:08
marioschem: ^12:08
ccamachomarios sure, Ill check it.12:09
chemmarios: yep12:10
marioschem: jistr i updated the ovs upgrade one last night :) got rid of the ceph repetition with your suggestions thanks12:10
*** ooolpbot has joined #tripleo12:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]12:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged]12:10
*** rbowen has quit IRC12:11
EmilienMmatbu, shardy_lunch: what is the plan for ? Where this code is going to live?12:12
derekhEmilienM: the only think I think I see is the testenv-client finishing the CI job here @ 10:48:05.5865012:12
derekhEmilienM: then pretty much ignoring all traffic from the geard server after that12:13
EmilienMderekh: could it be a firewall rule?12:13
EmilienMalso I'm curious why it only happens on ha & update scenarios12:13
*** cdearborn has joined #tripleo12:14
openstackgerritNuman Siddique proposed openstack/tripleo-heat-templates: OVN plugin configuration fixes
derekhEmilienM: Its possible, but I didn't think the firewall was running on those underclouds12:15
*** masco is now known as masco_afk12:15
EmilienMderekh: even if it would be, the rules would be the same as non-ha scenario12:16
derekhEmilienM: yup12:16
derekhEmilienM: I've recheck this, we can ssh to it in a while to see if we can see anything
openstackgerritNuman Siddique proposed openstack/tripleo-heat-templates: OVN bridge mappings for tripleo
EmilienMmatbu: I was thinking at starting the work to move it to a repo and package it soon, time it kind of running out quickly in this cycle12:18
*** udesale has quit IRC12:19
matbuEmilienM: we will discusss it this afternoon12:21
matbuEmilienM: but yes it could be useful, is there a doc somewhere on how should that be packaged ? (i can take care of it)12:25
*** lucasagomes is now known as lucas-hungry12:26
EmilienMmatbu: ask pabelanger12:28
EmilienMmatbu: he did that already I think12:28
*** owalsh is now known as owalsh-afk12:28
*** skramaja has quit IRC12:28
*** skramaja_ has joined #tripleo12:28
*** dprince has joined #tripleo12:29
*** pkovar has quit IRC12:30
*** masco_afk is now known as masco12:36
*** abregman is now known as abregman|afk12:37
*** kodokuu has quit IRC12:38
EmilienMbnemec: I think we can close it, it seems like it works fine now12:42
openstackLaunchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged]12:42
*** nmathew has quit IRC12:42
* EmilienM removing the alert and closing it12:43
*** jpena|lunch is now known as jpena12:43
gfidenteEmilienM this guy is blocked by CI from a few days12:44
*** pradk has joined #tripleo12:44
gfidentethere are 3 submissions in tripleo which depend on it12:44
EmilienMgfidente: man, everything is blocked right now12:44
gfidentehave any idea if/how we can push it a bit further?12:44
*** bfournie has quit IRC12:44
EmilienMgfidente: the qemu thing broke puppet and tripleo CI all week12:44
EmilienMwe merged almost nothing12:45
gfidenteyeah I was hitting that with quickstart locally12:45
*** pradk has quit IRC12:45
gfidentemy only working workaround to date12:45
gfidentehas been to downgrade libvirtd12:45
EmilienMgfidente: see - that's the fix for master CI12:45
openstackgerritMerged openstack/instack: Add changes for dib feature/v2 branch
EmilienMgfidente: but stable/newton, stable/mitaka and stable/hammer is still broken now, we're working on it. any help is welcome12:45
openstackgerritMerged openstack/instack-undercloud: Set Ironic cleaning network to ctlplane
gfidenteEmilienM but is it the same issue for all the releases?12:47
gfidenteor am I missing something?12:47
EmilienMkind of12:47
openstackgerritMerged openstack/instack-undercloud: Optional Cinder support for undercloud
EmilienMgfidente: there are multiple emails about that on openstack-dev12:47
*** rhallisey has joined #tripleo12:47
EmilienM[rdo-list] Known issue with CentOS 7.3 and qemu-kvm(-ev) 2.6.012:48
EmilienMgfidente: that ^12:48
gfidenteyeah I replied to that thread too12:48
EmilienMgfidente: the poi patch merged, do recheck now12:48
gfidenteI am not sure we want to push jewel for the mitaka relase12:48
gfidentewe shipped hewel in the overcloud images12:49
EmilienMgfidente: like I said, there is no way we push jewel in mitaka branch12:49
EmilienMgfidente: I -2 the idea and I'll keep my position12:49
gfidenteyeah I was saying the same thing12:49
*** tobias-fiberdata has joined #tripleo12:50
EmilienMgfidente: I did recheck on your puppet-ceph patch though I don't know why it failed, I'm looking at logs12:51
*** chlong has joined #tripleo12:51
gfidenteEmilienM same issue with nodes not starting in nova12:51
EmilienMsounds like a timeout12:51
adarazs_lunchjistr, hewbrocca: thanks for the bug against quickstart, let me know how that deployment goes!12:51
*** adarazs_lunch is now known as adarazs12:51
EmilienMgfidente: it doesn't seem related to your patch :/12:52
gfidenteyeah I mean it's the known issue with libvirt unable to start the guests12:52
gfidenteso the overcloud fails12:52
EmilienMgfidente: are you sure?12:53
*** tobias_fiberdata has quit IRC12:53
EmilienMit's multinode job, there is no nova here12:53
*** jprovazn has joined #tripleo12:54
gfidenteEmilienM pingtest12:57
EmilienMgfidente: no, overcloud failed before pingtest12:57
* EmilienM brb12:57
*** cdearborn has quit IRC12:58
jistradarazs: it always goes well, as long as one doesn't try to `virsh list` the wrong way :D13:00
adarazsjistr: yeah, what I usually do is "su - stack" completely, which is equivalent to using sudo with `-i` so I wouldn't have discovered this :)13:02
gfidenteEmilienM I see both scenarios failing on pingtest13:03
jistrmarios: re ovs upgrade -- looks good, thanks :)13:04
mariosthanks jistr13:05
*** pkovar has joined #tripleo13:06
*** skramaja_ has quit IRC13:06
*** skramaja has joined #tripleo13:06
*** shardy_lunch is now known as shardy13:06
EmilienMgfidente: do you have logs?13:07
*** ooolpbot has joined #tripleo13:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]13:10
*** amoralej is now known as amoralej|lunch13:10
*** pgadiya has quit IRC13:11
openstackgerritmathieu bultel proposed openstack/tripleo-quickstart-extras: add composable services deployment support for oooq
*** jeckersb_gone is now known as jeckersb13:12
*** oshvartz has quit IRC13:14
*** dmacpher has quit IRC13:15
*** dmacpher has joined #tripleo13:15
*** fzdarsky|lunch is now known as fzdarsky13:15
*** abregman|afk is now known as abregman13:16
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Run upgrade steps before post-deploy config
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move UpgradeInitCommand to role templates
*** apetrich has joined #tripleo13:20
*** cdearborn has joined #tripleo13:22
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras: Repositories setup role
*** saneax is now known as saneax-_-|AFK13:23
*** bfournie has joined #tripleo13:24
*** kodokuu has joined #tripleo13:24
*** jprovazn has quit IRC13:25
shardymatbu: Hey, I've got the upgrade working locally now - I've pushed the tht patches, but a few changes are needed to the tripleo-ci patch too, are you OK if I just push them?13:25
*** zoli|wfh is now known as zoli|lunch13:26
shardyI'm just testing again trying to combine the upgrade and converge, then I can refactor my local changes into your patch, if you're OK with that13:26
*** jayg|g0n3 is now known as jayg13:26
matbushardy: yep sure13:27
matbushardy: where is the tht review ?13:27
*** [1]cdearborn has joined #tripleo13:27
matbushardy: i have implemented the base for making the same test in quickstart yesterday13:27
shardymatbu: ^^ - I then need to change UpgradeInitCommand to handle disabling the old hiera element13:28
shardyas well as installing the new hiera heat-config hook13:28
matbushardy: it should be easy then to reused the same workflow in both (tripleo-ci and quickstart)13:28
shardyother than that, it works pretty well13:28
matbunice thanks13:28
*** lucas-hungry is now known as lucasagomes13:29
shardymatbu: nice re quickstart - I was thinking we'd need that pretty soon13:29
*** links has quit IRC13:31
*** ayoung_dadmode is now known as ayoung13:32
*** rbowen has joined #tripleo13:32
shardymatbu: did you have any preference re the git vs rpm comment in ?13:33
shardylocally I've been unpacking a newton tht RPM instead of cloning the git tree, but either should work13:34
matbushardy: i think git is fine but maybe rpm is more robust ?13:35
shardymatbu: Yeah it might be, IIRC we had spurious errors cloning from git URLs in the past, but my main motivation for RPM was to enable overlaying the tripleo-ci repo in future, then we can potentially make Depends-On work13:36
*** bkopilov has joined #tripleo13:36
shardyHopefully we won't need Depends-On for the stable tht version very often tho13:36
matbushardy: yep right13:37
*** sudipto has joined #tripleo13:37
*** sudipto_ has joined #tripleo13:37
shardyI'll push the RPM version then you can compare and see if you're happy with it13:37
*** limao has joined #tripleo13:37
dprinceshardy: hi, so I'm trying to get rid of my use of noauth middleware in Heat13:40
jaosoriorshardy: a bit of yaql help. Any idea why what's wrong with this one ? It's just trying to generate a list with the values of the maps13:41
dprinceshardy: since I've gone and mocked out a full on keystone it should be possible I think13:41
dprinceshardy: any ideas on how to work past this... I hit it when I disable noauth:
dprinceshardy: I guess I just need to beef up my service catalog fake?13:41
*** cdearborn has quit IRC13:41
dprinceshardy: I'd like it just simply not to query it though, which it doesn't seem to w/ noauth13:42
shardydprince: Are you overriding the default signal_transport?  Looks like it's trying to use CFN, which is the default13:43
dprinceshardy: I use Heat API directly, no CFN13:43
*** dhill_ has joined #tripleo13:44
dprinceshardy: which is why this on surprises me13:44
*** ansmith has joined #tripleo13:44
shardydprince: Yeah but *Deployment resources default to signalling via the CFN API, for historical reasons13:44
shardydprince: it can be globally configured in the heat.conf, sec13:44
dprinceshardy: default_deployment_signal_transport = HEAT_SIGNAL13:44
dprinceshardy: I set that13:44
shardyHrm, then I guess there's a bug where it's still trying to find the cfn endpoint and failing13:45
dprinceshardy: okay, well at least we agree this looks suspicios :)13:45
* shardy finds the code13:45
shardydprince: Yeah, it's some heat validation getting in the way - we try to ensure the services used by resources are in the catalog, a bit too eagerly in this case it seems13:46
dprinceshardy: FWIW, noauth seems to avoid this codepath entirely The patch has been posted since mid-summer though and no love as of yet13:47
dprinceshardy: so I figured I'd try without it... because it should be feasable now that I've got a keystone fake13:47
shardydprince: Ok, lets see if we can chase for some review love on then13:49
*** zoli|lunch is now known as zoli|wfh13:50
dprinceshardy: ack, thanks13:51
shardyjaosorior: try $[$.service, $.network])13:53
jaosoriorshardy: that worked, thanks13:54
*** kodokuu has left #tripleo13:59
openstackgerritJulie Pichon proposed openstack/tripleo-ui: Disallow underscores in plan names
jaosoriorshardy: is it possible to do string interpolation in yaql14:06
jaosoriorsomething like: $"$.service/$.network")14:07
shardyjaosorior: not sure tbh - I'd check the tests in the yaql tree for anything similar14:08
*** amoralej|lunch is now known as amoralej14:09
*** ooolpbot has joined #tripleo14:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]14:10
openstackgerritRaoul Scarazzini proposed openstack/tripleo-quickstart-extras: Baremetal undecloud role playbook fixes
*** mhenkel has joined #tripleo14:11
*** derekh is now known as derekh_brb14:12
jaosoriorshardy: found it :D14:14
jaosoriorthere's a concat and a format function14:14
shardyjaosorior: ah, nice14:15
openstackgerritKeith Schincke proposed openstack/puppet-tripleo: Add support for not using admin_token in Ceph/RGW
gfidenteadarazs if I use localhost as VIRTHOST14:17
gfidenteam I supposed to create the stack user myself?14:17
*** tzumainn has joined #tripleo14:17
adarazsgfidente: it should work if execute quickstart as another user (not root, not 'stack') and use "" instead of localhost. ansible is a bit finicky when using a local connection.14:18
adarazs127.0.0.2 goes around that. :)14:19
gfidenteadarazs right but the user itself14:19
gfidenteshould I create it anyway?14:19
adarazsgfidente: it will be created.14:19
gfidenteit seems to be trying to ssh on baremetal with that user14:19
gfidentebut the user isn't there14:19
*** chlong has quit IRC14:20
adarazsgfidente: ->
*** Vijayendra has quit IRC14:21
adarazsgfidente: this is where the user creation happens. you have to have ssh access to root@$VIRTHOST to make this work14:22
*** chlong has joined #tripleo14:22
adarazsas we initially ssh as root to create the non_root_user (stack)14:22
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add hook to generate metadata from service profiles
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add metadata settings for needed kerberos principals
gfidenteadarazs yeah I see that if virthost matches localhost it uses provision/local14:23
gfidentewhich doesn't create the stack user it seems, right?14:24
*** morazi has joined #tripleo14:24
adarazsgfidente: yeah, I think we have a couple of extra cases if you run quickstart with localhost as virthost, but it's poorly tested. I know trown is using "" to get around all the ansible limitations and run it locally14:24
marios    shardy matbu chem ccamacho mcornea tripleo upgrades squad call in 5 minutes @ (dial-in info on the pad)14:25
*** ealcaniz has joined #tripleo14:25
ccamachomarios yeahp :)14:25
adarazsgfidente: try that.14:25
mcorneamarios: thanks, joining14:25
gfidenteadarazs ack thanks14:25
*** egafford has joined #tripleo14:26
*** chlong has quit IRC14:26
*** chlong has joined #tripleo14:27
*** lblanchard has joined #tripleo14:30
*** Goneri has joined #tripleo14:31
*** paramite has quit IRC14:31
EmilienMchem: I'm not coming but you might want to talk about
*** morazi has quit IRC14:31
*** jcoufal has joined #tripleo14:31
chemEmilienM: I'm there already. Thanks for the pingback14:31
mariosshardy: are you able to join?14:33
*** pkovar has quit IRC14:33
*** oshvartz has joined #tripleo14:35
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras: Replace hardcoded stack user by ansible_user
shardymarios: ah, thanks, joining14:38
*** ealcaniz has quit IRC14:39
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci: Implement major upgrade for Newton to Ocata
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci: Fix REPO_PREFIX handling of ceph repos
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Force kill testenv-client
derekh_brbEmilienM: bnemec ^^ a really nasty hack, but it might give us some breathing room to properly debug the problem with the te-client hanging14:43
openstackgerritJohn Fulton proposed openstack/tripleo-specs: Integrate TripleO with Tendrl for External Storage Deployment/Management
EmilienMderekh_brb: nice :D14:44
derekh_brbEmilienM: bnemec I'm not even sure this will cause things to clean up properly on the te-broker, we'll have to check that if we are going to merge it14:44
*** derekh_brb is now known as derekh14:44
*** morazi has joined #tripleo14:44
*** ealcaniz has joined #tripleo14:47
*** ealcaniz is now known as ealcaniz|Orange-14:47
dprinceshardy: Add element to disable old hiera element14:48
*** dsariel has quit IRC14:48
dprinceshardy: I think that should handle it. A bit odd perhaps, and I could move it into the heat-templates tree if that makes more sense to you14:48
dprinceshardy: I think perhaps eventually we'd want an element that removes anything in libexec/os-refresh-config that isn't in a proper RPM package. That would solve the problem globally14:49
EmilienMbnemec: hi! do you think we'll make progress on during ocata-3 ?14:50
openstackgerritJohn Fulton proposed openstack/tripleo-specs: Integrate TripleO with Tendrl for External Storage Deployment/Management
*** athomas has quit IRC14:53
EmilienMshardy: do you think we can close which is duplicated with ?14:53
openstackLaunchpad bug 1611704 in tripleo "User defined passwords should be easier to define" [High,In progress] - Assigned to Saravanan KR (skramaja)14:53
*** ealcaniz|Orange- has quit IRC14:53
*** pkovar has joined #tripleo14:53
openstackgerritBrad P. Crochet proposed openstack/python-tripleoclient: Migrate overcloud update to a mistral workflow
*** b00tcat has quit IRC14:55
*** b00tcat has joined #tripleo14:56
*** udesale has joined #tripleo14:57
*** links has joined #tripleo15:01
*** athomas has joined #tripleo15:01
shardyEmilienM: Yes I think we can consider it a bug and track it via
openstackLaunchpad bug 1611704 in tripleo "User defined passwords should be easier to define" [High,In progress] - Assigned to Saravanan KR (skramaja)15:03
*** Goneri has quit IRC15:03
*** chlong has quit IRC15:03
shardydprince: ack, thanks, and +100 on removing all the non-package-owned-things ;)15:03
EmilienMshardy: excellent15:05
EmilienMdtantsur|brb, lucasagomes: what progress do we have on ? Still on track for ocata?15:05
*** pblaho has quit IRC15:06
*** pradk has joined #tripleo15:06
lucasagomesEmilienM, hi there, yes... the package for virtualbmc is all sorted. It's missing the work for tripleo-quickstart, my patch got a +2 but was -1'd yesterday15:06
lucasagomesso I'm going to address the comments for it today15:06
EmilienMlucasagomes: why tripleo-quickstart?15:06
lucasagomesEmilienM, cause that's the tool we use for development of tripleo ?15:07
lucasagomes(ssh driver being a test/develop driver15:07
EmilienMlucasagomes: do we test that in tripleo-ci?15:07
EmilienMlucasagomes: we haven't switched all jobs to oooq now, sshnaidm|afk is working on htat15:07
*** sshnaidm|afk is now known as sshnaidm15:08
lucasagomesEmilienM, right, so the remaining should also be updated. I will take a look at the tripleo-ci15:08
jistrweshay, adarazs, trown: folks how do you test changes to quickstart? do you re-run the whole thing from scratch every time? I'm used to making playbooks more-or-less idempotent and re-running them, but it doesn't seem like that's the best approach with OOOQ... are there some tricks you do to make the run partial or something?15:08
EmilienMlucasagomes: ok. I consider it still on track then15:10
*** ooolpbot has joined #tripleo15:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]15:10
lucasagomesEmilienM, right on15:10
trownjistr: there are kind of two types of changes, and I approach them differently15:10
*** numans has quit IRC15:10
trownjistr: If I am changing the virt-setup stuff I just re-run from scratch each time only to the point undercloud is installed15:11
*** tosky has quit IRC15:11
trownjistr: If I am changing something that happens after the undercloud is installed, I usually just hack directly on the scripts on the undercloud until I get it right, then convert that back to the jinja templates in quickstart and re-run from scratch to confirm my changes and submit them15:12
dprinceshardy: using something like this (with conventions) I think I could get rid of neutron for our network isolation support
dprinceshardy: I'm using that for the t-h-t undercloud installer now and it works really nicely for 'named' Neutron ports15:13
*** numans has joined #tripleo15:14
dprinceundercloud neutron... how do you like them apples15:14
derekhNetworking people, does this look normal? same tcp connection multiple processes
*** Goneri has joined #tripleo15:15
trownjistr: we do have tags on the tasks for doing partial runs, but that interface is a bit fragile given it is not CI'd at all... we are pretty resource constrained in what we can CI (as in number of concurrent jobs allowed in centosci) or it would make sense to have some jobs confirming some other subsets of tags working15:15
jistrtrown: ok, thanks. How do i do a partial run, e.g. to the point undercloud is installed? I guess with --tags or --skip-tags?15:15
jistrtrown: haha thanks15:15
shardydprince: Nice, yeah that's the sort of thing I imagined we might use to replace neutron, e.g just a big map of ips15:16
shardydprince: you could also combine it with the network composition stuff I started in so that you auto-generated the last octet e.g based on a combination of the network cidr and the node index15:17
dprinceshardy: my implementation looks like this ATM
trownjistr: actually with default tags and quickstart-extras.yml playbook you should get all of the templated scripts written out, but nothing run after the undercloud install15:18
dprinceshardy: ack on the NW composition stuff15:18
trownjistr: hmm... nope... I am wrong on that15:19
shardydprince: nice, yeah I guess we could handle explicit mappings and index based ones and neither would actually need neutron15:19
trownjistr: that is a bug since moving to quickstart-extras... there are roles not appropriately tagged in quickstart-extras.yml15:19
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move UpgradeInitCommand to role templates
dprinceshardy: it is working really great for the Undercloud. I'm not fully able to use overcloud.yaml. So we won't need a separate undercloud.yaml that I was prototyping before...15:20
trownjistr: I will make a launchpad for that and submit a patch15:21
EmilienMdtrainor: hey! are we still on track for ?15:21
jistrtrown: perfect, thanks a lot :) if we're not aiming for idempotency, then being able to do partial runs of particular parts would help a lot i think15:21
dtrainorhi, EmilienM.  tl;dr: yes.  just in the last 48 hours in the testing i've done, i've discovered that this work might not be suited for haproxy, instead apache, since apache is responsible for serving the ui.15:23
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Add ZaqarApiNetwork to the service net map
EmilienMdtrainor: end of ocata-3 is end of january. Are we going to finish the work by then?15:25
openstackgerritRaoul Scarazzini proposed openstack/tripleo-quickstart-extras: Baremetal undecloud role playbook fixes
trownjistr: idempotency is actually a goal, we just need more CI to get there though. being able to run quickstart jobs on OVB and nodepool will help alot though15:25
trownjistr: actually overcloud-deploy role has the tags in the main task list, so the include of it without a tag is actually fine... I still think there is some issue with the steps between undercloud install and overcloud-deploy but if you are working on THT/puppet-tripleo stuff running with default tags and quickstart-extras.yml playbook should get you where you want15:28
*** liverpooler has joined #tripleo15:28
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: DONT REVIEW: test
* trown reviews ^15:28
pandaderekh: I seems normal, different processes sharing the same socket, especially when you fork and the file descriptors gets duplicated.15:29
derekhpanda: ok15:29
jistrtrown: cool, thanks for the info. Btw the way i used to approach idempotency was to do just very easy checks, which, in case of executing larger operations in one swoop, would err towards the side of not running the larger chunk at all. E.g. "if i see keystone service running on the undercloud VM, then i will not run undercoud install, b/c it was most likely already run". I didn't go checking for all15:30
jistrservices in detail etc. Perhaps a bit subjective approach to ansible idempotency, but seemed to work well for me :)15:31
dtrainorEmilienM, i am aware, and yep.  We're discussing some concerns we've had about some defaults that we'll need for UI in order to accomplish this.  We'll have more answers shortly.15:33
trownjistr: ya that makes sense, there are lots of places in quickstart where there are checks like that, but to me ensuring idempotency requires CI testing it15:34
EmilienMdtrainor: ok, still on track then. Thanks15:37
trownjistr: another trick is to run with "--teardown none --retain-inventory", that will keep quickstart from tearing down the VMs and inventory before running... that is at least how you would uncover where we have bugs with idempotency :)15:38
*** owalsh-afk is now known as owalsj15:38
*** owalsj is now known as owalsh15:38
trownjistr: default is set to simulate starting from scratch because that is the only thing that is CI'd15:38
dtrainoryessir, thanks for bringing it up.15:38
jistrtrown: re "--teardown none --retain-inventory" -- nice, i'll definitely use it at some point. Thanks a bunch :)15:40
trownjistr: just to close loop on tags thing... only one role doesn't have tags, and it is only templating some files so shouldn't intefere, so I am not going to put a bug up for it15:40
jistrack, sounds fine15:41
*** liverpooler has quit IRC15:41
*** liverpooler has joined #tripleo15:41
trownjistr: but I am unconvinced the tags approach was good in the first place, so maybe we can rethink that interface at somepoint15:41
trownprobably just having more example playbooks that are indvidually CI'd with better docs on constructing and using custom playbooks would be better than tags15:43
weshaysshnaidm, think we've figured out why the container validate failed15:43
sshnaidmweshay, great, why?15:43
weshaythe containers are missing a hosts file15:44
weshaysshnaidm, so we're pretty dam close to full success again15:44
weshaythink we're clear to move forward and getting it upstream15:44
*** oshvartz has quit IRC15:44
weshayI see some -1's but I'd almost rather *not* change anything at this point15:45
weshaytrown, maybe you can weigh in15:45
jistrtrown: yea, or perhaps having the checks on playbook level rather than tasks level would make it more DRY and visible (tags on tasks level need to be specified in many places, so it's easier to miss it somewhere)15:45
sshnaidmweshay, ok, then just ping me when it's merged, I'm almost ready to put it in upstream..15:46
derekhpanda: sshnaidm EmilienM: looking back at the last ha job to pass, it was on centos 7.2 , the first job to have the timout error is centos 7.315:47
*** saibarspeis has quit IRC15:47
jistrtrown: also previously i did "stopping points" but those 1) assume you can always safely run from the beginning 2) they "purposefully failed" the ansible playbook at some point, so they gave a false appearance of failure (including return code). So it's kinda dirty and definitely not good for use in CI for example. I probably wouldn't go that way again.15:47
derekhthat doesn't narrow much down, probably means it could be any number of things.....15:47
sshnaidmderekh, that's bad.. seems like some things works there different15:47
derekhcould be15:48
*** masco has quit IRC15:49
trownjistr: ya in quickstart we have a mix of tags, example playbooks, and when variables to control execution... the worst of all worlds :)15:49
EmilienMderekh: mhh, we have fixed puppet-nova to configure nova correctly with the new qemu15:49
EmilienMderekh: do you have logs?15:50
trownjistr: personally I think playbooks are best, just declare exactly what you want to run and run it... but there is some trade-off in user-friendliness to non-ansible users15:50
trownthough the user-friendliness of tags has not been all that either15:52
EmilienMderekh: I don't see qemu bug here15:52
derekhEmilienM: that was the last HA job to pass, it was on 7.215:53
EmilienMderekh: ah ok15:53
*** jlinkes has quit IRC15:55
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras: Look for success message in logs when check status
jistrtrown: yea... personally i don't like tags too much for controlling what roles to execute. It's a bit too verbose / error prone, maybe they work better for more granular, perhaps role-spanning selection ("only generate configs but don't start any services" etc.). The when variables feel a bit better for role selection. And yea i agree custom playbook being cleaner but harder for folks wihout ansible15:59
jistrexperience (especially if the playbook is a bit larger/elaborate).15:59
trownya, when is nice too because it can go right into the yaml config or on the command line, where as tags are command line only16:01
*** Goneri has quit IRC16:01
*** trown is now known as trown|mtg16:01
*** abregman has quit IRC16:01
*** links has quit IRC16:04
*** udesale has quit IRC16:05
*** jbadiapa has quit IRC16:05
*** pkovar has quit IRC16:07
openstackgerritMichele Baldessari proposed openstack/puppet-pacemaker: Relax requires when creating resources on remote nodes
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo: Initial pacemaker remote profile support
*** ooolpbot has joined #tripleo16:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]16:10
*** jmelvin has joined #tripleo16:11
*** pblaho has joined #tripleo16:11
*** Goneri has joined #tripleo16:14
jmelvinping triple0, can someone check this pastebin please. i'm having trouple installing Mitaka undercloud:
*** tremble has quit IRC16:14
EmilienMjaosorior: is it something we still target for ocata-3 ?
*** ayoung has quit IRC16:17
shardyjmelvin: hi, heat::engine::auth_encryption_key looks wrong, it should be 32 characters long16:18
jmelvinshardy, right, but i'm wondering why that would happen?16:18
shardyjmelvin: I assume undercloud_heat_encryption_key isn't set to "16" in your undercloud.conf :)16:19
*** alee has joined #tripleo16:19
jaosoriorEmilienM: so, we are still targetting it for ocata-316:19
jaosoriorEmilienM: the service is already accepted in the big tent, and we're polishing some installation details16:20
jmelvinshardy, they have set: undercloud_heat_encryption_key = 1616:20
shardyjmelvin: Ok, that's the problem then :)16:20
jaosoriorEmilienM: but we do need to finish up the puppet modules in order to be able to take use of it in tripleo16:20
jmelvinshardy, change it to 32?16:20
jaosoriorEmilienM: cause at the moment the TLS work is REALLY tedious to set up, and this will ease a lot of the pain from it.16:20
shardyjmelvin: just unset it then we'll create a random 32 character string16:21
jmelvinshardy, so comment it out right?16:21
shardyjmelvin: yes16:21
EmilienMjaosorior: I understand you're polishing details. But are we going to merge everything before end of january?16:21
jmelvinshardy, thank you sir , i will give that a try16:21
EmilienMjaosorior: I still see a lot of dependencies here16:22
EmilienMand it seems very experimental to me now16:22
EmilienMjaosorior: can you give me list of blockers/wip in the blueprint whiteboard?16:22
jaosoriorEmilienM: will do16:22
EmilienMjaosorior: thanks. I'll see if we keep it for ocata or postpone it.16:23
*** almondjoy has joined #tripleo16:23
jaosoriorEmilienM: we're working very hard to keep it in schedule, without this the TLS-everywhere work is too hard to set up to consider it production-ready16:23
aleeEmilienM, I've been doing installs from tripleo quickstart with the patches I have for awhile now.16:24
EmilienMjaosorior: I know you're working hard :-) you don't need to convaince me16:24
*** jkilpatr_ has joined #tripleo16:24
EmilienMalee: I know it can work on your local dev environment. My concern is to make it work in production and productized in tripleo on time16:24
aleeEmilienM, there are a few dependencies - but we can enumerate them16:24
EmilienMalee: please do in the whiteboard.16:24
EmilienMalee: again, schedule is very short and we have 44 blueprints postponed from ocata-1 :)16:25
aleeEmilienM, understood -- just keep in mind as Oz said, TLS is very tedious to set up without novajoin16:26
EmilienMalee: that's not the discussion here, I know the feature it's important16:26
EmilienMalee: I have no magical power to extend the ocata cycle. It's short and that's it16:26
aleeEmilienM, but we'll enumerate the dependencies so you can see whats needed16:26
EmilienMalee: just keep in mind that we won't push for last-minutes features after ocata-316:27
*** jkilpatr has quit IRC16:27
EmilienMwe tried that during newton and some people here can tell it was not a good idea :-)16:27
aleeEmilienM, I think we're close -- but I'll let you make that determination16:28
aleeI understand things are tight based on holidays etc.16:28
EmilienMshardy: I see some overlap between and - wdyt?16:28
EmilienMalee: we don't have the same definition of "close" :-)16:29
EmilienMalee: as long as I don't see all patches merged and CI testing this feature, it's not close to me, it's in good shape16:29
shardyEmilienM: They are different I think - one is about doing validation between each configuration step, the other is about triggering the tripleo-validations ansible things via mistral16:30
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: TEST: Enable undercloud ssl in ha job
*** jaosorior has quit IRC16:30
EmilienMshardy: yeah but I thought we could use the same tooling, and avoid duplication here16:30
aleeEmilienM, ok - we'll detail where we are - what patches need to merge etc. and go from there.16:31
EmilienMcool thx16:31
*** jaosorior has joined #tripleo16:31
*** pkovar has joined #tripleo16:31
shardyEmilienM: Yeah in some future architecture perhaps we could, e.g it's likely all the validations will be ansible, just triggered differently16:31
shardybut right now we don't configure things via mistral, so they can't be driven the same way16:32
shardyEmilienM: I'll give some thought to how we might combine them in future, but for now I'd say keep them separate16:33
EmilienMshardy: lgtm16:33
*** rcernin has quit IRC16:34
EmilienMtrozet: any progress on congress & tacker? I don't see much updates in the blueprint16:34
EmilienMtrown|mtg: I see for congress but nothing for tacker. Are we still on track for ocata-3? should we postpone the two or one into pike?16:35
EmilienMtrozet: ^16:35
EmilienMshardy: I have some doubts about progress on
*** jaosorior has quit IRC16:39
derekhEmilienM: sshnaidm panda bnemec: see my most recent comment on , that about as much info as I've managed to get so far today16:39
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]16:39
*** jaosorior has joined #tripleo16:40
bnemecderekh: Oh, so this is 7.3 biting us again?16:40
EmilienMit's unclear to me why we only hit that on the ha & uodate jobs16:40
*** ccamacho is now known as ccamacho|out16:41
derekhbnemec: possibly16:41
bnemecEmilienM: I have a theory about that.16:41
bnemecI was looking at the firewall rules in host_info, and the nonha job has a lot more because it enables undercloud ssl.16:41
derekhwe pass some timeout on the HA job maybe?16:41
bnemecSo I pushed to test it.16:41
*** pcaruana has quit IRC16:42
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add hook to generate metadata from service profiles
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add metadata settings for needed kerberos principals
jaosoriorbnemec: won't that fail? since ha uses net-iso while non-ha doesn't. So there will be openssl errors16:43
bnemecjaosorior: It's only the undercloud.  That should work either way.16:43
jaosoriorah, right16:44
jaosorioryeah, then it should be fine16:44
pandaand anyway, this is only to check if testeenv client gets control again16:44
bnemecAlso true, although this would be a nice workaround that would unblock ovb ci until we can figure out what broke.16:45
openstackgerritPradeep Kilambi proposed openstack/instack-undercloud: Disable legacy ceilometer api
openstackgerritPradeep Kilambi proposed openstack/instack-undercloud: Add gnocchi support on undercloud
EmilienMakrivoka: how are we doing with ?16:46
EmilienMakrivoka: I think this one needs to be postponed to pike-1 right?16:46
*** mcornea has quit IRC16:47
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add metadata settings for needed kerberos principals
*** leanderthal is now known as leanderthal|afk16:48
*** hewbrocca is now known as hewbrocca_afk16:49
*** b00tcat has quit IRC16:49
*** bvandenh has quit IRC16:49
*** jpich has quit IRC16:51
bnemecI'm still seeing a lot of multinode jobs failing on deploy timeouts like this:
*** fragatina has joined #tripleo16:53
*** fzdarsky is now known as fzdarsky|afk16:53
*** b00tcat has joined #tripleo16:54
EmilienMbnemec: me too, and I failed to find why16:54
*** rcernin has joined #tripleo16:54
EmilienMbfournie: are we still on track for ?16:55
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here
pradkcould someone please review this again .. just rebased (already had 2 +2's)16:57
*** fragatina has quit IRC16:57
*** abehl has quit IRC16:58
bfournieEmilienM: I'd like to push this pike. The lldp plugin and cli dependencies are slower getting through review and some discussion came up in UI meetings two weeks ago in which lldp reporting may be used for building selection of networks for templates instead of validation, so there may need to be some tweaking done here16:59
*** limao has quit IRC17:00
EmilienMbfournie: ack17:01
*** ohamada has quit IRC17:01
*** bvandenh has joined #tripleo17:04
thervebnemec, EmilienM: looks like a connectivity issue on the node:
*** rbowen has left #tripleo17:06
*** jd__ has quit IRC17:08
*** jd__ has joined #tripleo17:09
*** ooolpbot has joined #tripleo17:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]17:10
openstackgerritBrent Eagles proposed openstack/puppet-tripleo: Add base profile for Octavia services
openstackgerritBrent Eagles proposed openstack/puppet-tripleo: Add initial profiles for rest of Octavia services
*** yprokule has quit IRC17:12
*** zoli|wfh is now known as zoli|gone17:14
*** jkilpatr_ has quit IRC17:15
*** aufi has quit IRC17:17
*** rhallisey has quit IRC17:19
*** rbrady is now known as rbrady-afk17:19
*** jpena is now known as jpena|away17:21
*** [1]cdearborn has quit IRC17:21
*** trown|mtg is now known as trown17:22
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Noop the updates and ha jobs
bnemecderekh: EmilienM: ^We've got a 5 hour queue on ovb jobs right now, which sucks for testing the patches intended to fix the problem.17:24
bnemecI'm thinking we run them selectively on the patches related to fixing the problem and fail fast for everything else.17:25
EmilienMbnemec: can we kill them all and resend our patch that try to fix it?17:25
bnemecEmilienM: I'm not sure of a safe way to kill the running jobs.17:25
*** spredzy has quit IRC17:26
bnemecEven killing the zuul job doesn't actually free up the testenv, so we'd probably just start hitting our quota limits.17:26
derekhbnemec: your thinking we exit 1 at the top of ?17:28
*** lblanchard1 has joined #tripleo17:28
bnemecderekh: Yeah, that's basically what the patch does.17:28
*** jkilpatr has joined #tripleo17:28
derekhbnemec: opp, /me didn't see the patch17:28
bnemecSo we never start the testenv-client and it never hangs.17:28
derekhbnemec: lgtm17:29
*** lblanchard has quit IRC17:30
derekhCould iptables6 be effecting ipv4 connections ? it fails to start on the failing jobs
derekhit started fine on the last job I found that passed17:31
derekhbnemec: EmilienM panda sshnaidm ^17:31
* derekh has gotta run17:32
*** derekh has quit IRC17:33
sshnaidmderekh, it fails consistently last days, also as ip4tables too17:33
*** jbadiapa has joined #tripleo17:33
bnemecThat seems concerning.17:33
sshnaidmbut ip4tables is restarted later and works,17:34
sshnaidmbut I didn't check ip6tables though..17:34
pandaI don't think ip6tables is going to affect ipv4 connection, but if ip6tables fails, maybe iptables has issues too17:34
*** yamahata has joined #tripleo17:35
*** arxcruz has quit IRC17:35
bnemecIt's so weird that deployments work though.17:35
bnemecIf the firewall had failed closed it should be breaking everything in the environment.17:36
*** dtantsur|brb is now known as dtantsur17:36
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Recreate initramfs within loopback image
*** jpena|away is now known as jpena|off17:38
*** thrash is now known as thrash|biab17:40
sshnaidmthere are job without ip6tables failure, but still killed, you can check on status page in gate-tripleo-ci-centos-7-ovb-nonha section:
sshnaidmI meant gate-tripleo-ci-centos-7-ovb-ha section17:41
*** trown is now known as trown|lunch17:43
*** panda is now known as panda|bbl17:44
*** jaosorior has quit IRC17:44
*** milan has quit IRC17:45
*** [1]cdearborn has joined #tripleo17:47
shardyEmilienM: Yeah you can defer to pike - it's probably not that much work, but I can't be sure I'll find time to work on it17:48
shardyI guess if the patches are ready we can move it back to ocata-3, but I don't think it's high priority at this point17:49
EmilienMshardy: ack17:50
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci: Implement major upgrade for Newton to Ocata
*** lucasagomes is now known as lucas-afk17:54
*** lmiccini has quit IRC17:55
dtantsurfolks, do we even enroll nodes in the -undercloud job?17:56
EmilienMdtantsur: YOU§17:57
EmilienMI have questions about your blueprints, stay here :P17:57
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-puppet-elements: opstools: ensure that collectd is available on overcloud images
*** florianf has quit IRC17:58
* dtantsur sloooowly walks away17:58
* EmilienM runs and closes the door with a key and eats the key17:58
EmilienMare we on track for ocata-3? or move it to pike-1?17:59
dtantsurEmilienM, we're still on track with both. RAID one is close to completion, I just need a bit more time. Volume one still looks good enough with Cinder undercloud patch merged today.18:00
openstackgerritgreghaynes proposed openstack/diskimage-builder: Create ubuntu/fedora test for pip-and-virtualenv
EmilienMdtantsur: excellent sir18:00
* EmilienM opens the door magically18:00
dtantsurEmilienM, my question about -undercloud job was actually related to It seems done code-wise, but I need some victim job to enable cleaning there.18:00
dtantsurI don't want to bring more potential for transient failures to ha and nonha jobs though18:01
*** ctayal has joined #tripleo18:01
dtantsurso I thought if we enroll nodes in the undercloud job, it can be a candidate... but I suspect we don't18:01
*** ctayal has quit IRC18:01
*** ctayal has joined #tripleo18:02
EmilienMdtantsur: can you set a priority to both please?18:02
* EmilienM brb lunch18:02
*** fragatina has joined #tripleo18:02
dtantsurEmilienM, for me, volume is high, raid and cleaning are low. I don't have ACL to change priorities in tripleo.18:02
EmilienMdtantsur: let me fix that18:03
*** arxcruz has joined #tripleo18:03
*** fragatina has quit IRC18:03
EmilienMdtantsur: weird, I don't see why it's blocking18:04
EmilienMI'll look after my lunch break18:04
*** fragatina has joined #tripleo18:04
dtantsurmaybe I'm not on some team.. dunno18:04
*** jbadiapa has quit IRC18:04
*** rhallisey has joined #tripleo18:04
*** dtantsur is now known as dtantsur|afk18:05
*** pkovar has quit IRC18:07
*** pkovar has joined #tripleo18:08
*** pkovar has quit IRC18:08
*** bvandenh has quit IRC18:08
shardyhave a good evening all18:09
bnemecdtantsur|afk: We don't enroll nodes in the undercloud job.  It's a standalone vm, so there aren't any nodes for it to manage.18:09
bnemeco/ shardy18:09
*** shardy has quit IRC18:10
*** ooolpbot has joined #tripleo18:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]18:10
bnemecAlthough we could conceivably set up a pyghmi fakebmc on the node and then we could register something at least.18:10
*** athomas has quit IRC18:12
bnemecEmilienM: It would be good to merge asap so we can start clearing the queue.18:13
*** chlong has joined #tripleo18:14
EmilienMbnemec: done18:14
bnemecEmilienM: Thanks18:14
* bnemec crosses his fingers for the multinode gate jobs18:15
mwhahahaEmilienM: do we have an alternative way of rechecking just the ovb jobs? the ML mail about recheck impact made me wonder how many of those rechecks were because ovb jobs were flakey but resulted in a full recheck18:21
openstackgerritLeif Madsen proposed openstack/tripleo-quickstart: Allow basic install to work on Fedora
*** rasca has quit IRC18:22
*** amoralej is now known as amoralej|off18:27
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Deployed server bootstrap via Heat
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Deployed server bootstrap via Heat
EmilienMmwhahaha: we don't until now18:31
mwhahahashould probably have one18:31
mwhahahaespecially if we are not merging stuff if ovb is red18:32
EmilienMmwhahaha: we might need to investigate in project-config. Since tripleo ovb jobs are in a different pipeline it might be possible, let me look18:35
EmilienMmwhahaha: yeah I think we can18:36
EmilienMmwhahaha: let me propose it18:36
mwhahahaWe should be able to18:37
*** rlandy has joined #tripleo18:37
*** rbrady-afk is now known as rbrady18:37
openstackgerritLeif Madsen proposed openstack/tripleo-quickstart: Allow basic install to work on Fedora
*** bana_k has joined #tripleo18:39
*** trown|lunch is now known as trown18:44
*** gfidente has quit IRC18:44
*** thrash|biab is now known as thrash18:44
openstackgerritTim Rozet proposed openstack/puppet-tripleo: Corrects default for glance API service
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Re-enable graphite metrics
*** dprince has quit IRC18:57
*** dprince has joined #tripleo18:58
openstackgerritDan Prince proposed openstack/python-tripleoclient: Deploy the undercloud with Heat
*** rlandy is now known as rlandy|mtg19:00
*** dsneddon has joined #tripleo19:01
ansiwenmy build still fails because puppet-ec2api is not installed. it is included rdo.yml, but it is not added to the PROJECTS variable. So where do I have to add it so it is installed? See here:
bnemecansiwen: You may need to add it to this list:
*** ooolpbot has joined #tripleo19:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]19:10
bnemecAlthough I've never been entirely clear on how that works.19:10
openstackgerritMerged openstack/diskimage-builder: Recreate initramfs within loopback image
*** fragatina has quit IRC19:11
ansiwenbnemec: ok, thanks... but that's weird. I mean, when the undercloud is installed, there must be some mechanism independent of the CI infra to install the required puppet packages. so is this mechanism not used in CI? I'm confused...19:13
*** ctayal has quit IRC19:14
ansiwenbnemec: where does this list of packages come from? see:
*** chem has quit IRC19:14
sshnaidmbnemec, it worked:
bnemecansiwen: Ah, then it probably needs to be added to the opm dep list:;a=blob;f=openstack-puppet-modules.spec;h=4df642ca2734d20fa7f98a77a81ef882f266036b;hb=HEAD19:16
*** [1]cdearborn has quit IRC19:17
ansiwenbnemec: aha! thanks!19:17
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: make quickstart-extras-requirements.txt a default requirements file
*** dprince has quit IRC19:18
*** [1]cdearborn has joined #tripleo19:18
bnemecsshnaidm: Have we figured out if that's going to break testenv cleanup?19:19
*** dprince has joined #tripleo19:20
trozetEmilienM: how long do I have till Ocata-3?19:22
trozetdsneddon: hi19:27
weshayEmilienM, ping.. there are periodic versions of  the puppet jobs right? e.g. gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/  /me looking for logs19:27
*** ctayal has joined #tripleo19:27
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: make quickstart-extras-requirements.txt a default requirements file
*** ctayal has quit IRC19:33
openstackgerritBen Kero proposed openstack/tripleo-quickstart-extras: Add generic undercloud setup role
*** rbrady is now known as rbrady-afk19:34
*** sudipto_ has quit IRC19:36
*** sudipto has quit IRC19:36
*** dprince has quit IRC19:38
*** ctayal has joined #tripleo19:39
*** dsariel has joined #tripleo19:39
*** sudipto has joined #tripleo19:41
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Deployed server bootstrap via Heat
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Set rabbitmq package_provider to yum
openstackgerritPradeep Kilambi proposed openstack/instack-undercloud: Add gnocchi support on undercloud
openstackgerritPradeep Kilambi proposed openstack/instack-undercloud: Disable legacy ceilometer api
*** sudipto has quit IRC19:49
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: make quickstart-extras-requirements.txt a default requirements file
*** nyechiel has quit IRC19:53
*** nyechiel has joined #tripleo19:53
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Add undercloud roles data and environment
*** rlandy|mtg is now known as rlandy19:56
*** dsneddon has quit IRC20:00
*** dsneddon has joined #tripleo20:01
dsneddontrozet, Hello20:01
trozetdsneddon: hey, so I filed a bug a little while ago to change the default glance backend to be file rather than swift20:02
dsneddontrozet, OK, got the bug link?20:02
EmilienMweshay: periodic jobs for puppet? no20:03
EmilienMweshay: we have one, for updating our constraints though20:03
trozetdsneddon: but now I am trying to figure out how this will work for HA, because it would need to create NFS by default and I only see a way to mount to an NFS with puppet-tripleo/manifests/glance/nfs_mount.pp20:03
openstackLaunchpad bug 1650348 in tripleo "Glance API service defaults to Swift storage" [Undecided,In progress] - Assigned to Tim Rozet (trozet)20:04
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: make quickstart-extras-requirements.txt a default requirements file
trozetEmilienM: radez was going to do the Congress part and I was going to work on tacker.  I think I can get tacker done by ocata-3, just need to get started on it20:04
EmilienMtrozet: well that's the problem20:05
EmilienMtrozet: blueprints not started by end of ocata-2 won't like be done by ocata-3. Have you seen the short schedule?20:05
trozetEmilienM: we already implemented for our THT fork in mitaka, I just need to port it to a composable service20:05
EmilienMtrozet: I haven't seen any progress until now on tacker until now20:05
trownEmilienM: is it a known issue that multinode job passes in check then fails in gate? or am I just really unlucky on
EmilienMI have serious doubts of finishing tacker and congress on time for end of january tbh20:06
trownEmilienM: that patch has passed in check only to fail in gate 5 times now20:06
EmilienMtrown: :(20:06
trozetEmilienM: well whats wrong with leaving it for now, and then update it beginning of Jan?20:07
EmilienMtrozet: have you seen my email about release status on openstack-dev?20:07
*** panda|bbl is now known as panda20:08
EmilienMtrozet: we won't push features at the end of january anymore, which means doing congress and tacker in less than one month is unrealistic, specially regarding holidays, etc20:08
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: make quickstart-extras-requirements.txt a default requirements file
trozetEmilienM: OK20:08
EmilienMweshay: what's your specific question for? what is the goal?20:08
trozetEmilienM: then I guess there is no choice but to move it?20:09
EmilienMtrozet: well, I'm asking you so you have the choice20:09
EmilienMtrozet: I just want to let you know that if by end of january tack and watcher are not merged and fully tested in CI, any patch will be -2 until we open Pike development.20:09
*** ooolpbot has joined #tripleo20:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]20:10
weshayEmilienM, to get the set of puppet and packstack upstream tests running on along w/ the tripleo-periodic jobs on the same hash20:10
trozetEmilienM: OK, I will see if I can get it working before the break and post patches20:10
weshaymeh.. maybe it's just best done in the set of rdo jobs we have20:10
EmilienMweshay: we don't have upstream periodic jobs for promotions. Ask dmsimard, we have RDO jobs in centos CI20:11
trozetdsneddon: is there anything that will configure each controller as an NFS server so you can use glance HA with NFS default out of the box?20:11
weshayya.. k20:11
dsneddontrozet, We don't have that today.20:11
dsneddontrozet, I mean, there actually probably are Puppet and Ansible scripts to do that, but they aren't integrated into RDO20:12
dsneddontrozet, er, tripleo, I mean20:12
trozetdsneddon: so then that means that file backend cannot be the default for HA glance api20:12
trownEmilienM: wrt your convo with lucas about virtualbmc work (just saw in scrollback looking for something else) it is only relevant for virt-setup20:12
trownEmilienM: and the only CI we have for virt-setup is quickstart20:13
EmilienMtrozet: we have 47 blueprints for Ocata 3 and we finished 1 blueprint in ocata-2. Trust me, it's not easy to manage20:13
dsneddontrozet, But that sounds like a good case for a composable service.20:13
trozetdsneddon: so then what if default is file for nonHA and Swift is default for HA, or we incorporate the missing puppet like you say to create NFS20:13
*** kjw3 has joined #tripleo20:13
EmilienMtrown: oh, ok... thanks20:13
trownEmilienM: it replaces pxe_ssh20:13
trozetdsneddon: so that is my dilemma...if a user does not include swift in his composable role with glance API, glance wont work currently20:14
trozetdsneddon: i feel like that is wrong, and the default should be file (the glance default), and not rely on another composable service being included by default20:15
trozetdsneddon: what do you think?20:15
dsneddontrozet, Yes, I agree with you, that's a good default model.20:16
*** derekh has joined #tripleo20:17
trozetdsneddon: so we can do that with a simple change for nonHA obviously, but HA we run into the problem where no NFS is created by default.  So how do we solve that for now?20:17
derekhbnemec: EmilienM sshnaidm panda its not nice but if we want it this appears to have worked
derekhthe testenv was clean up 20 minutes after the testenv client got killed
EmilienMderekh, bnemec: should we cancel ?20:18
EmilienMderekh: why 20 min?20:18
dsneddontrozet, Well, I think "for now" is probably requiring Swift with HA, but "real soon" could be an NFS composable service.20:18
derekhon a train at the moment with patching internet, just popping in to let ye know so ye can decide20:19
trozetdsneddon: OK that makes sense to me20:19
EmilienMderekh: i've +2, I'll let bnemec voting (approving one and block other)20:19
EmilienMderekh: I'm also afk for an errand now20:19
derekhEmilienM: let me see if if one of the many timeouts we have in the code20:19
dsneddontrozet, I haven't tried it, but does the Cinder service like RBD as a backend?20:20
openstackgerritLars Kellogg-Stedman proposed openstack/puppet-tripleo: add support for collectd
trozetdsneddon: so then my first thought is in environment/puppet-pacemaker.yaml, set parameter default for GlanceBackend to be swift, but then that doesnt work if someone passes storage_environment.yaml to say use ceph.  I don't know which parameter default it will take20:20
bnemecderekh: EmilienM: Yeah, I'm still a little concerned that we may end up leaving testenvs sitting around after jobs complete.20:20
bnemecI think they eventually get cleaned up, but if they sit there for an extra hour we're going to have quota problems.20:20
EmilienMbnemec: yeah :(20:21
derekhbnemec: its certainly possible, I'm happy to hold onto the patch until we run out of other options20:21
weshayEmilienM, container ci is passing the ping test.. w/
trozetdsneddon: we tested Cinder with hyperconverged ceph, seems to work OK20:22
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates: add collectd composable service
trozetdsneddon: but you remind me of another funny thing.  In the ceph-base.yaml, the heat parameter GlanceBackend is also defined, and it is defaulted to swift, which also makes no sense20:22
EmilienMweshay: where do you see it?20:22
*** eggmaste` has joined #tripleo20:23
*** eggmaster has quit IRC20:24
dsneddontrozet, Here's how inheritance works: first, everything in overcloud-resource-registry.yaml is evaluated, to set the initial resource definitions. Then, overcloud.j2.yaml sets all the default parameters, which are available to any subsequent included resources. Then, files included on the command line get evaluated, and any parameters set in overcloud.j2.yaml are overridden. If you include multiple files on the deployment command-l20:24
dsneddonine, then the last one included wins.20:24
*** tzumainn has quit IRC20:24
*** tzumainn has joined #tripleo20:25
dsneddontrozet, So if you set a default parameter in an environment file, that overrides the puppet-pacemaker.yaml.20:25
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci: Postci tests trial and error
trozetdsneddon: but that is only the environment file is specified after the -e puppet-pacemaker.yaml, right?20:25
dsneddontrozet, But, if you set a top-level parameter (not a parameter default), then subsequent parameter_defaults on the same property won't override the top-level parameter.20:25
dsneddontrozet, Right, you want to include puppet-pacemaker.yaml first, so that environment files specified after can override those params.20:26
trozetdsneddon: yeah but GlanceBackend isnt a global overcloud parameter, so cant use parameter:20:26
trozetdsneddon: yeah so we don't want order to it doesnt seem liek the solution is to set the parameter default in puppet-pacemaker.yaml20:27
*** ayoung has joined #tripleo20:27
trozetdsneddon: there is a pacemaker/glance-api.yaml, but doesnt look like it is used anymore:
dsneddontrozet, What you may need to do is set a top-level parameter, not parameter_default, so it won't be overridden.20:29
trozetdsneddon: i guess i could fix that to override the parameter20:29
trozetdsneddon: right, that would work too20:30
dsneddontrozet, What's the exact parameter(s)?20:31
*** shinobu|afk has joined #tripleo20:31
trozetdsneddon: so create the top level parameter, override the parameter_default in puppet-pacemaker, then change env files to override parameter instead of parameter_default20:31
trozetdsneddon: GlanceBackend20:31
trozetdsneddon: i think your solution is the best one20:32
derekhbnemec: EmilienM sshnaidm the geard server has keepalives set with the following params tcp_keepidle=7200, tcp_keepintvl=75, tcp_keepcnt=920:32
openstackgerritBen Kero proposed openstack/tripleo-quickstart-extras: overcloud-deploy: add variable for hypervisor_wait
derekhso after 2hrs of a idle tcp connection, it sends up to 9 keepalives every 75 seconds20:33
*** shinobu__ has quit IRC20:33
derekhthats 131 minues,20:34
derekhthe te got cleaned up almost exactly 131 minutes after it was created20:34
dsneddontrozet, I don't see that set in puppet-pacemaker.yaml.20:35
trozetdsneddon: it's not.  I'm adding it20:35
derekhso as best I can see if force killed testenvs will be cleaned up 131 minutes after they were created20:35
dsneddontrozet, Ah, of course.20:35
dsneddontrozet, Anyway, you want to set it as a parameter_default in puppet-pacemaker.yaml if you want to be able to override it later.20:36
trozetdsneddon: exactly.  thanks.  I will look into figuring out how to add NFS as a composable service20:36
*** kbyrne has quit IRC20:36
derekhat least that what the evidence would suggest*20:36
*** flepied has quit IRC20:36
openstackgerritMerged openstack/tripleo-quickstart-extras: overcloud-deploy: add variable for hypervisor_wait
dsneddontrozet, I'm happy to help if you have questions about the composable service, although shardy is probably the subject matter expert.20:37
trozetdsneddon: i think i've got the hang of that now after doing the ODL one20:38
*** shinobu|afk has quit IRC20:38
*** derekh has quit IRC20:41
trozetdsneddon: so one thing, if I define a top-level parameter, don't I need to pass it to the role's properties in overcloud.j2.yaml?20:49
trozetdsneddon: here I mean
dsneddontrozet, Yeah, that's what I mean by a top-level parameter.20:51
*** weshay is now known as weshay_bbiab20:51
dsneddontrozet, Although you can define parameters in environment files, but those will actually be considered separate and limited in scope to the nested stack.20:51
trozetdsneddon: so if I put it there as a property for hte dynamic roles J2 stuff, if a role does not include glance-api (with the local GlanceBackend param) will that fail?20:51
trownlarsks: bug for the ironic issue
openstackLaunchpad bug 1650374 in tripleo "[instack-undercloud] package upgrade should happen before service configuration" [Critical,Triaged]20:52
larskstrown, thanks!20:53
*** flepied has joined #tripleo20:54
akrivokaEmilienM: yes, most likely postpone to pike20:55
trownlarsks: oddly we have dependencies explicitly setup in the other direction, seemingly intentionally,
larskstrown, are those relevant?  For this fix we would want a notification dependency on the ironic-conductor service, right?  Or on all 'openstack' services. Or something like that. Right?20:58
trownlarsks: hmm ya, that is making sure openstack packages are upgraded first?21:02
dsneddontrozet, I am pretty sure that if you don't include the Glance role, that property won't matter, since the parameter will never be referenced.21:03
dsneddontrozet, Just like we set all sorts of parameters for network isolation that get ignored if you don't include network-isolation.yaml.21:04
trownEmilienM: is there a way to reference all openstack services in puppet... seems they are not tagged with openstack21:04
mwhahahatrown: no there is not21:04
mwhahahatrown: re: 1650374, shouldn't that get fixed with dependency for the ironic package? rather than updating all the packages before openstack?21:05
trownmwhahaha: how so? (does seem like potential packaging issue just not sure what the fix would be)21:06
*** Goneri has quit IRC21:07
mwhahahatrown: not sure but it seems like an issue in some dependency where it's not properly ensureing a min version or something21:07
trownmwhahaha: oh, ya it is not a dependency issue in that sense... simply restarting the service fixes it... but it may be a systemd service file issue21:08
mwhahahatrown: that error didn't seem to be ironic specific but rather some python crypto21:08
trownI see ironic-conductor has "Restart=on-failure" and other openstack services look to have "Restart=always"21:08
larskstrown, that just impacts what systemd will do when the service exits.21:09
trownlarsks: ah ok... found examples of both now actually too21:09
*** egafford has quit IRC21:10
*** ooolpbot has joined #tripleo21:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]21:10
mwhahahatrown: my thought on why it was a packaging issue is because it sounds like either it's relying on some other dependency that's not properly being pulled in when ironic gets installed.  the bug was for initial install right?21:10
larskstrown, also, is this actually a packaging problem?  That is, should the ironic package either (a) require a min. version of the appropriate python module?  Does conductor work *before* the upgrade?21:10
trownmwhahaha: no, that trace happens when importing nodes21:10
larsksOr (b) should the package take care of the restart?  That gets tricky, I guess.21:10
*** mhenkel has quit IRC21:11
trownmwhahaha: so ironic-conductor is happily running but with some outdated stuff in memory21:11
*** mhenkel has joined #tripleo21:11
mwhahahatrown: yea that's what i meant by the packaging thing because it shouldn't be, it should have had everything it was going to need before. so it's some package getting udpated breaks it because its a soft dependency or something and that package is not properly backwards compatible21:12
trozetdsneddon: ok cool21:13
mwhahahatrown: but this was a fresh install on a 7.2 image with a 7.3 packages, then as part of the install we do a package update right? just trying to understand how the system was put together21:13
*** jayg is now known as jayg|g0n321:13
trownmwhahaha: ya, it is a quickstart image built on 7.2, but the base repos for CentOS are now 7.3, so pretty much every base package gets updated when doing the undercloud install21:14
trownmwhahaha: once we promote a 7.3 based image the issue will go away (until the next time it happens)21:14
trownmwhahaha: a similar issue cropped up again with ironic-conductor when dowstream imported newton: ... so it does seem specific to ironic-conductor21:16 bug 1391602 in openstack-ironic "fail: openstack baremetal import --json instackenv.json, Exception registering nodes: No valid host was found. Reason: No conductor service registered which supports driver..." [High,Closed: worksforme] - Assigned to lmartins21:16
trownin that case though ironic-conductor was in a failed state post undercloud install21:16
mwhahahatrown: so that error is probably coming from python2-cryptography which i guess was 1.2.1 in 7.2 and is now 1.3.1 in 7.321:17
trownin this case it is running fine... just unable to actually handle pxe_ssh21:17
trownmwhahaha: indeed that is one of the packages that changed21:17
mwhahahatrown: it appears that 1.3.1 is not properly backwards compatible with 1.2.121:18
* mwhahaha shrugs21:18
trowntrozet: is this what you were seeing too (that made you hack out the upgrade from Apex CI)
openstackLaunchpad bug 1650374 in tripleo "[instack-undercloud] package upgrade should happen before service configuration" [Critical,Triaged]21:18
mwhahahaother than doing a  yum update prior to doing anything, not sure we should be accounting for this in tripleo21:18
trozettrown: it is the same error21:19
trownmwhahaha: wdyt of just adding a dependency on Service<| tag == 'ironic-service' |> -> Exec<| title == 'package-upgrade' |>21:19
trozettrown: although the reason for stopping package upgrade is bigger than just that error21:19
*** panda is now known as panda|Zz21:20
trozettrown: i really dont think yum update  should be in the install at all21:20
trowntrozet: ya, I agree (hence making it optional) :)21:20
mwhahahatrown: i think that's just going to end causing more problems21:20
openstackgerritBrent Eagles proposed openstack/puppet-tripleo: Add base profile for Octavia services
trozettrown: yeah but it shoudl be off by default.  If you produce some software that you validate with CI that it works, then when a user uses it, his software changes compared to what you validated against...21:20
mwhahahathis really seems like the base image needs to be the same point release as everything else21:20
openstackgerritBrent Eagles proposed openstack/puppet-tripleo: Add initial profiles for rest of Octavia services
mwhahahai would maybe we should add a check for that21:21
mwhahahabut we'll come up with 50 million work arounds every time a new one is releases21:21
mwhahahait might just be better to say sorry you need to upgrade your node to A.B before installing21:22
mwhahahaand leave it at that21:22
trownmwhahaha: well it is tricky with the upgrade in instack-undercloud... because the base image can be 7.2, but as soon as you run upgrade it is 7.321:22
mwhahahatrown: that's what i'm talking about, that shouldn't happen21:22
trownah... so run yum upgrade manually before running `openstack undercloud install`21:22
trownthat seems reasonable to document actually21:23
mwhahahawe should add a check in the instack-undercloud install to make sure we're at least A.B or something21:23
trownthough then the upgrade in instack-undercloud is a bit pointless21:23
mwhahahanot necessarily21:24
trownand without that, we wouldnt have this situation to begin with :P21:24
mwhahahaso maybe i'm confused, was this a fresh install or an upgrade21:24
mwhahahaI'm getting lost in what is being 'upgraded'21:24
mwhahahai'm saying the OS version on a new install should be at least a min version of something21:24
*** dsneddon has quit IRC21:25
mwhahahaso the tripleo repo should say, this was built against 7.3 for example. min version to install on is 7.321:25
mwhahahawe do not allow install to run on 7.2, please upgrade to 7.3 first21:25
mwhahahathen install21:25
trownya it is a fresh install... but on an image with packages pre-installed21:25
mwhahahai really don't like the images with package pre-installed for this specific reason but that's for another day21:26
trownwell, ya we are kind of getting the worst of both worlds by having some packages getting updated and some static21:26
mwhahahafor the undercloud, i think it should just be a blank os image21:27
mwhahahathe overcloud, it might make sense21:27
trownbut ideally we validate some content and it doesnt change so that others could reproduce it or use it for tripleo based projects ... ie OPNFV Apex21:27
trownhistorically, there was not a single test day in RDO that was successful before we adopted the image approach, so I think validated content has been a big win... the upgrade during undercloud install does mess that up a bit though21:28
mwhahahamaybe we need to shuffle the order of things, might it make more sense to config repos, yum update, then install?21:29
mwhahahait seems like we config repos, install, then yum update21:29
trownanywho... I think adding an upgrade in quickstart in the case an upgrade in the undercloud install will do it anyways will fix the current issue21:30
mwhahahaor is ironic-inspector already running before the installation and since it doesn't get anything new it's never restarted21:30
trownhmm... not sure... I dont think it should be running, but the alternative doesnt make a ton of sense either21:31
trowngotta run, will put some more in the bug later21:31
trowntrozet: going to see yurtseven :)21:31
mwhahahai'll take a look in a bit if i get some free cycles21:31
trozetmwhahaha: I think actually you want the opposite.  You don't want a blank OS image with packages installed later21:31
mwhahahatrozet: why's that?21:32
openstackgerritSteve Baker proposed openstack/tripleo-heat-templates: set -e in
*** trown is now known as trown|outtypewww21:32
trozetmwhahaha: that way you have control over a piece of undercloud software you know works21:32
mwhahahatrozet: shouldn't that be a static repo that has been validated?21:32
mwhahahatrozet: shouldn't the specific rdo repo be that?21:33
trozetmwhahaha: that is another way, but when you do yum update * in your install...21:33
mwhahahatrozet: So when I said yum update I meant base OS, not necessarily the RDO stuff as well21:33
trozettrown: oh is he "unbanned" tonight?21:33
openstackgerritSteve Baker proposed openstack/tripleo-heat-templates: Don't run inside docker
mwhahahatrozet: so you end up with a specific OS release -> specific RDO packages (which should be validated on the OS release)21:34
trozetmwhahaha: yeah but OOO/RDO cant control what gets updated in centos repos21:35
trozetmwhahaha: so then if centos repo like this last bug updates a package that we depend on, and it isnt compatible...21:35
trozetmwhahaha: when we deploy and overcloud image, we dont do a yum update on the image and connect to the internet right?21:36
mwhahahatrozet: yea makes sense. I think the issue here is that python2-cryptography is being updated later but that ironic relies on it is never restarted when it is updated21:36
trozetdeploy an*21:36
mwhahahatrozet: depends on how you're doing yum update, could be offline repos or mirrors. the point was more to be able to have consistent artifacts for OS/RDO as part of the install since we're getting a partial update of some stuff at some point21:37
trozetmwhahaha: yeah I just think that if trown produces an undercloud qcow2 and overcloud qcow2, they should contain the same software when i deploy it 3 weeks from the day it was validated and passed CI21:38
trozetmwhahaha: otherwise the validation with CI really means nothing21:38
mwhahahatrozet: but that means that image is tied to a specific point in time OS dependency repo21:39
mwhahahatrozet: which is not the case in either scenario21:39
trozetmwhahaha: what do you mean it isnt hte case?21:40
mwhahahacentos 7.2 image got centos 7.3 after the fact21:40
trozetmwhahaha: right21:40
trozetmwhahaha: i'm saying thats how it should be, not how it is21:40
mwhahahatrozet: yea i'm agreeing :D21:41
trozetmwhahaha: oh, cool :)21:41
mwhahahatrozet: that's what i was kinda pointing at if you start with a known good base OS, when you configure the openstack repo for install,  you get all the packages you need21:41
mwhahahabut that's not what happens since we're also relying on some of the base OS repo packages21:41
mwhahahaand in this case, python2-cryptography changed between 7.2 and 7.321:42
mwhahahaso it would be nice if the openstack repo/artifacts/whatever you want to call it knew what the min version of the base os was required for it21:42
mwhahahagiving you something similar to what we'd have in an image build21:43
trozetmwhahaha: oh you mean if you dont use the pre-build image?21:43
mwhahahaand even the prebuilt image, you would be able to know that the following update has to occur prior to installing openstack21:43
*** jkilpatr has quit IRC21:44
mwhahahaso if you had a 7.1 image, you go to run the install and it says you must update to 7.3 as this version has only been validated against 7.321:44
trozetmwhahaha: that woudln't happen because the image would have been built with 7.321:44
trozetmwhahaha: if you have the image, then nothing needs to be updated it should just work21:45
mwhahahatrozet: thats if your image had everything installed upfront which i don't think to be the case21:45
trozetmwhahaha: but if you just install undercloud on your host, and you need to pull packages, then it should install specific packages you need, and those should resolve dependencies, not just yum update all I think21:45
*** dsneddon has joined #tripleo21:46
trozetmwhahaha: it does, we install with no internet21:46
*** fragatina has joined #tripleo21:46
*** iranzo has quit IRC21:46
mwhahahatrozet: right but i'm not referring to that use case as that's a specific super controlled case21:46
*** shinobu_ has quit IRC21:47
mwhahahatrozet: i'm thinking more how I interact with the undercloud where I don't have an image for it. I have a basic centos VM that I then configure repos and install. which to me would be more like what someone might do at company XYZ looking to evaluate or start up a new cloud install21:48
mwhahahatrozet: There are folks who don't want tripleo doing the image build/install part which is where this specific issue will come up more often21:49
bnemecEmilienM: Good news!  I think we can just turn on undercloud ssl to unblock the ha and updates jobs:
*** almondjoy has quit IRC21:49
trozetmwhahaha: yeah so in that case where you just have your own VM and want to install undercloud, it should install specific packages and not do yum update.  To your point, if it tried to install some package that relied on the kernel updating, then it should warn you or something like you said about min reqs21:52
*** shinobu_ has joined #tripleo21:52
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Enable undercloud ssl on all OVB jobs
*** akrivoka has quit IRC21:59
*** fultonj has quit IRC21:59
*** jkilpatr has joined #tripleo22:01
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: DONT REVIEW: test
*** ansmith has quit IRC22:01
sshnaidmbnemec, any idea why ssl helps..?22:05
bnemecsshnaidm: The undercloud ssl config opens a bunch more ports in the firewall.  That's my best guess at this point.22:06
bnemecIt also explains why that didn't fix the mitaka jobs.  Those had a much simpler undercloud ssl config that didn't open extra ports (IIRC).22:06
sshnaidmbnemec, maybe it's worth to look at dropped packets in syslog, I've seen a few iptables logs there, maybe some of them from geard server..?22:07
sshnaidmwill look now..22:08
bnemecI need to reboot for a system update.22:09
*** ooolpbot has joined #tripleo22:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]22:10
*** ooolpbot has quit IRC22:10
*** ayoung has quit IRC22:10
*** bnemec has quit IRC22:10
*** dsneddon has quit IRC22:15
*** bnemec has joined #tripleo22:16
*** lblanchard1 has quit IRC22:17
openstackgerritSteve Baker proposed openstack/tripleo-heat-templates: Use the local collector to bootstrap deployed servers
openstackgerritSteve Baker proposed openstack/tripleo-heat-templates: docker: expose agent mounts as template param
larsksIn THT master, KeyStoneFernetKey0 (and 1) are now required parameters?22:20
EmilienMbnemec: oh wow22:21
EmilienMbnemec: should we block ?22:21
EmilienMlarsks: yes but generated by tripleoclient22:22
EmilienMyou don't need to specify them22:22
larsksEmilienM, I guess that means I need to update my tripleoclient...22:22
EmilienMlarsks: for sure :)22:22
bnemecEmilienM: Probably.  Once we get the workaround merged, most of the jobs will go back to finishing in a reasonable time and the queue should clear up by itself.22:22
EmilienMbnemec: sounds good. This week was terrible.22:23
bnemecEmilienM: Of course it was.  I just got back from PTO.  It's like a cosmic rule or something. ;-)22:24
EmilienMbnemec: i know the feeling :D22:24
*** liverpooler has quit IRC22:25
*** dsneddon has joined #tripleo22:26
*** egafford has joined #tripleo22:31
*** liverpooler has joined #tripleo22:31
*** jcoufal has quit IRC22:34
*** egafford has quit IRC22:35
*** rhallisey has quit IRC22:35
*** dsavineau has left #tripleo22:35
*** sai is now known as sai-out22:36
*** saneax-_-|AFK is now known as saneax22:40
*** bfournie has quit IRC22:47
*** liverpooler has quit IRC22:47
*** saneax is now known as saneax-_-|AFK22:48
Ngso I got some great hints earlier for making mistral notice new actions, but I'm also trying to hack on python-tripleoclient and even after blowing away openstackclient and tripleoclient and re-installing them both with, it's not seeing my added commands (which are in setup.cfg)22:50
NgI'm sure I'm missing something simple and obvious22:50
*** myoung is now known as myoung|afk22:50
*** nyechiel has quit IRC22:51
*** morazi has quit IRC22:53
*** jmelvin has quit IRC22:54
*** rlandy is now known as rlandy|bbl22:55
*** egafford has joined #tripleo22:55
*** flepied has quit IRC23:00
*** saneax-_-|AFK is now known as saneax23:03
*** ctayal has quit IRC23:05
*** ctayal has joined #tripleo23:06
*** ooolpbot has joined #tripleo23:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged]23:10
EmilienMbnemec: how do you want to proceed?23:15
EmilienMwe merge your stuff and revert the noop later?23:15
EmilienMyour stuff == ssl/undercloud23:16
bnemecThe noop actually failed the gate anyway, so it hasn't gone in.  I would propose we just merge the undercloud ssl instead.23:17
bnemecEmilienM: ^23:17
*** egafford has quit IRC23:18
EmilienMbnemec: I agree23:25
EmilienMbnemec: have you seen it passing CI?23:25
bnemecEmilienM: It's passed all the voting multinode jobs, and looks like the updates job is going to timeout on the same kind of hang we're seeing elsewhere.23:26
bnemecAnd the same basic patch got the ha job working in my test commit.23:27
bnemecAlso, the ovb jobs are completely hosed right now so it can't really make it any worse. :-)23:27
EmilienMbnemec: even if we +A now,i'm not sure it will move in gate directly23:27
bnemecEmilienM: Yeah, we'll have to wait for the multinode updates job to timeout first, but it should go in after that.23:28
EmilienMbnemec: I'm ok with that, really23:28
*** ayoung has joined #tripleo23:29
*** ctayal has quit IRC23:32
EmilienMbnemec: I need to leave for a couple of hours - again feel free to approve it if you see it passing. Otherwise I'll do it later in the evening. Maybe we can unblock CI for tomorrow, who know :)23:32
bnemecEmilienM: I already pulled the trigger.  I left a big comment explaining my reasoning. :-)23:32
*** ctayal has joined #tripleo23:33
*** pradk has quit IRC23:38
*** tzumainn has quit IRC23:50
*** [1]cdearborn has quit IRC23:53

Generated by 2.14.0 by Marius Gedminas - find it at!