Thursday, 2016-06-23

prometheanfireianw: we may have an update out00:01
prometheanfireiirc we redo our image a couple of times a week and one should be done tonight or tomorrow00:02
prometheanfiregentoo is still nv right?00:02
ianwit is00:02
*** tqtran has quit IRC00:03
prometheanfirethis is why :P00:03
prometheanfireI knew this would happen every now and then when a major package get's updated00:04
prometheanfirepart of what dib does is update the system00:04
prometheanfireso gcc, and glibc might take a while, too long for gate I think00:04
*** baoli has joined #openstack-infra00:06
ianw2016-06-22 15:40:59.704925 | x86_64-pc-linux-gnu-ranlib: libbackend.a: No space left on device <- seems to be the root cause00:06
*** baoli has quit IRC00:06
ianwi wonder if it's doing this on a tmpfs00:07
*** lmiccini has joined #openstack-infra00:07
ianwyeah, i think it is ... we should probably set no tmpfs in the job00:07
fungii'm inclined to agree. currently on nodepool.o.o... /dev/mapper/main-nodepoolbuild 1008G  528G  480G  53% /opt00:08
jesusauror tell portage to build gcc/glibc in a different location00:08
fungiso i don't think it's running out of /opt00:08
fungior put differently, i seriously doubt it takes nearly 0.5tb of scratch space to build recent glibc ;)00:09
prometheanfireianw: I'd +1 that00:10
ianwjesusaur: in this case, it's chrooted into a tmpfs so there's nowhere to go00:10
fungiall boxed in, frightened and vulnerable00:10
prometheanfirewe should let it go00:10
ianwfungi: this is juts for the functional test jobs, i'm 99% sure we do set the "notmpfs" variable for dib on nodepool builds00:10
fungioh, the jobs, not the image builds. got it00:11
fungiyeah, also our workers may be severely limited on available disk depending on where you're trying to set your scratch space00:11
prometheanfiresure it's glibc it's rebuilding?00:11
ianwprometheanfire: it's gcc ->
*** stevemar has quit IRC00:13
*** xarses has quit IRC00:14
*** stevemar has joined #openstack-infra00:14
fungihah, you're not my mutt window!00:14
*** baoli has joined #openstack-infra00:14
ianwputs on obi-wan cape ... "That's a failure I haven't seen in a long time" ... not since building x86-to-x86 "cross"-compilers for various dubious reasons for ESXi userspace00:14
fungiany reason involving esxi counts as "dubious"00:15
*** baoli has quit IRC00:15
openstackgerritMerged openstack-infra/system-config: Refactor gerrit commentlinks parameter
ianwit was actually pretty cool in that the entire build system was versioned together with the code, sort of like
prometheanfirestart here
*** stevemar has quit IRC00:18
prometheanfireI think it's the multislot support that did it00:19
*** amitgandhinz has joined #openstack-infra00:20
*** sarob_ has quit IRC00:20
prometheanfirebecause it was just a reinstall00:20
prometheanfireianw: this commit did it our side00:21
prometheanfireianw: it's the --with-bdeps=y that causes us to rebuild00:22
prometheanfireit's best practice to do it, but not technically needed00:22
*** akshai has quit IRC00:24
ianwgotta -funroll-loops :)00:25
*** amitgandhinz has quit IRC00:25
prometheanfiretechnically correct or functionally good enough, which do you prefer?00:27
*** achanda_ has quit IRC00:27
ianwpabelanger: so can we move zuul user creation back into puppet?  i feel like devuser is the wrong thing to be doing it -- for starters it's just picking up the authorized_keys of the build user which isn't really right00:27
ianwi know we don't like puppet, but it has adding users figured out at least00:27
*** sarob has joined #openstack-infra00:28
*** sdague has joined #openstack-infra00:28
prometheanfirethat's the thing I need right?00:28
ianwbut also it overwrites if someone is actually trying to create a "devuser" (i.e., me :)00:28
*** jamesmcarthur has quit IRC00:28
*** stevemar has joined #openstack-infra00:29
*** rhochmuth has joined #openstack-infra00:31
*** lvdongbing has joined #openstack-infra00:31
*** sarob has quit IRC00:32
*** baoli has joined #openstack-infra00:33
*** jamesmcarthur has joined #openstack-infra00:35
*** roxanaghe has quit IRC00:35
*** nwkarsten has joined #openstack-infra00:36
*** baoli has quit IRC00:36
openstackgerritIan Wienand proposed openstack-infra/project-config: Don't use tmpfs for DIB functional tests
*** xarses has joined #openstack-infra00:41
*** baoli has joined #openstack-infra00:41
ianwprometheanfire: ^ should help ... in theory00:41
*** jamesmcarthur has quit IRC00:42
*** baoli has quit IRC00:42
*** Jeffrey4l has joined #openstack-infra00:44
*** ddieterly has quit IRC00:44
greghaynesianw: huh, dib should protect against that - its supposed to fallback to non tmpfs if you dont have the necessary ram00:47
*** ddieterly has joined #openstack-infra00:47
ianwgreghaynes: but we do have enough ram ... it just defaults to a 2gb space00:48
*** ddieterly has quit IRC00:48
*** thorst has joined #openstack-infra00:48
*** gongysh has quit IRC00:48
*** mkoderer___ has quit IRC00:48
*** stevemar has quit IRC00:49
ianwi guess we could bump it up ... not sure either way is better or worse00:50
*** mkoderer__ has joined #openstack-infra00:50
*** stevemar has joined #openstack-infra00:50
*** sree has joined #openstack-infra00:50
greghaynesianw: or have dib check the tmpfs size in addition to ram00:50
Shrewsmordred: i'm ok with you cutting a new shade release00:52
*** fbo has quit IRC00:52
*** baoli has joined #openstack-infra00:54
openstackgerritJeff Grafton proposed openstack-infra/jenkins-job-builder: Add addTestResults to GitHub pull request builder trigger
pabelangerfungi: we haven't done the process for our long lived workers yet, so we actually need to add code for that.  However, ianw is raising some concerns about how we create zuul in DIB.  Let me think on the process tonight, and propose something to address both00:54
*** anthonyper has quit IRC00:54
*** mhu has quit IRC00:54
prometheanfireianw: ya, it'll still take forever, but that may be ok00:55
*** raunak has quit IRC00:55
pabelangerianw: we could, but I am surprised devuser is failing for you. I guess the element is mean to be included only once00:55
openstackgerritJeff Grafton proposed openstack-infra/jenkins-job-builder: Add addTestResults to GitHub pull request builder trigger
*** thorst has quit IRC00:56
ianwpabelanger: yeah, it wasn't really supposed to be a generic useradd mechanism00:56
*** rfolco has quit IRC00:56
ianwpabelanger: i mean there seems to be a lot of logic in things like jenkins::slave00:56
pabelangerianw: okay, I'll revisit the process in the morning00:56
pabelangermaybe see about using ansible inside DIB :D00:57
*** anthonyper has joined #openstack-infra00:59
*** stevemar has quit IRC01:00
*** stevemar has joined #openstack-infra01:00
*** nwkarsten has quit IRC01:02
*** baoli_ has joined #openstack-infra01:02
ianwpabelanger: ahh ... node ansible_host= ansible_user=jenkins01:02
ianwi wondered how it was working01:03
*** fbo has joined #openstack-infra01:03
*** baoli has quit IRC01:05
*** stevemar has quit IRC01:05
greghaynesI have thought about making a thing to include ansible in dib elements, seems like it would be doable and useful01:08
*** ddieterly has joined #openstack-infra01:09
ianwgreghaynes: i think you could actually rewrite dib in ansible01:09
* ianw goes and hides01:10
greghaynesI mean, I wouldnt object to that. I worry it might not end up being easier to grok01:10
greghaynesansible seems to get hard to parse when the logic gets at all complicated01:11
*** zhurong has joined #openstack-infra01:11
*** mhu has joined #openstack-infra01:12
*** nwkarsten has joined #openstack-infra01:12
*** jamesmcarthur has joined #openstack-infra01:13
*** stevemar has joined #openstack-infra01:13
nibalizergreghaynes: yep01:14
*** shashank_hegde has quit IRC01:15
*** salv-orlando has joined #openstack-infra01:18
*** PollyZ has joined #openstack-infra01:19
*** amitgandhinz has joined #openstack-infra01:20
*** hongbin has joined #openstack-infra01:23
*** PollyZ_ has joined #openstack-infra01:24
*** amitgandhinz has quit IRC01:25
*** salv-orlando has quit IRC01:25
*** PollyZ has quit IRC01:26
*** rhochmuth has quit IRC01:27
*** rhochmuth has joined #openstack-infra01:27
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: TestSubCommand inherit from UpdateSubCommand.
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Don't expose YamlParser methods unnecessarily.
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Move subcommand logic into subcommand classes.
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Move 'update' yaml parsing to JenkinsJobs.execute.
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Factor XmlJobGenerator out of YamlParser.
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Disentangle YamlParser and ModuleRegistry classes
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Remove deprecated 'extended-choice' property.
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Remove YamlParser references from JJB modules.
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Be explicit about objects exported by modules.
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Cleanup various deprecation warnings.
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Remove unused builder.Builder.update_job method
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Rename Builder.delete_job to Builder.delete_jobs.
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Merge builder.Builder and builder.Jenkins
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Move CacheStorage to its own module.
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Make internal builder.Jenkins attrs internal-only.
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Move 'delete' yaml parsing to JenkinsJobs.execute.
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Make builder.Jenkins inherit from jenkins.Jenkins
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Move load_files method to YamlParser from Builder.
*** hongbin has quit IRC01:29
*** sree has quit IRC01:30
*** ddieterly has quit IRC01:31
*** sree has joined #openstack-infra01:31
*** raunak has joined #openstack-infra01:31
*** nwkarsten has quit IRC01:31
waynrjeblair: hey we briefly talked about this a few weeks ago not sure if you remember but I finally got around to moving JJB macro expansion into the YamlParser so it is no longer dependent on XML expansion, here's the change:
*** nwkarsten has joined #openstack-infra01:32
waynrs/XML expansion/XML generation/01:32
*** ams__ has quit IRC01:33
*** sree has quit IRC01:35
*** banix has joined #openstack-infra01:35
*** esberglu has joined #openstack-infra01:36
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci: [NO MERGY] Test a fake periodic job
waynrhopefully this makes JJB 2.0 desirable for Zuul work :)01:37
*** jamesmcarthur has quit IRC01:37
*** stevemar has quit IRC01:40
*** stevemar has joined #openstack-infra01:40
*** jamesmcarthur has joined #openstack-infra01:40
*** sree has joined #openstack-infra01:42
*** sree_ has joined #openstack-infra01:43
*** baoli_ has quit IRC01:43
*** sree_ is now known as Guest980201:44
*** baoli has joined #openstack-infra01:44
*** jamesmcarthur has quit IRC01:45
*** jamesmcarthur has joined #openstack-infra01:45
*** sree has quit IRC01:47
openstackgerritIan Wienand proposed openstack-infra/system-config: Add zuul user for launching in puppet
*** Guest9802 has quit IRC01:48
*** yamahata has quit IRC01:48
ianwpabelanger: ^ my suggestion -- at least describes the issues i see with the current model anyway01:48
*** nwkarsten has quit IRC01:50
*** nwkarsten has joined #openstack-infra01:51
*** sree has joined #openstack-infra01:51
openstackgerritIan Wienand proposed openstack-infra/system-config: Add zuul user for launching in puppet
*** jamesmcarthur has quit IRC01:52
*** thorst has joined #openstack-infra01:53
*** nwkarsten has quit IRC01:55
*** sree has quit IRC01:55
pabelangerianw: ack, will review in the morning01:56
*** rhochmuth1 has joined #openstack-infra01:57
*** signed8bit_Zzz is now known as signed8bit01:57
*** sree has joined #openstack-infra01:57
*** rhochmuth has quit IRC01:58
pabelangerjeblair: mordred: going to run ansible-playbook to restart zuul-launchers so we are good for the morning01:58
pabelangerjhesketh: loop you in too^01:58
pabelangerit will be running on puppetmaster.o.o under screen01:58
jheskethpabelanger: thanks for the heads up01:58
pabelangerdoing zlstatic01.o.o first01:59
*** thorst has quit IRC01:59
*** claudiub has quit IRC02:00
*** esberglu has quit IRC02:00
*** afred312 has quit IRC02:00
*** tqtran has joined #openstack-infra02:01
*** sree has quit IRC02:01
*** sree has joined #openstack-infra02:03
*** zul has quit IRC02:03
*** afred312 has joined #openstack-infra02:04
*** zul has joined #openstack-infra02:04
*** Apoorva has quit IRC02:04
*** tqtran has quit IRC02:05
*** thorst has joined #openstack-infra02:05
jeblairpabelanger: thanks, just came back to do that myself02:05
pabelangerjeblair: jhesketh: running now on puppetmaster.o.o, zl03 is the first02:06
jeblairpabelanger, ianw: we don't have to use devuser, but we should create the user in dib, even if we make a new element for it02:06
pabelangerjeblair: ianw: Ya, we'll need to figure out the plan for long lived servers too, since we don't use DIB for them... yet02:07
*** sree has quit IRC02:07
*** bhavik has quit IRC02:08
jeblairi have to run now; more fun tomorrow02:09
*** sree has joined #openstack-infra02:09
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Tidy up zuul-env-reqs
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Tidy up absent rule
*** thorst has quit IRC02:11
*** stevemar has quit IRC02:14
*** sree has quit IRC02:14
*** tlian has quit IRC02:14
openstackgerritMerged openstack/diskimage-builder: Updated from global requirements
*** bhavik has joined #openstack-infra02:17
*** sree has joined #openstack-infra02:17
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Move macro expansion into YamlParser.
openstackgerritgreghaynes proposed openstack/diskimage-builder: Fix copyright in docs
*** hongbin has joined #openstack-infra02:20
*** jamesmcarthur has joined #openstack-infra02:21
*** amitgandhinz has joined #openstack-infra02:21
*** sree has quit IRC02:22
*** zul has quit IRC02:23
openstackgerritMerged openstack-infra/shade: Treat DELETE_COMPLETE stacks as NotFound
*** rajinir has quit IRC02:24
*** kushal has quit IRC02:26
*** amitgandhinz has quit IRC02:26
*** sree has joined #openstack-infra02:26
*** greghaynes has quit IRC02:27
*** nwkarsten has joined #openstack-infra02:28
*** raunak has quit IRC02:30
*** sdague has quit IRC02:30
*** sree has quit IRC02:31
*** sree has joined #openstack-infra02:33
*** stevemar has joined #openstack-infra02:34
*** aysyd has quit IRC02:35
*** zz_dimtruck is now known as dimtruck02:37
*** sree_ has joined #openstack-infra02:37
*** sree_ is now known as Guest9196002:38
*** gongysh has joined #openstack-infra02:38
*** sree has quit IRC02:38
*** greghaynes has joined #openstack-infra02:38
*** stevemar has quit IRC02:39
*** inc0 has joined #openstack-infra02:39
*** sree has joined #openstack-infra02:41
*** Guest91960 has quit IRC02:41
*** rhochmuth1 has quit IRC02:43
*** sree has quit IRC02:46
*** inc0 has quit IRC02:46
*** jamesmcarthur has quit IRC02:47
*** sree has joined #openstack-infra02:49
*** salv-orlando has joined #openstack-infra02:49
*** sree has quit IRC02:53
*** salv-orlando has quit IRC02:54
*** yanyanhu has joined #openstack-infra02:54
*** zhurong has quit IRC02:55
*** stevemar has joined #openstack-infra02:57
*** nwkarsten has quit IRC02:57
*** yamamoto_ has joined #openstack-infra03:02
*** achanda has joined #openstack-infra03:03
*** zhurong has joined #openstack-infra03:03
*** raunak has joined #openstack-infra03:04
ianwjeblair: tbh i'm really not sure reinventing user adding in dib is a great idea, it seems like something that puppet does just fine.03:05
*** nwkarsten has joined #openstack-infra03:07
*** sree has joined #openstack-infra03:07
*** achanda has quit IRC03:08
*** hongbin has quit IRC03:09
*** lvdongbing has quit IRC03:09
*** haaj has quit IRC03:10
openstackgerritIan Wienand proposed openstack-infra/system-config: Add zuul user for launching in puppet
*** sree has quit IRC03:11
*** sree has joined #openstack-infra03:14
*** jamesmcarthur has joined #openstack-infra03:14
*** raunak has quit IRC03:14
*** bhavik has quit IRC03:15
*** bhavik has joined #openstack-infra03:15
*** thorst has joined #openstack-infra03:16
*** ramishra has joined #openstack-infra03:17
*** sree has quit IRC03:18
*** lvdongbing has joined #openstack-infra03:18
*** csomerville has quit IRC03:18
*** baoli has quit IRC03:19
*** jamesmcarthur has quit IRC03:19
*** raunak has joined #openstack-infra03:19
*** stevemar has quit IRC03:20
*** rhochmuth has joined #openstack-infra03:20
*** thorst has quit IRC03:21
*** amitgandhinz has joined #openstack-infra03:22
*** baoli has joined #openstack-infra03:23
*** nwkarsten has quit IRC03:25
*** nwkarsten has joined #openstack-infra03:25
*** amitgandhinz has quit IRC03:27
*** ramishra has quit IRC03:27
*** stevemar has joined #openstack-infra03:28
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Lower the log level in tests
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add support for sqlalchemy reporter
*** stevemar has quit IRC03:32
*** ramishra has joined #openstack-infra03:32
*** sree has joined #openstack-infra03:33
*** garyk1 has joined #openstack-infra03:34
*** stevemar has joined #openstack-infra03:35
*** stevemar has quit IRC03:35
*** stevemar has joined #openstack-infra03:36
ianwgate-infra-puppet-apply-ubuntu-xenial: queued <- do we not have a lot of xenial nodes?03:37
*** sree has quit IRC03:37
*** rhochmuth has quit IRC03:39
*** gouthamr_ has quit IRC03:39
*** shashank_hegde has joined #openstack-infra03:41
jheskethianw: not many... 9 total, 3 in hold, 5 used, 1 ready03:41
*** raunak has quit IRC03:41
*** raunak has joined #openstack-infra03:43
*** signed8bit has quit IRC03:43
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Make ml2 job non-voting
*** lvdongbing has quit IRC03:47
*** shashank_hegde has quit IRC03:49
*** nwkarsten has quit IRC03:49
*** nwkarsten has joined #openstack-infra03:50
*** links has joined #openstack-infra03:51
*** Nikolay_St has quit IRC03:52
*** ramishra has quit IRC03:53
*** ramishra has joined #openstack-infra03:54
*** winggundamth has quit IRC03:54
*** yamamoto_ has quit IRC03:54
*** nwkarsten has quit IRC03:54
*** sree has joined #openstack-infra03:57
*** nwkarsten has joined #openstack-infra03:59
*** burgerk has joined #openstack-infra04:00
*** salv-orlando has joined #openstack-infra04:00
*** sree has quit IRC04:02
*** baoli has quit IRC04:10
*** haaj has joined #openstack-infra04:11
*** cody-somerville has joined #openstack-infra04:13
*** salv-orlando has quit IRC04:14
*** sree has joined #openstack-infra04:17
*** achanda has joined #openstack-infra04:18
*** thorst has joined #openstack-infra04:19
*** shashank_hegde has joined #openstack-infra04:20
*** javeriak has joined #openstack-infra04:20
*** dimtruck is now known as zz_dimtruck04:21
*** amitgandhinz has joined #openstack-infra04:23
*** thorst has quit IRC04:25
*** amotoki has joined #openstack-infra04:25
*** cody-somerville has quit IRC04:27
*** rossella__ has joined #openstack-infra04:27
*** esikachev has joined #openstack-infra04:27
*** amitgandhinz has quit IRC04:27
*** Sukhdev has joined #openstack-infra04:28
*** PollyZ_ has quit IRC04:28
*** maishsk has quit IRC04:29
*** rossella_ has quit IRC04:30
*** raunak has quit IRC04:31
*** esikachev has quit IRC04:32
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Move the command processing into the launcher cmd
*** javeriak has quit IRC04:34
*** raunak has joined #openstack-infra04:35
*** psachin has joined #openstack-infra04:38
*** winggundamth has joined #openstack-infra04:38
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Move the command processing into the launcher cmd
waynrhmm, it the "parallel" module in JJB seems to be a misnomer04:42
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Move the command processing into the launcher cmd
*** yamamoto_ has joined #openstack-infra04:45
*** Liuqing has joined #openstack-infra04:46
*** javeriak has joined #openstack-infra04:46
*** burgerk has quit IRC04:47
*** javeriak has quit IRC04:50
*** salv-orlando has joined #openstack-infra04:50
*** banix has quit IRC04:52
*** shahid__ has joined #openstack-infra04:52
*** ramishra has quit IRC04:52
*** Sukhdev has quit IRC04:55
*** burgerk has joined #openstack-infra04:55
*** jaosorior has joined #openstack-infra04:55
*** ramishra has joined #openstack-infra04:58
*** salv-orl_ has joined #openstack-infra04:59
*** burgerk has quit IRC05:00
waynrhmm, replacing threading.Thread with multiprocessing.Process causes unit tests to hang....05:02
*** salv-orlando has quit IRC05:02
waynri guess i'll just have to gut the parallel module to make it actually parallel...the fun never ends!05:02
*** yamahata has joined #openstack-infra05:08
*** sree_ has joined #openstack-infra05:08
*** sree has quit IRC05:09
*** gongysh has quit IRC05:09
*** sree_ is now known as Guest7781405:09
*** Nikolay_St has joined #openstack-infra05:12
*** ramishra has quit IRC05:23
*** thorst has joined #openstack-infra05:23
*** amitgandhinz has joined #openstack-infra05:24
*** abregman has joined #openstack-infra05:25
*** amitgandhinz has quit IRC05:28
*** thorst has quit IRC05:31
openstackgerritKhai Do proposed openstack-infra/system-config: Change commit message length validation to 72 chars
*** coolsvap has joined #openstack-infra05:32
*** Liuqing has quit IRC05:33
*** shashank_hegde has quit IRC05:34
waynri wonder if i am just kidding myself about JJB 2.0 ever getting reviewed05:34
*** shardy has joined #openstack-infra05:36
*** javeriak has joined #openstack-infra05:37
*** ramishra has joined #openstack-infra05:37
*** shashank_hegde has joined #openstack-infra05:38
*** kaisers1 has quit IRC05:38
*** shardy has quit IRC05:38
*** kaisers has joined #openstack-infra05:42
*** binbincong has quit IRC05:43
*** shahid__ has quit IRC05:43
*** Guest77814 has quit IRC05:43
*** sree has joined #openstack-infra05:44
*** roxanaghe has joined #openstack-infra05:44
*** ramishra has quit IRC05:44
*** nwkarsten has quit IRC05:45
*** sshnaidm_ has joined #openstack-infra05:45
*** nwkarsten has joined #openstack-infra05:45
*** roxanaghe has quit IRC05:45
openstackgerritChristian Fetzer proposed openstack-infra/jenkins-job-builder: Update Gitlab to >=1.2.0
*** ramishra has joined #openstack-infra05:49
*** nwkarsten has quit IRC05:50
*** lvdongbing has joined #openstack-infra05:50
*** camunoz has quit IRC05:51
*** rhochmuth has joined #openstack-infra05:51
*** _nadya_ has joined #openstack-infra05:54
*** raunak has quit IRC05:55
*** yolanda has joined #openstack-infra05:56
*** binbincong has joined #openstack-infra06:00
openstackgerritSpencer Krum proposed openstack-infra/puppet-openstack_infra_spec_helper: Pin puppet-lint-absolute_classname-check to 0.1.3
*** tqtran has joined #openstack-infra06:02
openstackgerritIan Wienand proposed openstack/diskimage-builder: Handle locales install on Fedora 24
*** ramishra has quit IRC06:06
*** tqtran has quit IRC06:06
*** pgadiya has joined #openstack-infra06:08
*** rossella__ has quit IRC06:11
*** ramishra has joined #openstack-infra06:11
*** aeng has quit IRC06:12
*** dingyichen has quit IRC06:12
*** claudiub has joined #openstack-infra06:13
*** rcernin has joined #openstack-infra06:15
openstackgerritGal Sagie proposed openstack-infra/project-config: Add networking-l2gw fullstack job
*** linuxmaniac_ has joined #openstack-infra06:17
*** linuxmaniac has quit IRC06:17
openstackgerritMerged openstack-infra/project-config: python-saharaclient is required also in PROJECTS
*** ramishra has quit IRC06:19
*** ramishra has joined #openstack-infra06:21
openstackgerritMerged openstack-infra/project-config: Don't use tmpfs for DIB functional tests
openstackgerritMerged openstack-infra/project-config: Request to add new project Masakari
openstackgerritMerged openstack-infra/system-config: Use root to restart our zuul-launcher service
openstackgerritMerged openstack-infra/project-config: Check git tag matches xstatic version
*** hdaniel has joined #openstack-infra06:24
*** amitgandhinz has joined #openstack-infra06:24
*** maishsk has joined #openstack-infra06:27
*** Qiming has quit IRC06:28
*** andreas_s has joined #openstack-infra06:28
openstackgerritMerged openstack-infra/project-config: Remove tests on magnum liberty.
*** thorst has joined #openstack-infra06:28
*** amitgandhinz has quit IRC06:29
*** yolanda has quit IRC06:30
*** prabu has joined #openstack-infra06:30
*** maishsk has quit IRC06:31
*** isaacb has joined #openstack-infra06:33
*** sdake has joined #openstack-infra06:34
openstackgerritMerged openstack-infra/project-config: Add gate-ironic-agent-tox-func to IPA gate
*** Qiming has joined #openstack-infra06:35
*** thorst has quit IRC06:36
*** _nadya_ has quit IRC06:36
*** javeriak has quit IRC06:43
*** sree has quit IRC06:44
*** changzhi has joined #openstack-infra06:44
*** sree has joined #openstack-infra06:44
*** Qiming has quit IRC06:47
*** zeih has joined #openstack-infra06:47
*** zeih has quit IRC06:48
*** zeih has joined #openstack-infra06:48
*** jamielennox is now known as jamielennox|away06:48
*** Hal has joined #openstack-infra06:51
*** Hal is now known as Guest2541706:51
*** javeriak has joined #openstack-infra06:52
*** TaylorHuang has joined #openstack-infra06:52
*** gongysh has joined #openstack-infra06:52
*** Qiming has joined #openstack-infra06:54
*** TaylorH has quit IRC06:55
*** achanda has quit IRC06:56
*** esikachev has joined #openstack-infra06:59
openstackgerritMerged openstack-infra/project-config: Retiring monitoring-for-openstack step 2
*** achanda has joined #openstack-infra07:01
*** shardy has joined #openstack-infra07:02
*** rcernin has quit IRC07:02
*** achanda has quit IRC07:06
*** samP has joined #openstack-infra07:07
*** isaacb has quit IRC07:08
*** ajmiller has quit IRC07:09
*** jpich has joined #openstack-infra07:11
*** tesseract- has joined #openstack-infra07:13
*** esikachev has quit IRC07:13
*** tesseract- has quit IRC07:14
*** tesseract- has joined #openstack-infra07:14
*** hdaniel has quit IRC07:17
*** rcernin has joined #openstack-infra07:17
*** salv-orl_ has quit IRC07:18
*** changzhi has quit IRC07:18
*** woodster_ has quit IRC07:18
*** kzaitsev_mb has joined #openstack-infra07:20
*** Nikolay_St has quit IRC07:21
*** sdake_ has joined #openstack-infra07:21
*** rhochmuth has quit IRC07:22
*** zhurong has quit IRC07:22
*** furlongm_ has joined #openstack-infra07:22
*** furlongm has quit IRC07:22
*** zhurong has joined #openstack-infra07:23
*** sdake has quit IRC07:24
*** e0ne has joined #openstack-infra07:24
*** eezhova has joined #openstack-infra07:25
*** yaume has joined #openstack-infra07:25
*** amitgandhinz has joined #openstack-infra07:25
*** Na3iL has joined #openstack-infra07:26
*** jlanoux has joined #openstack-infra07:26
*** furlongm has joined #openstack-infra07:26
*** furlongm_ has quit IRC07:27
*** sree has quit IRC07:27
*** dizquierdo has joined #openstack-infra07:28
*** watanabe_isao has joined #openstack-infra07:29
*** Nikolay_St has joined #openstack-infra07:29
*** jlanoux has quit IRC07:29
*** amitgandhinz has quit IRC07:29
*** abregman has quit IRC07:30
*** abregman has joined #openstack-infra07:31
*** eezhova has quit IRC07:32
*** thorst has joined #openstack-infra07:34
*** jlanoux has joined #openstack-infra07:34
*** bhavik has quit IRC07:34
*** furlongm has quit IRC07:35
*** furlongm has joined #openstack-infra07:35
*** arxcruz has joined #openstack-infra07:36
*** thorst has quit IRC07:41
*** furlongm has quit IRC07:42
*** hashar has joined #openstack-infra07:42
*** watanabe_isao has quit IRC07:43
*** amoralej|off is now known as amoralej07:43
*** samP has quit IRC07:46
*** Na3iL has quit IRC07:46
*** matrohon has joined #openstack-infra07:48
*** kzaitsev_mb has quit IRC07:49
*** nmagnezi has joined #openstack-infra07:52
*** ifarkas has joined #openstack-infra07:53
*** bunting has quit IRC07:53
*** bunting has joined #openstack-infra07:54
*** sflanigan has quit IRC07:55
*** gus has quit IRC07:57
*** gus has joined #openstack-infra07:57
*** asselin has quit IRC07:58
*** isaacb has joined #openstack-infra07:59
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:01
*** shahid__ has joined #openstack-infra08:02
*** jtomasek_ has joined #openstack-infra08:02
*** abregman has quit IRC08:03
*** zhurong has quit IRC08:03
*** yolanda has joined #openstack-infra08:04
*** salv-orlando has joined #openstack-infra08:05
*** markvoel_ has quit IRC08:05
*** dfflanders has quit IRC08:08
*** zhurong has joined #openstack-infra08:10
*** sree has joined #openstack-infra08:10
*** Na3iL has joined #openstack-infra08:10
*** isaacb has quit IRC08:11
*** e0ne has quit IRC08:11
*** _nadya_ has joined #openstack-infra08:15
*** ams__ has joined #openstack-infra08:16
*** _nadya_ has quit IRC08:16
*** _nadya_ has joined #openstack-infra08:16
openstackgerritSpyros Trigazis proposed openstack-infra/project-config: [magnum] Add install-guide job
*** e0ne has joined #openstack-infra08:18
*** abregman has joined #openstack-infra08:18
*** jtomasek_ has quit IRC08:20
*** dtardivel has joined #openstack-infra08:23
*** amitgandhinz has joined #openstack-infra08:26
*** acabot has quit IRC08:26
*** sree has quit IRC08:27
*** gongysh has quit IRC08:29
*** sree has joined #openstack-infra08:29
*** lucas-afk is now known as lucasagomes08:29
*** acabot has joined #openstack-infra08:29
*** shashank_hegde has quit IRC08:29
*** zhurong_ has joined #openstack-infra08:30
*** hosanai has joined #openstack-infra08:30
*** zhurong has quit IRC08:30
*** amitgandhinz has quit IRC08:31
*** Qiming has quit IRC08:31
*** daemontool has joined #openstack-infra08:31
*** sree has quit IRC08:34
*** jcooley has joined #openstack-infra08:35
*** sambetts|afk is now known as sambetts08:36
*** jcooley has left #openstack-infra08:36
openstackgerritSpyros Trigazis proposed openstack-infra/project-config: [magnum] Add install-guide job
*** thorst has joined #openstack-infra08:39
*** derekh has joined #openstack-infra08:44
*** HeOS has quit IRC08:44
*** Qiming has joined #openstack-infra08:44
*** thorst has quit IRC08:46
openstackgerritThomas Herve proposed openstack-infra/project-config: Don't run tempest tests in zaqar gate
*** Jeffrey4l has quit IRC08:47
*** eezhova has joined #openstack-infra08:48
*** salv-orlando has quit IRC08:49
*** rcernin has quit IRC08:50
*** salv-orlando has joined #openstack-infra08:50
*** amotoki has quit IRC08:53
therveWhat's the procedure to exclude a job from master and forward?08:54
openstackgerritMerged openstack-infra/storyboard: Add TAGS_ADDED and TAGS_DELETED to event_types.ALL
therveI see lots of example for not running on old stable branches, but not the other way around08:54
*** salv-orlando has quit IRC08:55
*** dtantsur|afk is now known as dtantsur08:59
*** mhickey has joined #openstack-infra08:59
*** Jeffrey4l has joined #openstack-infra09:00
*** yolanda has quit IRC09:01
*** rcernin has joined #openstack-infra09:02
*** achanda has joined #openstack-infra09:03
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: UPDATE REPO FOR DELOREAN_REPO_URL
*** tqtran has joined #openstack-infra09:03
*** ramishra has quit IRC09:04
*** ramishra has joined #openstack-infra09:04
*** amotoki has joined #openstack-infra09:05
*** markvoelker has joined #openstack-infra09:05
*** infra-red has joined #openstack-infra09:06
*** achanda has quit IRC09:08
*** tqtran has quit IRC09:08
*** markusry has joined #openstack-infra09:08
*** dmk0202 has joined #openstack-infra09:09
*** markvoelker has quit IRC09:10
*** jaosorior has quit IRC09:11
*** Qiming has quit IRC09:11
*** jaosorior has joined #openstack-infra09:11
openstackgerritThomas Herve proposed openstack-infra/project-config: Remove Heat lbaasv1 from master
*** arif-ali has quit IRC09:15
*** binbincong has quit IRC09:16
*** binbincong has joined #openstack-infra09:16
*** electrofelix has joined #openstack-infra09:20
*** Qiming has joined #openstack-infra09:20
*** maishsk has joined #openstack-infra09:20
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: UPDATE REPO FOR DELOREAN_REPO_URL
*** zul has joined #openstack-infra09:23
*** arif-ali has joined #openstack-infra09:24
*** zhenguo_ has joined #openstack-infra09:25
*** amitgandhinz has joined #openstack-infra09:27
openstackgerritYuval Brik proposed openstack-infra/project-config: Use Smaug configuration from Smaug's repository
*** Jeffrey4l has quit IRC09:28
*** arif-ali has quit IRC09:29
*** amotoki has quit IRC09:30
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: UPDATE REPO FOR DELOREAN_REPO_URL
*** amotoki has joined #openstack-infra09:32
*** amitgandhinz has quit IRC09:32
*** ociuhandu has joined #openstack-infra09:32
*** yamahata has quit IRC09:33
*** amotoki has quit IRC09:33
*** infra-red has quit IRC09:33
*** abregman is now known as abregman|afk09:34
*** kzaitsev_mb has joined #openstack-infra09:37
*** javeriak has quit IRC09:38
*** amotoki has joined #openstack-infra09:39
*** _milan_ has quit IRC09:39
*** acoles_ is now known as acoles09:41
*** Jeffrey4l has joined #openstack-infra09:42
*** javeriak has joined #openstack-infra09:42
*** salv-orlando has joined #openstack-infra09:42
*** isaacb has joined #openstack-infra09:42
*** sdake_ has quit IRC09:42
*** jamielennox|away is now known as jamielennox09:43
*** thorst has joined #openstack-infra09:44
*** esikachev has joined #openstack-infra09:44
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: UPDATE REPO FOR DELOREAN_REPO_URL
*** kzaitsev_mb has quit IRC09:48
*** kzaitsev_mb has joined #openstack-infra09:49
*** amotoki has quit IRC09:50
andreykurilinhi everyone! where I can find code of test-results publisher?
*** thorst has quit IRC09:51
*** pcaruana has joined #openstack-infra09:51
*** ociuhandu has quit IRC09:55
*** tosky has joined #openstack-infra09:55
*** amotoki has joined #openstack-infra09:58
AJaegertherve: just pin it to the current branches like "branch: ^(stable/(kilo|liberty)).*$"09:59
therveAJaeger, Yeah I think I found the solution, thanks10:00
*** daemontool has quit IRC10:00
*** shardy has quit IRC10:00
*** abregman|afk is now known as abregman10:01
*** sdake has joined #openstack-infra10:01
*** coolsvap has quit IRC10:02
*** lvdongbing has quit IRC10:02
AJaegertherve: should it apply here as well?
*** coolsvap has joined #openstack-infra10:03
AJaegertherve: the rule for ^gate-heat-dsvm-functional-convg-mysql-.*$ overrides your v1 condition10:03
therveAh, yeah...10:04
AJaegertherve: the codnition is correct otherwise. It's just the override...10:05
therveThat's annoying :)10:05
therveI don't get why it can't do multiple matches, anyway10:05
*** markvoelker has joined #openstack-infra10:07
therveUrg that hurts my brain10:07
*** eezhova has quit IRC10:09
openstackgerritThomas Herve proposed openstack-infra/project-config: Remove Heat lbaasv1 from master
openstackgerritThierry Carrez proposed openstack-infra/storyboard-webclient: Fix alignment of long-running simple events
*** bhavik has joined #openstack-infra10:11
*** markvoelker has quit IRC10:11
*** yaume has quit IRC10:12
*** ociuhandu has joined #openstack-infra10:13
openstackgerritAndrey Kurilin proposed openstack-infra/project-config: Add pytest_results.html to test-results publisher
*** zhurong_ has quit IRC10:14
*** eezhova has joined #openstack-infra10:15
*** bhavik has quit IRC10:15
openstackgerritJulien Danjou proposed openstack-infra/project-config: panko: enable functional testing
*** dtantsur is now known as dtantsur|bbl10:17
AJaegertherve: you have a pasto in there, otherwise should be fine.10:17
therveAJaeger, Thanks a ton!10:19
openstackgerritThomas Herve proposed openstack-infra/project-config: Remove Heat lbaasv1 from master
*** _degorenko|afk is now known as degorenko10:20
openstackgerritHisashi Osanai proposed openstack-infra/project-config: Fix Monasca Analytics python27 job
*** zul has quit IRC10:23
*** shardy has joined #openstack-infra10:24
*** HeOS has joined #openstack-infra10:25
*** amitgandhinz has joined #openstack-infra10:27
*** sdake_ has joined #openstack-infra10:28
*** dmk0202 has quit IRC10:29
*** sdake has quit IRC10:32
*** amitgandhinz has quit IRC10:32
*** zul has joined #openstack-infra10:35
*** eezhova has quit IRC10:38
*** zul has quit IRC10:38
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Add experimental functional job
*** ramishra has quit IRC10:41
*** hrubi has quit IRC10:41
*** hrubi has joined #openstack-infra10:41
*** ramishra has joined #openstack-infra10:41
*** coolsvap has quit IRC10:42
*** coolsvap has joined #openstack-infra10:44
*** jpeeler has quit IRC10:44
*** thorst has joined #openstack-infra10:49
*** sorantis has joined #openstack-infra10:50
*** zul has joined #openstack-infra10:52
wznoinskhi infra, any issues with ci-watch or it's just me?10:52
*** derekh has quit IRC10:53
wznoinskdetailes: gives no table with changes and results10:55
*** rhochmuth has joined #openstack-infra10:55
*** thorst has quit IRC10:56
*** jistr is now known as jistr|mtg10:56
*** salv-orl_ has joined #openstack-infra10:58
*** coolsvap has quit IRC10:58
*** rhochmuth has quit IRC11:00
*** coolsvap has joined #openstack-infra11:00
*** salv-orlando has quit IRC11:01
*** YorikSar has quit IRC11:02
*** _nadya_ has quit IRC11:02
*** dtantsur|bbl is now known as dtantsur11:03
*** YorikSar has joined #openstack-infra11:04
*** achanda has joined #openstack-infra11:05
*** ddieterly has joined #openstack-infra11:05
*** sorantis has quit IRC11:05
*** markvoelker has joined #openstack-infra11:07
*** rhochmuth has joined #openstack-infra11:09
*** rhochmuth has quit IRC11:09
*** achanda has quit IRC11:09
*** yanyanhu has quit IRC11:10
*** sshnaidm_ is now known as sshnaidm11:11
*** markvoelker has quit IRC11:13
*** eezhova has joined #openstack-infra11:15
*** zul has quit IRC11:16
*** amrith has quit IRC11:16
*** zul has joined #openstack-infra11:17
*** amrith has joined #openstack-infra11:17
*** aviau has quit IRC11:17
*** aviau has joined #openstack-infra11:17
*** lucasagomes is now known as lucas-hungry11:19
*** amoralej is now known as amoralej|lunch11:19
*** jamesmcarthur has joined #openstack-infra11:19
*** sdake_ has quit IRC11:21
*** sdague has joined #openstack-infra11:22
*** lvdongbing has joined #openstack-infra11:23
*** sarob has joined #openstack-infra11:23
*** jamesmcarthur has quit IRC11:23
*** kzaitsev_mb has quit IRC11:25
*** ekarlso has quit IRC11:26
openstackgerritJulien Danjou proposed openstack-infra/project-config: panko: enable functional testing
*** daemontool has joined #openstack-infra11:28
*** amitgandhinz has joined #openstack-infra11:28
*** sarob has quit IRC11:28
*** snarwade is now known as snarwade|busy11:29
*** ddieterly has quit IRC11:31
*** prabu has quit IRC11:31
*** dizquierdo has quit IRC11:32
*** amitgandhinz has quit IRC11:33
*** dmk0202 has joined #openstack-infra11:33
*** ddieterly has joined #openstack-infra11:35
*** YorikSar has quit IRC11:36
*** YorikSar has joined #openstack-infra11:37
*** ddieterly has quit IRC11:38
*** ldnunes has joined #openstack-infra11:39
*** thorst has joined #openstack-infra11:41
*** _nadya_ has joined #openstack-infra11:42
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Add experimental functional job
*** kzaitsev_mb has joined #openstack-infra11:43
*** gouthamr has joined #openstack-infra11:47
*** _nadya_ has quit IRC11:47
bkeroGood morning everyone11:48
*** zhurong has joined #openstack-infra11:48
*** rhallisey has joined #openstack-infra11:49
*** ekarlso has joined #openstack-infra11:50
*** maishsk has quit IRC11:51
*** _nadya_ has joined #openstack-infra11:51
*** ddieterly has joined #openstack-infra11:51
*** armax has joined #openstack-infra11:53
*** rfolco has joined #openstack-infra11:53
*** coolsvap_ has joined #openstack-infra11:53
*** salv-orl_ has quit IRC11:53
*** coolsvap has quit IRC11:53
*** weshay has joined #openstack-infra11:54
*** sree has joined #openstack-infra11:54
*** daemontool_ has joined #openstack-infra11:54
*** daemontool has quit IRC11:55
*** eharney has joined #openstack-infra11:55
*** tpsilva has joined #openstack-infra11:56
*** banix has joined #openstack-infra11:56
*** coolsvap has joined #openstack-infra11:57
*** rhallisey has quit IRC11:57
*** rhallisey has joined #openstack-infra11:58
*** coolsvap_ has quit IRC11:58
*** ddieterly has quit IRC11:59
*** markvoelker has joined #openstack-infra11:59
*** dprince has joined #openstack-infra12:01
*** Srinu has joined #openstack-infra12:01
Srinufungi: hai12:01
*** lucas-hungry is now known as lucas-afk12:04
*** achanda has joined #openstack-infra12:06
SrinuHi. when i am running jenkins job in Jenkins CI while the test is in progress the slave went offline so my test report failure12:07
*** snarwade|busy is now known as snarwade12:07
*** coolsvap has quit IRC12:08
*** eezhova has quit IRC12:09
*** achanda has quit IRC12:10
*** maishsk has joined #openstack-infra12:12
*** kgiusti has joined #openstack-infra12:12
*** aysyd has joined #openstack-infra12:13
*** zeih has quit IRC12:13
*** sshnaidm has quit IRC12:14
*** eezhova has joined #openstack-infra12:14
AJaegersdague: could you review some time this change for project-config? I fear it removes too many lines...
*** zeih has joined #openstack-infra12:17
sdagueAJaeger: well, if they think it works for them12:18
sdaguethe real trick of it is what's in their devstackgaterc I think12:18
*** sshnaidm has joined #openstack-infra12:18
*** ramishra has quit IRC12:19
*** lvdongbing has quit IRC12:21
*** jamesmcarthur has joined #openstack-infra12:21
*** jcoufal has joined #openstack-infra12:21
*** pradk has joined #openstack-infra12:21
*** sarob has joined #openstack-infra12:22
*** binbincong has quit IRC12:22
AJaegerI would have expected the enable_plugin stay in project-config12:22
*** amotoki has quit IRC12:23
AJaegersdague: seems they moved everything over:
openstackgerritAndrey Kurilin proposed openstack-infra/project-config: Add pytest_results.html to test-results publisher
*** ddieterly has joined #openstack-infra12:24
*** amoralej|lunch is now known as amoralej12:25
EmilienMjeblair, pabelanger: hey, could you log into nodes that fail (re: puppet ci, async issue)12:26
*** sarob has quit IRC12:26
*** jcoufal_ has joined #openstack-infra12:26
*** amitgandhinz has joined #openstack-infra12:29
*** jcoufal has quit IRC12:29
openstackgerritEvgeny Sikachev proposed openstack-infra/project-config: Added pre_hook for sahara cli tests
*** erlon has joined #openstack-infra12:30
*** ramishra has joined #openstack-infra12:30
*** thiagop-afk has joined #openstack-infra12:31
*** thiagop-afk is now known as thiagop12:32
*** ddieterly has quit IRC12:32
*** lvdongbing has joined #openstack-infra12:33
SrinuAJaeger: Hi. when i am running jenkins job in Jenkins CI while the test is in progress the slave went offline so my test report failure12:33
*** armax has quit IRC12:33
*** amitgandhinz has quit IRC12:34
*** trown|outtypewww is now known as trown12:34
*** kushal has joined #openstack-infra12:38
*** matt-borland has joined #openstack-infra12:39
*** jamesmcarthur has quit IRC12:40
*** psilvad has joined #openstack-infra12:41
*** yamamoto_ has quit IRC12:41
*** ramishra has quit IRC12:41
*** ccamacho is now known as ccamacho|lunch12:42
*** banix has quit IRC12:44
*** kushal has quit IRC12:44
*** kushal has joined #openstack-infra12:46
*** pradk has quit IRC12:46
*** kushal has quit IRC12:47
*** kushal has joined #openstack-infra12:47
*** salv-orlando has joined #openstack-infra12:47
*** sdake has joined #openstack-infra12:48
*** tongli has joined #openstack-infra12:48
*** javeriak has quit IRC12:48
*** Ravikiran_K has joined #openstack-infra12:50
*** javeriak has joined #openstack-infra12:51
*** asselin has joined #openstack-infra12:51
*** rlandy has joined #openstack-infra12:52
*** nwkarsten has joined #openstack-infra12:53
*** zeih has quit IRC12:53
*** signed8bit has joined #openstack-infra12:54
*** garyk1 has quit IRC12:55
openstackgerritHisashi Osanai proposed openstack-infra/project-config: Fix Monasca Analytics python27 job
*** tlian has joined #openstack-infra12:55
*** denisra has quit IRC12:55
*** hferenc has quit IRC12:56
*** xyang1 has joined #openstack-infra12:56
*** edmondsw has joined #openstack-infra12:57
*** zeih has joined #openstack-infra12:57
*** hosanai has quit IRC12:57
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Use ssh-agent instead of ugly ssh-wrapper scripts
*** baoli has joined #openstack-infra13:01
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Add ability to use SSH instead of HTTP to get refs
*** shardy has quit IRC13:03
*** eharney has quit IRC13:03
*** Srinu has quit IRC13:08
*** lezbar__ has joined #openstack-infra13:08
*** mdrabe has joined #openstack-infra13:09
*** sigmavirus24_awa is now known as sigmavirus2413:09
*** ociuhandu has quit IRC13:09
*** rhochmuth has joined #openstack-infra13:11
*** sree has quit IRC13:11
*** rhochmuth has quit IRC13:11
*** lezbar has quit IRC13:12
*** zz_dimtruck is now known as dimtruck13:15
*** yaume has joined #openstack-infra13:15
*** Srinu has joined #openstack-infra13:16
mordredShrews, infra-root: releasing shade 1.10.0 in just a few unless someone screams13:16
Shrewsmordred: k13:17
SrinuI am running jenkins jobs when the build is in progress the slave went offline this is the log of the slave
Srinuanyone face this issue before13:18
mordredhey Srinu ... I'm not really awake yet13:19
*** signed8bit is now known as signed8bit_Zzz13:21
*** signed8bit_Zzz is now known as signed8bit13:21
*** psilvad has quit IRC13:21
*** signed8bit is now known as signed8bit_Zzz13:21
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: UPDATE REPO FOR DELOREAN_REPO_URL
*** akshai has joined #openstack-infra13:25
*** pgadiya has quit IRC13:26
*** signed8bit_Zzz is now known as signed8bit13:26
*** sree has joined #openstack-infra13:28
dimsAJaeger : looks like we have 2 +2's on, can you please +W it?13:28
*** coolsvap has joined #openstack-infra13:29
*** rossella__ has joined #openstack-infra13:30
*** _ari_ has joined #openstack-infra13:30
*** claudiub has quit IRC13:31
*** jistr|mtg is now known as jistr13:31
*** mriedem has joined #openstack-infra13:32
*** david-lyle_ has joined #openstack-infra13:32
*** inc0 has joined #openstack-infra13:32
dimsthanks! AJaeger13:33
*** baoli_ has joined #openstack-infra13:33
*** ddieterly has joined #openstack-infra13:33
*** psilvad has joined #openstack-infra13:33
*** ccamacho|lunch is now known as ccamacho13:33
*** eharney has joined #openstack-infra13:33
*** esikachev has quit IRC13:34
*** yolanda has joined #openstack-infra13:34
*** piet has joined #openstack-infra13:34
*** baoli has quit IRC13:35
*** achanda has joined #openstack-infra13:35
*** david-lyle has quit IRC13:36
*** hdaniel has joined #openstack-infra13:36
*** mriedem1 has joined #openstack-infra13:36
*** jpeeler has joined #openstack-infra13:37
*** esberglu has joined #openstack-infra13:37
openstackgerritMerged openstack-infra/project-config: import the release tools that need to run on secure nodes
AJaegerdims: you're welcome13:38
AJaegermordred: you're releasing shade while not fully awake? ;)13:38
* AJaeger sends some coffee to mordred13:38
*** mriedem1 has left #openstack-infra13:38
*** kushal has quit IRC13:39
*** mriedem has quit IRC13:39
mordredAJaeger: mmm. coffee13:40
*** salv-orlando has quit IRC13:40
*** javeriak has quit IRC13:41
*** salv-orlando has joined #openstack-infra13:41
*** mriedem has joined #openstack-infra13:41
*** dizquierdo has joined #openstack-infra13:41
toskyAJaeger, yolanda: thanks for the help and sorry for all the changes to Sahara jobs, we are learning on the way and we almost done hopefully13:42
pabelangerEmilienM: we need to enable more ansible debug logs, which we can enable today.  jeblair landed a patch last night to help with the debugging13:43
*** hashar has quit IRC13:43
pabelangerrolling restarts are still on going13:43
pabelangercurrently on zl0513:43
yolandahi tosky , no problem13:44
EmilienMpabelanger: ok13:44
EmilienMpabelanger: I noticed it's mostly on xenial jobs if not only13:44
EmilienMpabelanger: did you figure the same?13:44
*** dimtruck is now known as zz_dimtruck13:44
pabelangerEmilienM: yes, we picked that node yesterday too.13:44
EmilienMpabelanger: ok13:44
EmilienMpabelanger: did you see this thing on centos7 jobs?13:45
bkeroHave we succeeded in keeping a failing node around to debug?13:45
mordredEmilienM: it's been a hard enough problem that we've now added a feature to zuul and a feature to nodepool to allow us to debug it13:45
mordredbkero: we have!13:45
mordredbkero: jeblair added a feature to tell nodepool not to delete X nodes after job Y fails13:45
pabelangerEmilienM: I only seen xenial yesterday, but I didn't look much into centos-713:45
bkerobut I'm guessing no root cause analysis yet13:45
*** rbrndt has joined #openstack-infra13:45
bkeromordred: that sounds like a super useful feature for debugging failing jobs13:46
*** achanda has quit IRC13:46
mordredwe got a node yesterday, but after digging in needed the debug logs to be on13:46
mordredso jeblair then added a feature to zuul to allow us to toggle debug logs without having to restart zuul13:46
pabelangerbkero: clarkb creating this PR, but I don't believe we have tested it yet13:46
EmilienMmordred: ack, thx13:47
EmilienMpabelanger, mordred: let me 5 min to dig all our CI jobs and see if centos7 has same issue13:47
bkeropabelanger: fun, I wonder when we'll get that in a released version13:48
EmilienMpabelanger, mordred: it's really on xenial13:48
openstackgerritMerged openstack-infra/project-config: Added pre_hook for sahara cli tests
bkeroand how many times that loop runs13:49
*** coolsvap has quit IRC13:49
*** ramishra has joined #openstack-infra13:49
pabelangerbkero: gives us a better way to track the loop runs, but still up for discussion if we want to do that13:50
*** PollyZ has joined #openstack-infra13:50
*** rhochmuth has joined #openstack-infra13:50
*** coolsvap has joined #openstack-infra13:50
pabelangerWould like to stabilize async_wrapper first13:51
*** hashar has joined #openstack-infra13:52
openstackgerritMerged openstack-infra/project-config: Use Smaug configuration from Smaug's repository
openstackgerritMerged openstack-infra/project-config: panko: enable functional testing
*** oferby has quit IRC13:53
Shrewsmordred: pabelanger: did anyone try executing the jobs locally now that we keep the ansible files around?13:53
*** Ravikiran_K has quit IRC13:53
bkeroI still think the additional logging will be useful, even after async_wrapper is fixed13:53
bkeroWhenever I see smaug I still think of
pabelangerShrews: I didn't have much time yesterday to work on the held node.13:54
*** hashar has quit IRC13:54
*** rossella__ has quit IRC13:55
*** itisha has joined #openstack-infra13:55
*** esberglu has quit IRC13:55
*** bhavik has joined #openstack-infra13:55
*** ddieterly is now known as ddieterly[away]13:56
*** yolanda has quit IRC13:56
*** esikachev has joined #openstack-infra13:56
*** pt_15 has joined #openstack-infra13:57
Shrewsi'd be very interested to see if zuul_runner exits the same when run locally13:57
*** pradk has joined #openstack-infra13:57
*** zul_ has joined #openstack-infra13:59
*** ddieterly[away] is now known as ddieterly13:59
*** hdaniel has quit IRC13:59
*** zul has quit IRC14:00
*** rbradfor has joined #openstack-infra14:00
fungimordred: planning to stamp out a new glean too? i think cinerama and some others were hoping for a release with some recently merged fixes14:01
*** sree has quit IRC14:01
*** zul_ has quit IRC14:01
*** esberglu has joined #openstack-infra14:02
*** sree has joined #openstack-infra14:02
*** zul has joined #openstack-infra14:02
*** rbradfor has quit IRC14:02
*** esberglu has quit IRC14:02
openstackgerritMerged openstack-infra/project-config: [magnum] Add install-guide job
*** sree has quit IRC14:03
*** sree has joined #openstack-infra14:04
*** achanda has joined #openstack-infra14:05
*** rbradfor has joined #openstack-infra14:05
openstackgerritMerged openstack-infra/project-config: Remove Heat lbaasv1 from master
openstackgerritMerged openstack-infra/project-config: Run Heat functional tests in Identity v3-only
openstackgerritMerged openstack-infra/project-config: Don't run tempest tests in zaqar gate
*** sree has quit IRC14:06
*** achanda has quit IRC14:07
*** sree has joined #openstack-infra14:07
*** shardy has joined #openstack-infra14:09
*** sree has quit IRC14:11
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: UPDATE REPO FOR DELOREAN_REPO_URL
*** woodster_ has joined #openstack-infra14:12
*** zqfan has quit IRC14:13
*** sdake has quit IRC14:13
*** zhurong has quit IRC14:15
*** kdas__ has joined #openstack-infra14:16
*** sdake has joined #openstack-infra14:16
pabelangerjeblair: mordred: after 12 hours, all our zuul-launchers are now graceful restarted14:17
pabelangerhappy that our playbook worked for 12 hours14:17
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run identity v3 only job in heat
pabelangerlittle sad it took 12 hours14:18
*** zhurong has joined #openstack-infra14:18
*** baoli_ has quit IRC14:19
*** rhochmuth has quit IRC14:19
fungiprobably gets complicated to tell ansible to do them in parallel but not to have more than n launchers in graceful shutdown at a time?14:20
*** baoli has joined #openstack-infra14:21
mordredfungi: ansible actually knows about batching rolling things14:21
mordredfungi: so that shoudl be very easy14:21
fungiso when things are quiet, we could tell it to do up to 4 at a time? that would be handyu14:22
*** eezhova has quit IRC14:22
mordredfungi:   serial: 114:22
*** piet has quit IRC14:22
*** bhavik has quit IRC14:23
*** zz_dimtruck is now known as dimtruck14:23
mordredfungi: is the line14:23
*** amotoki has joined #openstack-infra14:23
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Create Merger Git repos using connection names
*** amitgandhinz has joined #openstack-infra14:23
fungiso that's an actual integer knob, not just a boolean flag?14:23
*** kdas__ is now known as kushal14:23
*** psilvad_ has joined #openstack-infra14:23
*** kushal has quit IRC14:23
*** kushal has joined #openstack-infra14:23
*** psilvad has quit IRC14:24
openstackgerritMonty Taylor proposed openstack-infra/system-config: Make batch size configurable for z-l restart
mordredfungi: ^^ untested14:25
EmilienMpuppet xenial jobs are too unstable for us, we need to land some patches and it's blocked I think I'm going to -nv them14:29
pabelangermordred: added comment about using strategy14:30
fungiEmilienM: i've not been following closely unfortunately. have any idea what's unstable there?14:30
EmilienMfungi: async thing with zuul14:31
fungiEmilienM: oh, that. and it's only on xenial?14:31
fungidoesn't affect trusty?14:31
EmilienMfungi: we don't run much trusty nodes lately14:32
EmilienMfungi: trusty nodes are run for mitaka/liberty branches14:32
*** amotoki has quit IRC14:32
EmilienMfungi: but since newton we deploy on xenial14:32
*** salv-orlando has quit IRC14:35
*** salv-orlando has joined #openstack-infra14:36
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: xenial jobs are now -nv
EmilienMmwhahaha, degorenko, fungi ^14:37
mordredpabelanger: oh yeah. totally agree14:37
*** ayoung has quit IRC14:37
*** jistr is now known as jistr|mtg14:38
*** pblaho has quit IRC14:38
fungiEmilienM: and you're not seeing the same instability on centos 7 jobs either?14:38
EmilienMfungi: nope14:38
EmilienMso, since zuul migration, we're having async issue with our integration jobs (they don't run ansible but puppet btw), on xenial nodes14:39
*** burgerk has joined #openstack-infra14:39
*** zhurong has quit IRC14:40
fungibut you run similar integration jobs on centos 7 and they've remained stable through the zuul migration?14:40
*** amotoki has joined #openstack-infra14:40
*** inc0 has quit IRC14:41
*** links has quit IRC14:41
*** pcaruana has quit IRC14:41
*** claudiub has joined #openstack-infra14:43
*** baoli has quit IRC14:43
AJaegerEmilienM: So, ansible+puppet on Xenial = broken is the current theory?14:44
EmilienMyes but this is only an observation, jeblair / pabelanger / mordred are investigating lately and today we should make progress14:44
EmilienMthe thing is we could not login a failed node to see what's wrong14:44
*** kushal has quit IRC14:45
EmilienMand now jeblair did that feature in nodepool, we'll be able to ssh a failed node and see what happenned14:45
*** jamesmcarthur has joined #openstack-infra14:45
EmilienMbut in the meantime, we can't continue to block our CI14:45
EmilienMnewton cycle is going on and we have a backlog of patches to land14:45
EmilienMFuel & TripleO are depending on Puppet modules, we can't block our CI anymore14:45
*** esikachev has quit IRC14:45
AJaegermaking it non-voting will not block our continous testing - and you still have voting jobs, so +2 from my side.14:47
EmilienMyeah, we still want them to continue the debug14:47
EmilienMjust make them non-voting will allow say "ok the patch is passing on centos7, and would also pass on ubuntu, so let's land it"14:47
AJaegerEmilienM: agreed14:48
*** baoli has joined #openstack-infra14:48
EmilienMthx AJaeger, mordred14:48
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add a node for artifact signing jobs
*** jlanoux has quit IRC14:50
*** inc0 has joined #openstack-infra14:51
*** ramishra has quit IRC14:51
*** coolsvap has quit IRC14:52
*** xarses has quit IRC14:52
*** baoli has quit IRC14:52
*** jaosorior has quit IRC14:53
*** deadnull_ has joined #openstack-infra14:53
*** Nikolay_St has quit IRC14:54
*** baoli has joined #openstack-infra14:54
*** coolsvap has joined #openstack-infra14:55
*** pilgrimstack has joined #openstack-infra14:55
*** armax has joined #openstack-infra14:57
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update config file provider docs to have default options
*** deadnull_ has quit IRC14:58
*** yamamoto has joined #openstack-infra14:59
*** david-lyle_ is now known as david-lyle14:59
*** shardy has quit IRC14:59
*** tbarron|errand is now known as tbarron15:00
openstackgerritMerged openstack-infra/project-config: puppet: xenial jobs are now -nv
*** esikachev has joined #openstack-infra15:01
*** xarses has joined #openstack-infra15:01
*** mhickey has quit IRC15:02
*** markvoelker_ has joined #openstack-infra15:03
*** amotoki has quit IRC15:03
*** markvoelker has quit IRC15:03
*** amit213 has quit IRC15:04
*** Na3iL has quit IRC15:05
*** eezhova has joined #openstack-infra15:05
*** pblaho has joined #openstack-infra15:05
*** tqtran has joined #openstack-infra15:06
*** bhavik has joined #openstack-infra15:07
*** zeih has quit IRC15:07
*** ramishra has joined #openstack-infra15:08
*** shardy has joined #openstack-infra15:08
*** esberglu has joined #openstack-infra15:08
*** amotoki has joined #openstack-infra15:08
*** xarses has quit IRC15:10
*** tqtran has quit IRC15:10
cloudnullo/ all15:10
fungicloudnull: all's quiet on the osic front? i hope you're here for a social visit and no a bearer of bad tidings!15:12
fungier, not a15:12
*** eezhova has quit IRC15:12
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Store history of a comment when editing
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Add a way to retrieve the history of edited comments
cloudnullfungi: not at all.15:13
cloudnulljust here to say hi.15:14
cloudnullbeen on holiday for a while. so its good to hear the osic front is all quite :)15:14
*** sdake_ has joined #openstack-infra15:14
fungicloudnull: absolutely. good job!15:15
*** pilgrimstack has quit IRC15:15
fungiyou should go on holiday more often ;)15:15
* cloudnull puts it on the books15:15
*** baoli has quit IRC15:15
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Clean floating IPs on tripleo-test-cloud-rh1
*** sdake has quit IRC15:16
pabelangerEmilienM: are we leaking FIPs in tripleo?15:16
EmilienMpabelanger: I discussed with derek yesterday, and he mentionned that he had to manually release floating ips from time to time15:17
EmilienMotherwise our CI would get stuck, since no  job could run15:17
EmilienMI think it's related to what I'm trying to do15:18
EmilienMpabelanger: he wanted to setup a crontab that clean this up15:18
EmilienMpabelanger: but I'm sure we can solve it in another way15:18
*** amotoki has quit IRC15:18
clarkbEmilienM: nodepool has a built in cleanup fips thing we can enable on tht cloud15:19
*** vhosakot has joined #openstack-infra15:19
clarkb(assuming it got fixed in shade)15:19
pabelangerEmilienM: not sure I follow "related to what I'm trying to do" comment15:19
*** sdake_ has quit IRC15:20
pabelangerbut honestly, I never looked at tripleo-test-cloud-rh1 for leaked IPs15:20
*** jlanoux has joined #openstack-infra15:20
EmilienMclarkb: is it what you mentionned?15:21
*** maishsk has quit IRC15:21
*** ddieterly is now known as ddieterly[away]15:21
EmilienMpabelanger: what I mean is, I looked a nodepool and I found this feature, I thought it could help us to remove unsassigned floating ips15:21
*** dizquierdo has quit IRC15:21
clarkbyes but depends on shade being fixed and I dunno if it goy fixed15:21
EmilienMclarkb: do we have a launchpad or something?15:22
greghaynesianw: looks like the dib cache change broke our CI15:22
pabelangerclarkb: Yup, shade was released with the fix15:22
clarkbpabelanger: and we installed it and restarted nodepool?15:22
*** dizquierdo has joined #openstack-infra15:22
pabelangerclarkb: ya, it landed a while back.  maybe 3+ weeks15:22
clarkb(nodepool was restarted yeaterday so if release is older we are good)15:22
*** _nadya_ has quit IRC15:22
*** shahid__ has quit IRC15:23
EmilienMthis one?15:23
*** baoli has joined #openstack-infra15:23
mordredclarkb: can't use it with tripleo-ci15:23
EmilienMah no
pabelangerclarkb: oh, maybe I am confusing with another fix15:23
EmilienMmordred: why?15:23
*** baoli has quit IRC15:23
mordredbecause we use nova api to do floating ips on the tripleo cloud15:23
mordredand it's not safe to run the cleanup fips with the nova api15:23
clarkboh right15:23
pabelangerAh, right. For got about that15:24
openstackgerritgreghaynes proposed openstack-infra/project-config: Make dib_cache dir with sudo in dib jobs
greghaynesianw: ^15:24
EmilienMdamn, so I can abandon my patch?15:24
mordredwe use the nova api because the neutron api for create/attach does not work on the tripleo cloud15:24
*** ddieterly[away] is now known as ddieterly15:24
mordredyah. sorry15:24
EmilienMmordred: can you comment on please?15:24
mordredEmilienM: I _think_ the neutron api doesn't work because tripleo-test-cloud-rh1 control plane is old and there is a bug?15:24
*** baoli has joined #openstack-infra15:25
EmilienMmordred: that's possible, I have no access on it, so who know?15:25
mordredEmilienM: I don't know if you're going to deploy a new control plane with the rh2 cloud or if  it's just a move15:25
EmilienMdprince: can you confirm?15:25
*** shardy has quit IRC15:25
openstackgerritMerged openstack-infra/puppet-openstack_health: Add build_newrev to the list of ignored keys
pabelangerrh2 is neutron, according to clouds.yaml for nodepool15:25
EmilienMcool, so I -1
EmilienMwe can add the boolean ^15:26
pabelangertested on it yesterday too, for AFS mirror15:26
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Use ssh-agent instead of ugly ssh-wrapper scripts
pabelangerworked the first time15:26
EmilienMpabelanger: see the patch for rh2, not goot then15:26
mtreinishnibalizer: ^^^ \o/ it finally landed15:26
pabelangerEmilienM: maybe a follow up patch?15:26
mordredpabelanger: cool - so clouds.yaml for rh2 doesn't have the nova override and auto_ip in launch_node worked properly?15:27
mordredthat's great new15:27
pabelangerEmilienM: otherwise, we can start landing rh2 today15:27
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: WIP: Allow comments to be edited by their owner
EmilienMpabelanger: ok, you can +A, and I'm proposing a following patch15:27
pabelangermordred: Ya, this was their patch:
pabelangerwhich I used with launch-node.py15:28
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Add ability to use SSH instead of HTTP to get refs
clarkbpabelanger: and it floating IP'd and all that?15:28
pabelangerclarkb: Ya15:28
mordredso we shoudl be able to enable clean-fips for that cloud in nodepool.yaml then15:29
pabelanger too15:29
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Clean floating IPs on tripleo-test-cloud-rh2
EmilienMpabelanger: ^ done15:29
pabelangerworked first try, which made me happy15:29
pabelangerokay, if no objections, I'll help get rh2 online today15:30
pabelangerI think we need to restart nodepool-builder too15:30
EmilienMpabelanger: excellent15:30
mordredpabelanger: it's almost like our automation works or something15:30
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Clean floating IPs on tripleo-test-cloud-rh2
*** sdake has joined #openstack-infra15:31
pabelangermordred: I can abandon my research on cloning15:32
*** salv-orlando has quit IRC15:33
*** ldnunes has quit IRC15:33
*** sdake_ has joined #openstack-infra15:34
openstackgerritDoug Hellmann proposed openstack-infra/project-config: stop merging tags from stable branches into master
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Use ssh-agent instead of ugly ssh-wrapper scripts
*** coolsvap has quit IRC15:35
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Use ssh-agent instead of ugly ssh-wrapper scripts
*** PollyZ has quit IRC15:36
*** PollyZ has joined #openstack-infra15:36
*** sdake has quit IRC15:36
*** ayoung has joined #openstack-infra15:37
*** Rockyg has joined #openstack-infra15:37
*** jamesmcarthur has quit IRC15:37
*** cody-somerville has joined #openstack-infra15:44
openstackgerritMerged openstack-infra/project-config: Add a temporary tripleo cloud "rh2"
*** ramishra has quit IRC15:45
*** ramishra has joined #openstack-infra15:46
*** dmk0202 has quit IRC15:46
*** jamesmcarthur has joined #openstack-infra15:46
*** coolsvap has joined #openstack-infra15:47
*** psachin has quit IRC15:47
_alastor_Hey, is this the right place to ask about the ci-watch tintri pages?15:48
*** armax has quit IRC15:48
*** nwkarsten has quit IRC15:49
*** nwkarsten has joined #openstack-infra15:49
BobBallI think so15:50
_alastor_The cinder one hasn't posted any results here in at least 24-hours15:51
mmedvede_alastor_: the person who is supporting ci-watch deployment at tintri is apoorvad15:51
mmedvedebut he is not on irc right now15:51
_alastor_mmedvede: ok, I'll check with him when he comes online15:52
_alastor_mmedvede: in the meantime, do we have an alternative place to look at the results?15:53
jeblairpabelanger, mordred, Shrews: good morning!15:53
jeblairShrews: in the errors we are seeing, zuul_runner exits correctly at the end of the job and before the timeout.  it's just that ansible erroneously gives up before that happens.15:53
asselinrcarrillocruz, is infra still using git+git:// now that there's no puppetmaster?15:53
*** ramishra has quit IRC15:53
pabelangerjeblair: morning15:54
mordredasselin: yes15:54
mordredasselin: we use it to run puppet apply15:54
mmedvede_alastor_: I am not aware of any other place. I was working on getting ci-watch deployed on an infra VM, but very busy with other stuff right now15:54
asselinmordred, ok, docs state it requires a puppetmaster, maybe just out of date.
fungi_alastor_: i'm guessing not. it isn't currently managed by the infra team so there's not a lot of help we can provide. is the plan for solving that but i'm not sure how far along it's gotten15:55
mordredasselin: ah - one sec15:55
jeblairianw, pabelanger: i don't care what creates the user on the image, as long as it's not system-config puppet.  it should be completely self-contained and usable by both us and others.  we need a good story for "if you want to use zuul with nodepool, you should create dib images like this..."  personally, i think it would be better for the system as a whole to not depend on both ansible *and* puppet, so i'd rather it be created some ...15:56
jeblair... other way.  personally, as a dib user, i'd be pretty surprised if there wasn't a nice way to get a user on a disk image -- i mean, that's very much in the wheelhouse of something that creates disk images i think.15:56
*** andreas_s has quit IRC15:56
mmedvedefungi: there is puppet code that deploys ci-watch here, need to get it merged15:56
_alastor_fungi: Thanks.  I'll keep an eye on that spec15:57
asselinmordred, rcarrillocruz fyi, i'm interested in using this: so looking for docs/starting points to make it happen in my env.15:57
*** xarses has joined #openstack-infra15:57
pabelangerjeblair: completely agree15:57
openstackgerritMonty Taylor proposed openstack-infra/ansible-role-puppet: Update README with info about puppet apply
mordredasselin, rcarrillocruz ^^15:58
openstackgerritMerged openstack-infra/project-config: Clean floating IPs on tripleo-test-cloud-rh2
*** ldnunes has joined #openstack-infra15:58
dprinceEmilienM: I'd have to check. Our neutron is very old at this point15:58
mordredjeblair: yah- and the devuser element exists and whatnot for getting a user and a public key on15:59
mordreddprince: is the rh2 a new install?15:59
asselinmordred, adding an example playbook for puppet apply at the end would be helpful15:59
mordredasselin: agree15:59
dprincemordred: yes, we should just wait to use that15:59
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add new db api method to get recent successes by metadata
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add a node for artifact signing jobs
mordreddprince: cool. yah - the neutron fip issue we see with rh1 is not present in rh2 - so I was assuming it was newer :)15:59
openstackgerritMatthew Treinish proposed openstack-infra/project-config: Fix preseed script for the brave new world
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: add a "ptl" command to show info about how to contact a PTL
*** bhavik has quit IRC16:02
*** Nikolay_St has joined #openstack-infra16:03
*** markvoelker has joined #openstack-infra16:03
pabelangermy only concern about devuser (and DIB elements in general) is the potential creep of configuration management that could be added into elements.  I do believe tooling like puppet and ansible are better for that.  And honestly, I feel if we had an run-ansible-playbook element we'd likely start adding playbooks for our zuul worker elements16:03
*** markvoelker_ has quit IRC16:04
*** pilgrimstack has joined #openstack-infra16:05
*** e0ne has quit IRC16:05
*** jcooley has joined #openstack-infra16:05
greghaynesI wouldnt make that an element, but if you want to run ansible in dib it should be trivial (just... run ansible). If theres a more consistent API wanted for that though it'd probably need to go in dib directly16:06
mordredgreghaynes: I16:06
mordredgreghaynes: I've sketched out a design for it a few times in my head ... the nice thing with dib-ansible is ansible's chroot connection plugin16:07
mordredwhich should make it pretty cool to deal with16:07
mordredgreghaynes: I should maybe write down the things that are in my brain16:07
pabelangerYa, I haven't tried chroot with ansible, but it does look interesting16:07
greghaynesYea, it makes a lot of sense. Where itll get wierd is how you make it interleave with the existing script ordering16:07
*** tphummel has joined #openstack-infra16:08
greghaynese.g. I want this playbook run before install.d/50-bar16:08
mordredgreghaynes: yah16:09
jeblairif you look at this:
jeblairthere's a list of elements there, that pretty clearly says "to make a fedora-23 image for nodepool, build it up from these parts"16:11
greghaynesI do disagree that elements and config management are somehow exclusive though, thats how the existing config mgmt stuff works. Really its a question of whether youre using elements for composition or youre using an element to call out to something else doing your composition. I think the former is a ton less complex so not sure why that isnt the goal16:11
jeblairand by and large, it's pretty legible16:11
jeblairyou can tell what each one is doing, and why you would want it on your nodepool image16:11
jeblairexcept 'puppet'16:11
jeblairthat's sort of 'magic goes here'16:12
greghaynesYep, exactly what I'm thinking :)16:12
jeblairso i'd love it if that went away, and we had 'install-zuul-user'... and, honestly, hopefully not much else :)16:12
*** bhavik has joined #openstack-infra16:13
*** dtantsur is now known as dtantsur|afk16:14
pabelangergreghaynes: for me, I've always seen elements used to create the base image. And call out to do configuration using another tool.16:16
*** yamahata has joined #openstack-infra16:16
dhellmannare folks aware of "disk quota exceeded" failures in the wheel jobs?16:16
dhellmannwe're seeing a lot of failures on those now that the release team is watching the output by email16:17
fungidhellmann: no, but i'll have a look on the server and see if it's piled up some cruft16:17
*** tesseract- has quit IRC16:17
greghaynespabelanger: Thats not really realted to the issue is what I am getting at - The API for an element is intentionally 'something that is executable' so you can call out to whatever tool works best, the real issue is that composition needs to happen from elements not one element that composes all the things16:17
dhellmannfungi: I haven't done an exhaustive review to see if that's the cause of all of the failures16:18
*** dosaboy has quit IRC16:18
fungidhellmann: it shouldn't impact the release process, but it does imply that we're not building any new centos 7 wheels in our wheel mirrors (unrelated to publishing and mirroring wheels of our releases on pypi)16:19
dhellmannfungi : yeah, I was more worried about the release folks ignoring failure emails because "oh, it's always that wheel build failure"16:19
*** raunak has joined #openstack-infra16:19
*** raunak has quit IRC16:20
dhellmannI mean, I do also want it to work right for its own sake. :-)16:20
fungidhellmann: interestingly, we've got tons of free disk space on that server at the moment, so presumably the build itself eats a lot of space or is relying on a very small filesystem for something16:20
dhellmannwell, it's a "quota exceeded" error, not a disk full error16:20
dhellmannso I wonder if there's a misaligned config somewhere?16:20
clarkbcheck the afs volume maybe16:20
fungithis is afs quota16:21
fungi"k5start: unable to create PAG: Disk quota exceeded"16:21
*** sdake_ is now known as sdake16:21
funginot disk _space_ exceeded16:21
fungii glossed over the word quota earlier16:21
fungiso we presumably just need to increase our quota for the wheel mirrors?16:21
fungichecking our afs docs now to see if that's something we have details on16:22
jeblairthat console log file is large.16:23
* jeblair kills firefox16:23
fungilooks like we do `fs setquota /afs/ <some number of bytes>`16:23
*** maishsk has joined #openstack-infra16:24
*** amotoki has joined #openstack-infra16:24
jeblairfungi: i'm a little confused about getting that error from k5start16:24
*** jamesmcarthur has quit IRC16:24
pabelangerour AFS quotas look to be okay16:24
pabelangerI may have run the incorrect command16:25
fungiis there a special meta-volume k5start is using?16:25
*** ifarkas has quit IRC16:25
jeblairfungi: k5start's job is to get a kerberos tgt, and afs service ticket, and create a linux PAG group to hold them16:25
fungiyeah, that quota looks sane16:25
fungiso it's when trying to write the pag that it's dying on this16:26
*** raunak has joined #openstack-infra16:26
jeblairwhere do those get written?  (tmp?)16:26
fungithat's what i'm looking for16:26
*** Apoorva has joined #openstack-infra16:27
*** jamesmcarthur has joined #openstack-infra16:27
*** pilgrimstack has quit IRC16:27
*** Nikolay_St has quit IRC16:28
*** esberglu has quit IRC16:28
fungilooks like the error is generated here...
jeblairsudo su - jenkins; k5start -t -f /etc/wheel.keytab service/wheel -- bash16:29
jeblairthat will reproduce the error ^16:29
jeblairwhen i run it as root, it succeeds16:29
fungionly readable by jenkins, not writable16:30
fungioh, the keytab is, but shouldn't be written to anyway16:30
fungiso nevermind16:30
*** kushal has joined #openstack-infra16:31
fungiseeing if i can figure out how k_setpag() decides where to write16:32
*** esberglu has joined #openstack-infra16:32
jeblairweirdly, /var/cache/openafs is empty16:33
jeblairi don't know if that's where it writes, but that's where i expect the afs client cache data to be16:33
fungionly writable by root, fwiw16:33
jeblairfungi: that should be fine -- it's only written to by the cache manager which runs as root16:34
fungik5start doesn't look setuid16:34
jeblairfungi: and k5start is doing an ioctl to do the pag thing16:34
fungithought you were suggesting that's where it might try to write the pag16:34
fungiare you stracing it?16:35
jeblairfungi: i do expect the cache manager to be writing things (cached files) to that directory -- i don't know if it stores pag data there too.16:35
*** matrohon has quit IRC16:35
jeblairfungi: i did to see the ioctl16:35
jeblairfungi: it's the ioctl that returns the error16:35
jeblairthat's the cacheinfo file16:36
jeblairi would like to restart the cache manager16:37
fungino objection from me16:37
* jeblair looks for something vaguely resembling an init script16:37
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update sonar plugin
jeblairwhere do the unit files or whatever live on centos7?16:38
mordredjeblair: you do systemctl something something16:38
fungithey're in /usr/share/systemd/something i think16:38
*** dosaboy has joined #openstack-infra16:39
jeblairwhat i actually did was tab-complete16:39
fungiif you want to modify them you copy them to /etc/systemd/something and modify that to override16:39
jeblairi just wanted to know the name :)16:39
jeblairservice open<TAB>16:39
fungiyeah, happens to me all the time16:39
*** pvaneck has joined #openstack-infra16:40
*** mriedem1 has joined #openstack-infra16:40
jeblairsystemd says:16:40
jeblairJun 23 16:40:27 afsd[27561]: afsd: Error calling AFSOP_CACHEINODE: not configured16:40
*** pilgrimstack has joined #openstack-infra16:40
mordredjeblair: is the config file on centos somewhere else?16:41
jeblairmordred: yeah, i think our puppet accounts for it...16:41
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update sonar plugin
*** jistr|mtg is now known as jistr16:42
mordredor, rather /usr/vice/etc16:42
jeblairyeah, that dir looks okay so far16:43
*** ddieterly is now known as ddieterly[away]16:44
jeblairthere are 4 afsd's running16:44
*** mriedem has quit IRC16:44
*** jpich has quit IRC16:44
anteayareview-dev is 500'ing for me16:44
jeblairsystemd said the word 'failed' literally 6 times16:44
jeblairdoes that mean success?16:44
anteayamaybe just that one patch, how odd16:44
jeblairmordred, fungi: i would like to reboot the machine16:45
mordredjeblair: ok16:45
jeblairpabelanger: ^?16:45
pabelangerjeblair: okay16:45
fungijeblair: the wheel builder node? fine with me. it should be nondisruptive16:46
jeblairthere is a job running.  we'll see what happens :)16:46
mordredI have so many snarky systemd comments in my brain I don't even know how to start16:46
fungithere's just about always a job running there anyway since they take an hour-ish and get rerun for every single release tag16:46
fungiso unless nobody's tagged anything for a while, the wheel builders are pretty much always backlogged16:47
nibalizer/usr/vice seems like where I would keep my etc files16:47
fungimordred: i've long since burned out the systemd snark centers of my brain16:47
funginibalizer: /miami/vice16:47
*** ddieterly[away] is now known as ddieterly16:48
jeblairhow long till we start to wonder when it's going to come back up?16:48
pabelangerAnsible exit code: 316:49
nibalizeroh AFS was originally named 'vice'16:49
jeblairright on :)16:49
pabelangerseem see what z-l does :)16:49
openstackgerritZara proposed openstack-infra/storyboard-webclient: Show Each Task ID in Story Detail View
pabelangeroh neat, it is looping16:49
jeblairnibalizer: vice and venus16:49
jeblairso, uh, i guess ill dig out some rax creds and see if i can see a console16:50
*** akshai has quit IRC16:51
mordredjeblair: yay!16:51
*** dizquierdo is now known as dizquierdo_afk16:53
*** mriedem1 is now known as mriedem16:54
*** pilgrimstack has quit IRC16:55
jeblairum.  when i go to that page i get "this plugin is disabled".16:55
jeblairi don't know what "this plugin" is, unfortunately.16:55
jeblairoh, if i click on the lego, i see it's java16:56
jeblairi didn't think they required java16:56
*** arxcruz has quit IRC16:56
mordredfor the consone16:56
mordredit's a java applet16:56
mordredjeblair: we could cloud and just create a new centos7 wheel builder ... it doesn't have an precious data16:57
jeblairmordred: we could spend a lot of time doing that to get to this point :(16:57
mordredjeblair: that's a very good point16:57
*** akshai has joined #openstack-infra16:58
jeblairokay, even enabling java is a no go16:59
jeblairi'll press the reboot button harder16:59
jeblair(this is strange; i am certain i have used the rax console many times)17:00
jeblairpressing reboot harder worked17:00
*** Srinu_ has joined #openstack-infra17:01
*** haaj has quit IRC17:01
jeblairi have starte openafs (systemd did not do it automatically -- maybe because it failed?  maybe because i did not sacrifice?)17:01
jeblair/var/cache/openafs/ has file in it17:01
*** haaj has joined #openstack-infra17:02
mordredjeblair: we have to do systemctl enable to tell systemd to automatically start a service on boot17:02
jeblairand k5start bash as jenkins worked17:02
mordredjeblair: woot!17:02
jeblairmordred: okay, i have run systemctl enable openafs-client.service17:02
*** pilgrimstack has joined #openstack-infra17:02
jeblairi would have expected puppet to have done that, but who knows17:02
jeblairit said it created symlinks when i ran that, so i think that confirms it was not enabled17:03
jeblair(i ran it again and it said nothing)17:03
mordredjeblair: maybe there is more magic we need to tell puppet to tell systemd17:03
mordredcrinkle, EmilienM: do you happen to know what we need to do to tell systemd to enable a service to start on boot in puppet?17:03
*** azvyagintsev has joined #openstack-infra17:04
*** esberglu_ has joined #openstack-infra17:04
clarkbdouble check we have a service ensure enabled?17:04
Srinu_Jeblair: i am running jenkins job when the build in progress the slave went offline and did you face this issue can you check this logs please17:04
openstackgerritMonty Taylor proposed openstack-infra/puppet-openafs: Enable the openafs-client service
*** isaacb has quit IRC17:06
openstackgerritMonty Taylor proposed openstack-infra/puppet-openafs: Enable the openafs-client service
mordredEmilienM: thanks!~17:06
*** daemontool_ has quit IRC17:06
*** haaj has quit IRC17:06
*** pilgrimstack has quit IRC17:06
EmilienMmordred: yw!17:06
*** dizquierdo_afk has quit IRC17:07
*** esberglu has quit IRC17:07
asselinrcarrillocruz, how do you run ? All the parts at the top are supposed to be modified before using?17:09
*** esberglu_ has quit IRC17:10
rcarrillocruzAt the airport17:10
Srinu_Asselin: hi17:10
rcarrillocruz-e cloud blah17:10
rcarrillocruzSame with other values17:10
*** Jeffrey4l has quit IRC17:11
*** coolsvap has quit IRC17:11
*** shashank_hegde has joined #openstack-infra17:11
*** nmagnezi has quit IRC17:11
Srinu_Asselin: did you face that issue17:12
*** esberglu has joined #openstack-infra17:12
*** rdx_matt has quit IRC17:12
asselinrcarrillocruz, ok....I'm still ansible not following what you mean by -e cloud blah.17:12
pabelangerjeblair: do you think we still need cgroups on our long lived zuul workers? Here is a ML post with more details:
asselinSrinu_, ?? which issue?17:13
*** tqtran has joined #openstack-infra17:13
rcarrillocruzYou can pass param valued from CLI with -e17:13
Srinu_Slave went offline while build in progress17:14
rcarrillocruz-e cloud=devstack-admin17:14
jeblairpabelanger: good question, we should ask clarkb to help refresh our memories on that17:14
Srinu_Asselin: can you check this logs17:14
*** signed8bit is now known as signed8bit_Zzz17:14
rcarrillocruzThat as an ansible-playbook argument17:14
*** markmcd1 has quit IRC17:14
*** markmcd has quit IRC17:14
pabelangerSrinu_: I am unsure what else you are looking for. Multiple people have commented about OFFLINE_NODE_WHEN_COMPLETE not being properly setup. You should look to be fixing that17:15
pabelangerjeblair: ack17:15
*** vhosakot has quit IRC17:16
Srinu_Pabelanger: i fixed that issue17:16
*** signed8bit_Zzz is now known as signed8bit17:16
*** markmcd has joined #openstack-infra17:17
pabelangerSrinu_: you are still saying your slave went offline during a build, that is because nodepool is likely deleting it still17:17
pabelangerlook in your nodepool debug.log and see what is happening to the node17:17
*** devananda has quit IRC17:17
Srinu_Pabelanger: check this logs17:17
pabelangerSrinu_: right, the slave was deleted17:18
pabelangerlikely by nodepool17:18
asselinrcarrillocruz, I get an ansible yaml sytax error: /home/asselin/git/openstack-system-config/playbooks/launch_node.yml': line 46, column 53,17:18
Srinu_Pabelanger: because of ssh connection closed17:19
jeblairpabelanger, mordred, Shrews: okay, so i think the next step is to delete the previously held nodes, enable verbose logging in all the launchers, then wait for nodepool to catch another one, yeah?17:19
pabelangerSrinu_: the ssh connection likely closed because the server we deleted.  Does the node still exist?17:19
asselinSrinu_, yeah, what pabelanger said, or network failed. Check the timestamps in the nodepool log & jenkins console log files to see which happened first.17:19
pabelangerjeblair: +117:20
*** coolsvap has joined #openstack-infra17:20
Srinu_Pabelanger: when slave went offline the slave will be deleted by nodepool17:20
*** ddieterly is now known as ddieterly[away]17:20
jeblairpabelanger, mordred, Shrews: verbose enabled on all launchers17:21
clarkbpabelanger: jeblair the cgroups are there to restrict memory for the most part on the jenkins user processes. This means OOMkiller targets those processes and not eg systemd17:21
jeblairpabelanger, mordred, Shrews: we'll need to find a job started *after* 17:20 for that to take effect17:21
clarkbthe risk is far lower now that we really only run truated things on them17:22
clarkbbut could be a good safety net for wheel builds and the like17:22
Srinu_Pabelanger: some tests are passed but some are failed due to slave went offline17:22
*** raildo is now known as raildo-afk17:22
openstackgerritZara proposed openstack-infra/storyboard-webclient: Show Each Task ID in Story Detail View
*** sdake has quit IRC17:22
jeblairEmilienM: we're now set up to capture lots of debug output during the next failure17:23
EmilienMjeblair: ok thx, do you want me to kick off lot of CI jobs?17:23
EmilienM(like 2 or 3 patches)17:23
pabelangerSrinu_: right, when nodepool is finished with a nodepool, part of the process is to make the node offline in jenkins then delete the node from the cloud.  If a job is still running when that happens, then OFFLINE_NODE_WHEN_COMPLETE is not configured properly.  Only after a job is SUCCESS or FAILURE, does nodepool delete a node17:23
*** BobBall is now known as BobBall_AWOL17:23
jeblairEmilienM: that would be great17:23
Srinu_Pabelanger: some jobs are passed and some jobs are failing due to slave went offline17:24
*** sarob has joined #openstack-infra17:24
pabelangerSrinu_: if you look at your jobs in jenkins, you case see which node they run on.  each job should be limited to a single node17:24
*** kushal has quit IRC17:24
pabelangerSrinu_: so, if 2 jobs are running on the same slave, then you have a configuration error17:24
*** vhosakot has joined #openstack-infra17:25
*** PollyZ has quit IRC17:25
pabelangerclarkb: okay, so it should like we need to move cgroups out from jenkins into a generic puppet module17:25
*** trown is now known as trown|lunch17:25
pabelangerclarkb: or set it up in our DIB element17:25
pabelangeractually, not DIB, since this is long lived workers17:26
EmilienMjeblair: we currently have 12 xenial jobs (that are supposed to work, and 12 others that were -nv before), that are running now17:26
*** shashank_hegde has quit IRC17:26
EmilienMjeblair: /me afk lunch -- will monitor that17:27
*** ayoung has quit IRC17:28
Srinu_Pabelanger: i enabled single slave use node which will work same as offline node when complete. The job that are in queue are not taking the same node and then i run the jenkins jobs but 4 passed and remaining failed due to slave went ofline17:28
asselinrcarrillocruz, where are the dependent roles defined? ERROR! the role 'bootstrap_puppet' was not found17:29
pabelangerSrinu_: you need to figure out why the slave went offline. Use the node id and look into nodepool debug logs17:30
pabelangerSrinu_: either nodepool deleted it, or you have a networking issue17:30
Srinu_Pabelanger: i got this error in logs check this log pleae17:30
pabelangerSrinu_: you have posted that log 3 times now, it doesn't provide the right data you need.17:31
pabelangerSrinu_: look at nodepool, not jenkins17:31
Srinu_Pabelanger: when slave went offline and node is deleting automatocally17:31
asselinSrinu_, /var/log/nodepool/nodepool.log check the timestamps for actions on that node17:32
*** PollyZ has joined #openstack-infra17:32
pabelangerwhat asselin said17:32
*** vhosakot has quit IRC17:33
Srinu_Pabelanger: in nodepool log it is deleting and creating a new slave17:33
pabelangerSrinu_: show us your nodepool debug.log using paste.openstack.org17:34
pabelangerfungi: do you mind reviewing before I patch bomb our puppet modules?17:34
*** jamesmcarthur has quit IRC17:34
*** tosky has quit IRC17:34
pabelangerfungi: adding other-requirements.txt for bindep17:35
*** Srinu_ has quit IRC17:36
*** thiagop has quit IRC17:36
*** jcooley has quit IRC17:37
*** gabriel-bezerra has quit IRC17:37
*** thiagop has joined #openstack-infra17:37
fungipabelanger: do you happen to have any numbers on how much speedup that gave on that project?17:38
*** rcernin has quit IRC17:38
*** electrofelix has quit IRC17:38
fungipabelanger: which of our jobs there are using install-distro-packages, out of curiosity?17:38
*** gabriel-bezerra has joined #openstack-infra17:39
*** armax has joined #openstack-infra17:39
pabelangerfungi: in rax we shave about 2mins off our puppet-lint jobs17:41
mordredpabelanger: I look forward to your patch bomb17:42
*** salv-orlando has joined #openstack-infra17:42
pabelangerfungi: right now, only puppet-lint and syntax and beaker-rspec use it17:42
*** mriedem1 has joined #openstack-infra17:43
fungipabelanger: awesome, make sure you use a consistent topic on the patchbomb and i'll approve whichever ones are passing all their jobs17:43
*** mriedem has quit IRC17:44
pabelangerfungi: ack17:44
esbergluI’m hitting this error when trying to spawning a new instance17:44
*** vhosakot has joined #openstack-infra17:45
*** abregman has quit IRC17:45
pabelangeresberglu: looks like you need to migrate your database17:45
pabelangerlet me find the ML post17:45
*** pvaneck has quit IRC17:45
*** SumitNaiksatam has joined #openstack-infra17:46
fungipabelanger: also, i think we might have a cookiecutter template for infra puppet modules? if we do, make sure to update that too17:46
esberglupabelanger: thanks!17:46
funginibalizer: ^ probably same goes for the gem patch17:46
pabelangerfungi: good idea17:46
*** sarob_ has joined #openstack-infra17:48
pabelangerjeblair: is a failure, but surprisingly the extra verbosity is not logged in ansible_log.txt.17:49
*** salv-orl_ has joined #openstack-infra17:49
*** salv-orlando has quit IRC17:49
nibalizerwe do17:50
openstackgerritMerged openstack-infra/puppet-openstackci: Add other-requirements.txt for bindep support
pabelangerjeblair: it was captured by zuul-launcher17:50
nibalizerjeblair: whomever else is hacking on nodepool, our ibm team got caught by the schema change today17:51
*** sarob has quit IRC17:51
*** shashank_hegde has joined #openstack-infra17:51
jeblairnibalizer: yep, i sent an email:
nibalizeryah... i know17:53
*** e0ne has joined #openstack-infra17:54
*** tosky has joined #openstack-infra17:54
nibalizerbut i thought the reccomendation is to continously deploy, then we threw out something that isnt continously deployable17:54
nibalizerso that seems incongruent17:55
jeblairnibalizer: yep.  it's unfortunate.  there is a patch to add migrations that could use some love.  or hopefully we'll be on zookeeper before too much longer.17:55
*** jlanoux has quit IRC17:57
fungiwow... a hunt to track down an error with my current artifact signing implementation change took me all the way back to (yes, just 7, that's not a paste error)17:58
*** _nadya_ has joined #openstack-infra17:58
nibalizerjeblair: more than unfortunate... its downtime17:58
nibalizerseems wrong to me to break downstreams just to add a comment field to an operation that is very rarely run17:58
jeblairnibalizer: are you venting or do you have a concrete suggestion?17:59
*** mriedem1 is now known as mriedem17:59
fungiconcrete suggestion of mine would be that downstreams interested in maintaining continuous deployments of nodepool without vetting them would do well to help with one of the existing proposed fixes for that issue18:00
nibalizerfungi: continuous deployment is something we coded into puppet-openstackci, its not an expert-use-case, its the reccomended use case18:01
jeblairthe whole system is, currently, an expert use case18:01
jeblairi'm working on fixing that, but it's going to be a while18:02
funginibalizer: well, to be fair, it is something that was extracted from our config repo. it is also not free from needing improvement18:02
jeblairpuppet-openstackci is "openstack's ci system the way openstack runs it"18:02
*** kzaitsev_mb has quit IRC18:02
*** haaj has joined #openstack-infra18:02
fungimordred: i know this is an insane longshot, but do you remember what you wanted to use that pubring.gpg for? or should i try to import it and find out?18:02
jeblaireven if we add migrations, or some other fix, there will almost certainly be breaking changes of other types and other components in the future18:03
nibalizeras a downstream you have two options, fall behind or continously deploy18:03
*** PollyZ has quit IRC18:03
jeblairso it would be great for folks to be susbscribed to the mailing list for things like that18:03
*** PollyZ has joined #openstack-infra18:03
mordredfungi: uhm18:04
mordredfungi: which pubring.gpg ?18:04
fungimordred: in
fungimordred: here's the key it injects...
*** PollyZ has quit IRC18:05
mordredfungi: wow. you ... you just linked to change 718:05
fungimordred: yes, i was marvelling over that earlier18:06
mordredfungi: I now see that marvel18:06
nibalizerjeblair: hrm, im trying to think of ways to make things better18:06
fungimordred: pretty certain this is actually used in zero places at this point. i'm going to propose deleting that from the puppet-jenkins module (which is where it was eventually shuffled over to)18:06
mordredfungi: I believe those are there from when we used to make debs18:06
*** akshai has quit IRC18:06
openstackgerritMerged openstack-infra/project-config: Make dib_cache dir with sudo in dib jobs
jeblairnibalizer: i think the alembic change would go a long way18:07
AJaegerjeblair, pabelanger: With EmilienM's change to rename the xenial jobs to -nv, you might need to change the name of the jobs to hold...18:07
mordredfungi: because, you know, it used to be the workflow for us to make source packages and upload them to a PPA on launchpad and then use apt-get build-dep on the slaves to make sure that the depends were installed18:07
jeblairAJaeger: very good point18:07
nibalizerjeblair: would you be willing to send an email like the one you did, but send it a few days ahead of time? that way folk can be ready or turn off their CD if they rae not18:07
fungimordred: neat! the failures on the change i have proposed which led me to that nugget of history could eventually be used to sign secure apt repos ;)18:07
*** e0ne has quit IRC18:07
nibalizerjeblair: i don't know what alembic means18:07
pabelangerAJaeger: Oh, that might explain why I haven't seen a held node18:07
jeblairAJaeger: that explains why we didn't catch it18:07
mordrednibalizer, jeblair: I agree, I think the alembic change will help with most of these18:07
*** haaj has quit IRC18:07
mordrednibalizer: alembic is the database-migratoins patch18:08
jeblairnibalizer: alembic == database migrations18:08
jeblairnibalizer: in this instance, no.  that patch series was an emergency fix to help us debug an ongoing operational problem that has caused a project to switch half their jobs to non-voting18:08
mordrednibalizer: if we get that patch fixed up and merged, then nodepool schema changes should be safe for CD18:09
jeblairnibalizer: i think on the desperation scale, this is pretty close to the far end of things.18:09
*** PollyZ has joined #openstack-infra18:09
jeblairnodepool job-create gate-puppet-openstack-integration-3-scenario001-tempest-ubuntu-xenial-nv --hold-on-failure 318:10
jeblairpabelanger, AJaeger:^  look right?18:10
*** yamahata has quit IRC18:10
nibalizerjeblair: we're talkign about commenting a nodepool hold?18:10
nibalizeror something else?18:10
mordredjeblair: ++18:10
jeblairnibalizer: yes, it was a pre-requisite for the auto-hold failed node patch18:10
*** e0ne has joined #openstack-infra18:11
openstackgerritRick Aulino proposed openstack-infra/project-config: Upgrade experimental searchlight ES 2.3.2
jeblairnibalizer: which i think is, frankly, something nodepool operators have been screaming for for a while18:11
*** sdake has joined #openstack-infra18:11
*** csomerville has joined #openstack-infra18:11
mordredI do agree that a longer notice period on the mailng list when we start landing things like zookeeper and friends is totally fair18:11
jeblairi do not18:12
nibalizerjeblair: also 'emergency operational problem' is what was created for more organizations18:12
*** PollyZ has quit IRC18:12
pabelangerjeblair: it looks correct18:12
jeblairnibalizer: are you still trying to tell me something i already know?18:12
pabelangerjeblair: we can also add gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-xenial-nv for additional coverage if needed, I've seen that fail too18:13
jeblairnibalizer: again, we *clearly* understood the potential impact which is why i sent the email18:13
*** e0ne has quit IRC18:13
*** rdx_matt has joined #openstack-infra18:13
nibalizerjeblair: im not sure im being clear18:13
*** rdx_matt has quit IRC18:13
*** rdx_matt has joined #openstack-infra18:14
*** sshnaidm is now known as sshnaidm|afk18:14
pabelangerjeblair: regarding alembic and nodepool, we'd only support upgrade right? Not sure I'd want to deal with the downgrade path and testing18:15
*** cody-somerville has quit IRC18:15
clarkbpabelanger: yes, at least thats how the existing patch was written iirc18:15
jeblairpabelanger: yes, only upgrade18:15
mordredpabelanger: ++18:16
clarkbpabelanger: fungi why doesn't have any content? even if we don't need to install the deps becaues they are already there we should list them18:16
clarkbthen apt/yum will nop and say all is well18:16
*** degorenko is now known as _degorenko|afk18:16
clarkbor is the issue more that we install puppet from not the distro and all that?18:17
fungiclarkb: the implication is that it has no dependencies worth mentioning, but if there are some i'm game for having them listed there18:17
fungi_prerequisite_ dependencies that is18:17
clarkbfungi: it depends on puppet for one18:17
clarkbbut also beaker?18:17
fungiyeah, seems reasonable to list puppet in there, and whatever else we're preinstalling if the jobs don't install them explicitly already18:18
nibalizerjeblair: i guess i expected to hear 'yea that was our bad, we're gonna not do that in the future'18:18
*** armax has quit IRC18:18
funginibalizer: we likely will do that in the future18:18
nibalizeror at the least 'this is how we can work together on some agreements to make it so when we do break you, yuo have a chance to not be broken'18:18
nibalizerinstead the attitude is 'yep, lol'18:18
pabelangerclarkb: we could add puppet18:18
funginibalizer: the attitude is "there are changes which could be implemented to make that smoother, let's work on it18:19
jeblairnibalizer: i won't lie to you.  it was not a mistake.  and we *might* do it again in the future.  i will say that we understand the problem and the significance, and we try to avoid doing it, and we try to mitigate it as best we can.18:19
pabelangerclarkb: but, because our jobs passed when the file was empty, I stopped there18:19
jeblairnibalizer: the response is definitely not LOL18:19
jeblairnibalizer: i am not laughing18:19
openstackgerritJohn L. Villalovos proposed openstack-infra/elastic-recheck: Revert "Add e-r query for 1590139"
jeblairnibalizer: i am, in fact, quite upset that you think i might be18:19
nibalizerso when mordred said 'i agree longer notice for thing slike nodepool is fair' and you said 'i do not, nope'18:20
clarkbpabelanger: they pass because as part of the image build we preinstall all the deps, but I think we should still try to be explicit (though that may lead to bitrot and won't necessarily communicate we use puppet's packages not distro packages)18:20
jeblairnibalizer: that is far too speculative to discuss18:20
*** ajmiller has joined #openstack-infra18:20
jeblairnibalizer: i wish that mordred did not bring up what kind of notice or what we might do for zookeeper18:20
jeblairnibalizer: it is an entirely different problem18:20
*** zul has quit IRC18:20
jeblairnibalizer: to that, i will say, of course we don't want to break people and we want to make transitions as easy as possible18:20
nibalizerwell i will go back to ibm and try to convince them to get on the infra list18:21
*** ddieterly[away] has quit IRC18:21
*** amoralej is now known as amoralej|lunch18:21
*** amoralej|lunch is now known as amoralej|off18:21
funginibalizer: i agree with jeblair (and disagree with mordred) on that. we shouldn't slow the pace of development just to warn users that there are potentially breaking changes coming. nodepool is still a 0.x release, there are no stability guarantees for its interfaces18:21
nibalizerwe'll try to be vigilant, and when possible if you could give us the heads up that something is in the pipe (even if it is already landing) that would be really great18:21
pabelangerclarkb: I agree with that, if somebody tried to run the job out of the gate, it would likely fail.  Let me update the patch18:21
jeblairnibalizer: i do not know what that will entail, which is why i don't want to say something like "we will give you a weeks notice before we zk" because i don't know what it's really going to look like18:21
*** yamamoto has quit IRC18:22
*** PollyZ has joined #openstack-infra18:22
clarkbpabelanger: if we can put comments in bindep (I think we can) we could add a comment saying "puppet installed from puppet's repos" or something too to try and communicate that18:22
pabelangerclarkb: sure18:22
fungiyeah, bindep totally supports comments18:22
*** yamamoto has joined #openstack-infra18:22
nibalizerfungi: the 0.x forever isn't a good experience either though, projects that have existed for years and have users and are sitll 0.x just create conversations like this one18:22
jeblairnibalizer: yes, i will try to give as much notice as i can18:23
notmorganjeblair: this is related to the schema change, right?18:23
*** ddieterly has joined #openstack-infra18:24
*** akshai has joined #openstack-infra18:24
mordrednotmorgan: yah18:24
jeblairnibalizer: my full-time job is to make these things better.  they will get better, but it will take time.  some of them might even get a little worse before they do get better.18:24
fungipabelanger: clarkb: i recommend something along the lines of the header comment block i have proposed in too18:24
nibalizerjeblair: sure, and we want to be on upstream and on matser18:24
nibalizerbecuase I've seen the perma-nodepool-fork and its not pretty18:24
fungipabelanger: clarkb: so that people know what that file is and why it's there and how they can use it18:24
*** zul has joined #openstack-infra18:24
pabelangerfungi: okay, let me get a new patch up for that18:24
pabelangerand I'll pull in the tox.ini change too18:25
erlonguys, does anyone have see this failure in tempest gate?18:25
erlon -> gate-tempest-dsvm-neutron-full-liberty18:25
*** Sukhdev has joined #openstack-infra18:27
*** jcooley has joined #openstack-infra18:27
*** jamesmcarthur has joined #openstack-infra18:27
nibalizerjeblair: so what about cuting a 1.0? that way we can use version numbers to know about upgrade impact?18:27
nibalizerjust as a spitball idea, im sure there are reasons not to18:28
*** salv-orl_ has quit IRC18:28
jeblairnibalizer: well, i mean, we didn't even make a release here, so it doesn't address the CD issue18:28
jeblairnibalizer: honestly, the biggest gain we can do here is the alembic change18:28
mordredyah. I'm going to rebase that change real quick18:28
nibalizerwe could stop CDing if there were frequent releases18:28
jeblairit really will take care of 90% of the problem18:28
mordredI should have followed through on it like a year ago18:29
nibalizerso if mordred rebases that, i will use my rusty python eyes and review18:29
notmorgani was going to offer to resurrect it18:29
notmorganfor the short term.18:29
openstackgerritDoug Hellmann proposed openstack-dev/pbr: generate the reno cache during sdist if reno is installed
nibalizeryah lets just push on that18:29
mordrednotmorgan: woot. actually - if you want to take it, that would be great18:29
openstackgerritJeremy Stanley proposed openstack-infra/puppet-jenkins: Remove unused GnuPG keyring
notmorganmordred: yeah happy to do that. i need a break from staring at gear18:29
*** yamamoto has quit IRC18:30
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add a node for artifact signing jobs
mordrednotmorgan: the thing jeblair does in gertty I think is even better than what I have in this patch so far18:30
notmorganmordred: automatic migration on startup?18:30
mordrednotmorgan: which is gertty noticing that a migration needs to be run and running it for you, rather than needing a db upgrade command to be run18:30
mordredjeblair: ^^ thoughts?18:31
jeblairoh yeah, lets do that18:31
notmorganmordred: yeah that was my plan actually, was typing that18:31
jeblaircause otherwise it's sort of a puppet nightmare18:31
notmorgani mean, fail forward is the right answer here18:31
mordredand will introduce a breaking change to downstream deployments in order to avoid breaking changes :)18:31
notmorganesp. in 0.XX18:31
clarkbthe trickiest bit is really going to be making sure you don't miss any of the db changes in the last year and a half18:31
jeblairyou can create a new base migration18:31
mordredclarkb: I think just making that initial schema file froam sctach18:32
notmorganjeblair: ++18:32
mordredyah. that18:32
jeblairjust start from the current place18:32
clarkbthat works too18:32
nibalizernotmorgan: so whatever puppet-nodepool does18:32
notmorganwe start from *today* and go forward. sorry if that breaks you.18:32
nibalizeri think it just reloads?18:32
notmorganbut that is unwinding this specifically forward18:32
*** markusry has quit IRC18:32
pabelanger+1 to start from today18:32
notmorganchances are you were going to break otherwise.18:32
fungialternative is to make a single migration that starts from the most recent tag? not sure how easy that is though but might help people who upgrade from release to release (or is the first one a catch-22 there?)18:32
nibalizeroh hahah puppet-nodepool doesn't restart nodepool, neat18:33
jeblairyeah, that's intentional18:33
*** pvaneck has joined #openstack-infra18:33
jeblairit's a bit disruptive18:33
jeblair(we can revisit that after builders are in zk)18:33
clarkberlon: you might have better luck in the qa or stable maintenance channel?18:33
openstackgerritPaul Belanger proposed openstack-infra/puppet-openstackci: Add puppet dependency for bindep
nibalizerjeblair: ah ok18:33
clarkberlon: have you at least identified the failure and can link directly to that?18:33
*** _nadya_ has quit IRC18:34
jeblairpabelanger: so even though we didn't hold that node, did you get anything useful from the launcher logs?18:35
*** akshai has quit IRC18:35
notmorganmordred: i'll get on rebasing and making that thing all workable shortly18:35
*** salv-orlando has joined #openstack-infra18:36
pabelangerjeblair: I didn't see too much actually. confirms we are still SSHing into the system.18:36
*** lvdongbing has quit IRC18:37
*** th3rt has joined #openstack-infra18:37
pabelangerjeblair: need to step away for 30mins, when I get back I'll recheck some jobs in an attempt to catch a new node18:37
*** vhosakot has quit IRC18:37
*** trown|lunch is now known as trown18:37
jeblairmordred, Shrews, pabelanger, notmorgan: yeah, '-vvv' still isn't telling us why async_status failed.  i think we need serious ansible debugging help here.18:38
*** PollyZ has quit IRC18:39
pabelangerSo, apparently we can do stragegy: debug to launcher the ansible debugger. I've never tried it before18:39
Shrewsjeblair: my recommendation would be either abadger or bcoca. jimi is in the act of moving18:40
*** tongli has quit IRC18:40
* fungi wonders how intensely verbose that gets18:40
*** rodrigods has quit IRC18:40
*** rodrigods has joined #openstack-infra18:40
clarkbpabelanger: jeblair in this case you probably want to debug the status process on the remote end, not the launcher itself?18:41
*** coolsvap has quit IRC18:41
jeblairclarkb: probably so (maybe there's a small chance the problem is launcher-side)18:42
*** coolsvap has joined #openstack-infra18:42
jeblairclarkb: i want to see what async_status run on the worker output18:42
pabelangermaybe have the discussion about using poll: 0 and calling the async_status module directly as a possible solution to moving away from
*** r-mibu has quit IRC18:45
*** r-mibu has joined #openstack-infra18:45
Shrewspabelanger: i think that's a discussion worth having, but i think we need to bring in one of the ansible core guys to give their opinion, too18:46
*** vhosakot has joined #openstack-infra18:47
*** vhosakot has quit IRC18:47
*** vhosakot has joined #openstack-infra18:48
*** rdx_matt has quit IRC18:48
mordredShrews: yah. it seems like we might need more help from our friends over there - maybe we shoudl go chat with them in #ansible-devel ?18:49
*** PollyZ has joined #openstack-infra18:49
*** tonytan4ever has joined #openstack-infra18:49
*** hongbin has joined #openstack-infra18:51
*** Sukhdev has quit IRC18:51
pabelangerI have to run my errand now, I'll catch up on the discussions once I get back18:51
*** HeOS has quit IRC18:52
*** ccamacho is now known as ccamacho|Zzz18:52
*** kzaitsev_mb has joined #openstack-infra18:54
mordredjeblair: do we have a good link to the ansible logs of a failed run handy?18:54
*** maishsk_ has joined #openstack-infra18:54
jeblairmordred: yeah and
jeblairmordred: the second is the -vvv output18:54
*** Sukhdev has joined #openstack-infra18:54
fungihas anybody else noticed how much smoother our node utilization graphs are at peak now that the jenkins api is out of the picture?18:55
*** thorst is now known as thorst_away18:55
*** maishsk has quit IRC18:55
clarkbwe also don't seem to leak ready nodes18:55
*** vhosakot has quit IRC18:55
fungiit's like... "wham, yep here the quota"18:56
*** maishsk has joined #openstack-infra18:56
*** vhosakot has joined #openstack-infra18:56
pleia2how nice :)18:56
* clarkb gets back to unpacking the office18:56
fungithe zuul job queue "running" line at peak is the most impressive part i think18:56
*** ajmiller has quit IRC18:57
*** gordc has quit IRC18:57
jeblairthat reminds me, i think geard isn't recording disconnects properly in some cases, which is why the number of workers is so high18:57
jeblairi have not been able to figure that one out18:57
jeblair(can not repro locally)18:57
jeblairbut the other 3 lines should be accurate18:58
fungimy take on the graphs is that our node deployment and cleanup times are way more consistent (which would make sense given how long jenkins would take to respond under load, and how often it would fail to act on api calls)18:59
*** maishsk_ has quit IRC18:59
*** akshai has joined #openstack-infra18:59
*** amit213 has joined #openstack-infra19:00
*** shamail has joined #openstack-infra19:00
*** raunak has quit IRC19:01
*** sambetts is now known as sambetts|afk19:01
*** mdrabe has quit IRC19:03
*** haaj has joined #openstack-infra19:03
*** esikachev has quit IRC19:04
*** csomerville has quit IRC19:04
erlonclarkb: I'v tried the qa, but they are hard to get19:06
*** ajmiller has joined #openstack-infra19:06
erlonclarkb: but, this is one of the errors in q-*, there are others, but they does not see to be related to cinder19:07
clarkberlon: that gives yo umore info19:07
*** ajmiller has quit IRC19:07
clarkberlon: that seems like an issue either with the test asking for the wrong IP or with neutron/dhcp/etc19:07
*** jcooley has quit IRC19:08
*** ajmiller_ has joined #openstack-infra19:08
*** hashar has joined #openstack-infra19:09
*** haaj has quit IRC19:09
erlonclarkb: if were a scenario test I could understand why some IP weren't working, but is a create_snapshot19:09
*** mdrabe has joined #openstack-infra19:09
erlonclarkb: anyway, Ill still try to reach folks on QA19:10
clarkberlon: create_snapshot requires a VM to snapshot...19:10
*** gouthamr has quit IRC19:10
erlonclarkb: hmm19:10
*** mat128 is now known as mat128|away19:12
*** flepied1 has joined #openstack-infra19:19
*** ddieterly is now known as ddieterly[away]19:20
*** Rockyg has quit IRC19:20
EmilienMmordred: have you seen the async issue anywhere else than Ubuntu Xenial?19:20
mordredEmilienM: we haven't - but given conversation in #ansible-devel I think that might not be the actual cause19:21
mordredEmilienM: there are a couple of issues we may have found in the ansible side, looking in to patches now19:21
*** e0ne has joined #openstack-infra19:21
*** jamesmcarthur has quit IRC19:21
clarkbmordred: you saw my PR ya?19:22
mordredclarkb: yah. bcoca had a comment about removing the first sleep - so looks like putting that back and accounting for it in the loop may be more betterer19:22
*** flepied has quit IRC19:22
mordredclarkb: we've got two other potential bugs identified now too19:22
*** htruta is now known as henrique19:22
clarkbuh I need to see this comment, the first sleep is still there, it just happens in the loop?19:23
jeblairpabelanger, mordred, Shrews: i'm turning off verbose on all the launchers since we have what we need for this phase19:23
*** henrique is now known as Guest7841919:23
clarkbmordred: that comment is wat worthy, I will ahve to think on it19:23
*** Guest78419 is now known as htruta19:23
*** cody-somerville has joined #openstack-infra19:24
*** jamesmcarthur has joined #openstack-infra19:24
jeblairShrews, mordred: when we have those patches ready, we can manually install a patched ansible on all the workers.  since we exec, we won't need to restart or anything, so we can iterate on this pretty quickly.19:24
jeblairShrews, mordred: regardless of whether bcoca wants the async_status debug output in ansible, i think we should still patch that in on ours to save an extra debug cycle if there's still something else amiss19:25
*** jamesmcarthur has quit IRC19:26
*** PollyZ has quit IRC19:27
mordredjeblair: agree19:28
*** jamesmcarthur has joined #openstack-infra19:28
*** PollyZ has joined #openstack-infra19:29
*** salv-orlando has quit IRC19:30
*** yamamoto has joined #openstack-infra19:30
pabelangerjeblair: ack19:31
*** ayoung has joined #openstack-infra19:31
*** e0ne has quit IRC19:31
*** jamesmcarthur has quit IRC19:32
*** akshai has quit IRC19:32
EmilienMjeblair: can I abandon the ci jobs patches?19:32
pabelangersounds like we have some things to try now19:34
*** jamesmcarthur has joined #openstack-infra19:34
*** e0ne has joined #openstack-infra19:34
openstackgerritAndreas Florath proposed openstack/diskimage-builder: Refactor: block-device handling (local loop)
wznoinskis ci-watch broken? I can see at least one recent Jenkins comment that should cause the to generate19:34
mmedvedewznoinsk: it appears so19:35
*** gouthamr has joined #openstack-infra19:35
*** PollyZ has quit IRC19:35
mmedvedeApoorva: are you aware of the problems with ci-watch deployed at tintri?19:37
*** PollyZ has joined #openstack-infra19:37
clarkbmordred: responded19:38
clarkbyou can't comment on lines outside of the default diff context in github PRs19:38
clarkbits sort of annoying19:38
*** akshai has joined #openstack-infra19:39
*** yamamoto has quit IRC19:40
*** ddieterly[away] is now known as ddieterly19:44
*** tphummel has quit IRC19:44
*** rdx_matt has joined #openstack-infra19:44
fungithat seems like a shortcoming19:45
fungithough gerrit doesn't let you comment on lines in files not touched by a change too, which i've wanted at times19:46
*** acoles is now known as acoles_19:47
*** esikachev has joined #openstack-infra19:48
*** th3rt has quit IRC19:49
*** e0ne has quit IRC19:50
*** derekh has joined #openstack-infra19:50
*** salv-orlando has joined #openstack-infra19:52
*** e0ne has joined #openstack-infra19:54
derekhI see the patch to add rh2 to nodepool has landed, I also have a experimental job to start using the cloud,
jeblairEmilienM: i'm hoping we'll be ready for another attempt within a couple of hours19:55
jeblairEmilienM: so, whatever you think best :)19:56
derekhdoes that have to wait until nodepool is restarted or something so that nodes are available19:56
mmedvedenibalizer: do you know if your (use https when talking ot pypi) shoud also be ported to infra/puppet-pip?19:56
*** dtardivel has quit IRC19:57
pabelangerderekh: yes, we need to restart nodepool-builder19:57
pabelangerwhich will happen once our DIB uploads are finished for the day19:57
EmilienMjeblair: ok, /me nothing on the patch now19:57
derekhpabelanger: ok thanks19:57
*** jamesmcarthur has quit IRC19:58
nibalizermmedvede: we moved all the infra stuff to the openstack_pip provider in the infra pip module19:58
*** jamesmcarthur has joined #openstack-infra19:58
dmsimardWhere do I start looking if nodepool sees that there are queued jobs but doesn't create any VMs ? The node list is clean, the quotas are fine and the max-servers is not reached. There's no obvious errors in nodepool.log or debug.log.19:59
dmsimardIt'll create, like, at most 2 VMs for a job, but no more than that20:00
dmsimard(this is outside of openstack-infra)20:00
*** vhosakot has quit IRC20:02
*** vhosakot has joined #openstack-infra20:02
mmedvedenibalizer: ah, I see now. thanks20:03
fungidmsimard: one common cause for that is if nodepool has lost contact with zuul's gearman, since it won't see the demand any longer20:03
*** esikachev has quit IRC20:04
*** ayoung has quit IRC20:04
pabelangerYa, zuul should be reporting what the current demand is, and reacts according to that20:05
fungijroll: looking at the virtual sprint writeup, i see that "June 23 2016 00:00 to 04:00 UTC ... Audio quality issues inhibited communication. Session ended at 00:18 UTC." any details on that?20:05
*** haaj has joined #openstack-infra20:05
*** jamesmcarthur has quit IRC20:05
jrollfungi: hrm, I wasn't there, mat128|away would know but just took off for the weekend, maybe TheJulia20:05
*** jamesmcarthur has joined #openstack-infra20:05
fungiTheJulia: any recollection whether it was all participants or just some? sip clients, dial-in, both? no discernable pattern?20:06
*** abregman has joined #openstack-infra20:07
notmorganmordred: is it safe to assume: DB is empty -> create_all(), if db exists, assume we are "today" on nodepool?20:08
dmsimardfungi: thanks I'll look20:08
*** gouthamr has quit IRC20:08
notmorganjeblair: ^ cc.20:08
*** rajinir has joined #openstack-infra20:08
notmorganor do we want to say "if db isn't empty, do a drop/recreate and let nodepool *then* own it"?20:09
*** jcoufal_ has quit IRC20:09
fungineat, rdo is using a gerrit instance for their repos too? change count is low so must be a recent development. also... they have it setup for openid from github not the fedoraproject ipsilon instance. i wonder if that's something they're interested in working out20:10
*** gouthamr has joined #openstack-infra20:10
*** e0ne has quit IRC20:10
jeblairnotmorgan: i think assume today is okay20:10
fungidmsimard: ^ any idea about that?20:10
*** haaj has quit IRC20:11
dmsimardfungi: is a software factory instance ( )20:11
dmsimardtl;dr openstack-infra in a box20:11
fungidmsimard: neat-o20:11
dmsimardIt's a project started by Enovance a while back, it's a Red Hat project now20:12
notmorganjeblair: ok wfm, just checking before making that kind of assumption :)20:12
dmsimardtristanC is one of the engineers that works on it :)20:12
pabelangerfungi: look at asterisk logs, I see a codec mismatch during that time. Basically starts at 2016-06-23 00:07:55.723 could be related20:12
fungidmsimard: yeah, i'm aware of software factory, just didn't know rdo was using it20:12
dmsimardIt's fairly recent, like maybe 2 months top20:13
fungidmsimard: that's roughly what it looks like from mining some old change ids out of it20:13
*** banix has joined #openstack-infra20:13
dmsimardfungi: as you can tell from my question, we're still working out some kinks :)20:13
fungidmsimard: curious if there's any interest from the rdo devs in using as an openid provider with it, and whether anyone's looked into trying it for that and whether there are any gerrit-side gotchas there20:14
dmsimardI don't know, you guys want to switch from launchpad to fedora account system ?20:16
*** gyee has joined #openstack-infra20:16
pleia2well, the software driving the fedora account system (ipsilon)20:17
dmsimardI'm a user of software factory, not a developer so I don't know if they have something like that in the pipe20:17
dmsimardpleia2: aaaaaah, understood20:17
*** ayoung has joined #openstack-infra20:18
*** rdx_matt has quit IRC20:18
*** armax has joined #openstack-infra20:18
*** gordc has joined #openstack-infra20:20
mtreinishjeblair: have you ever hit this:
mtreinishjeblair: it looks like nova blows up sqlite20:21
mtreinishI truncated the output there it was very large20:22
mtreinishand just listed every nova change20:22
fungidmsimard: right, it's "just another openid/oidc" provider implementation, mainly curious since it's a red hat thing and software factory is a red hat thing and rdo is a red hat thing whether they were looking at switching from github login to that (or having both available)20:24
fungibut i also have no idea how much crossover there is between the rdo devs and the fedora project20:24
fungicould be the number of users on who also have fedora project logins is small20:25
jeblairmtreinish: well, drat20:25
*** PollyZ has quit IRC20:25
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Allow users to be filtered by openid
dmsimardfungi: I'll ask and get back to you20:26
*** vhosakot has quit IRC20:26
*** kgiusti has left #openstack-infra20:26
fungidmsimard: not super important, but if you happen to find out i'd be interested to know20:26
*** e0ne has joined #openstack-infra20:27
*** tonytan4ever has quit IRC20:27
*** zul has quit IRC20:27
*** rbradfor has quit IRC20:27
*** itisha has quit IRC20:27
*** rlandy has quit IRC20:27
*** TaylorHuang has quit IRC20:27
*** linuxmaniac_ has quit IRC20:27
*** fbo has quit IRC20:27
*** mkoderer__ has quit IRC20:27
*** sheeprine has quit IRC20:27
*** AJaeger has quit IRC20:27
*** notmorgan has quit IRC20:27
*** melwitt has quit IRC20:27
*** timsim has quit IRC20:27
*** erikwilson has quit IRC20:27
*** lewo has quit IRC20:27
*** patricku has quit IRC20:27
*** NobodyCam has quit IRC20:27
*** Daviey has quit IRC20:27
*** Ng has quit IRC20:27
*** freerunner has quit IRC20:27
*** timrc_ has quit IRC20:27
*** mika has quit IRC20:27
*** Anticimex has quit IRC20:27
*** igorbelikov has quit IRC20:27
*** ericksonsantos has quit IRC20:27
*** smcginnis has quit IRC20:27
*** trown has quit IRC20:27
*** peterlisak has quit IRC20:27
*** liusheng has quit IRC20:27
*** dimtruck has quit IRC20:27
*** Kiall has quit IRC20:27
*** ZZelle has quit IRC20:27
*** zigo has quit IRC20:27
*** zaro has quit IRC20:27
*** warpc has quit IRC20:27
*** lindsayk has quit IRC20:27
*** patrickeast has quit IRC20:27
*** ogelbukh has quit IRC20:27
*** zxiiro has quit IRC20:27
*** bryan_att has quit IRC20:27
*** alexm2 has quit IRC20:27
*** DuncanT has quit IRC20:27
*** jbryce has quit IRC20:27
*** mgagne has quit IRC20:27
*** mnencia has quit IRC20:27
*** BobBall_AWOL has quit IRC20:27
*** cburgess has quit IRC20:27
*** onovy has quit IRC20:27
*** morgabra has quit IRC20:27
*** bastelfreak has quit IRC20:27
*** senk_ has quit IRC20:27
*** phschwartz has quit IRC20:27
*** Shrews has quit IRC20:27
*** mat128|away has quit IRC20:27
*** _fortis has quit IRC20:27
*** jvrbanac has quit IRC20:27
*** dutsmoc has quit IRC20:27
*** odyssey4me has quit IRC20:27
*** lbragstad has quit IRC20:27
*** isviridoz has quit IRC20:27
*** j_king has quit IRC20:27
*** jamielennox has quit IRC20:27
*** mattoliverau has quit IRC20:27
*** mcarden has quit IRC20:27
*** puiterwijk has quit IRC20:27
*** mat128 has joined #openstack-infra20:27
*** j_king has joined #openstack-infra20:27
*** matt6434 has joined #openstack-infra20:27
*** lbragstad_ has joined #openstack-infra20:27
*** mika has joined #openstack-infra20:27
*** Kiall has joined #openstack-infra20:27
*** BobBall_AWOL has joined #openstack-infra20:27
*** mika has quit IRC20:27
*** mika has joined #openstack-infra20:27
*** onovy has joined #openstack-infra20:27
*** erikmwilson has joined #openstack-infra20:27
*** jvrbanac has joined #openstack-infra20:27
*** phschwartz has joined #openstack-infra20:27
*** Anticimex has joined #openstack-infra20:27
*** sheeprine has joined #openstack-infra20:27
*** smcginnis has joined #openstack-infra20:27
*** peterlisak has joined #openstack-infra20:27
*** notmorgan has joined #openstack-infra20:27
*** notmorgan has quit IRC20:27
*** notmorgan has joined #openstack-infra20:27
*** TaylorHuang has joined #openstack-infra20:27
*** alexm2 has joined #openstack-infra20:27
*** timrc has joined #openstack-infra20:27
*** rlandy has joined #openstack-infra20:27
*** AJaeger has joined #openstack-infra20:27
*** cburgess has joined #openstack-infra20:27
*** ZZelle has joined #openstack-infra20:27
*** ericksonsantos has joined #openstack-infra20:28
*** liusheng has joined #openstack-infra20:28
*** HeOS has joined #openstack-infra20:28
*** morgabra has joined #openstack-infra20:28
*** tonytan4ever has joined #openstack-infra20:28
*** rbradfor has joined #openstack-infra20:28
*** melwitt has joined #openstack-infra20:28
*** senk has joined #openstack-infra20:28
*** freerunner has joined #openstack-infra20:28
*** mnencia has joined #openstack-infra20:28
*** bastelfreak has joined #openstack-infra20:28
*** zul has joined #openstack-infra20:28
*** mkoderer__ has joined #openstack-infra20:28
*** odyssey4me has joined #openstack-infra20:28
*** melwitt is now known as Guest1319120:28
*** puiterwijk has joined #openstack-infra20:28
*** mcarden has joined #openstack-infra20:28
*** trown has joined #openstack-infra20:28
jeblairmtreinish: that was added in Iae6d789409b88f4e3cb35ac9e4f839f8e4d79a9720:28
jeblairmtreinish: i think we can take a different approach there20:28
jeblairmtreinish: i'll hack it up real quick20:28
*** _fortis has joined #openstack-infra20:29
*** zigo has joined #openstack-infra20:29
*** Ng has joined #openstack-infra20:29
*** lindsayk has joined #openstack-infra20:29
*** Shrews has joined #openstack-infra20:29
*** linuxmaniac has joined #openstack-infra20:29
*** woodster_ has quit IRC20:29
*** warpc has joined #openstack-infra20:30
*** ewindisch has quit IRC20:30
*** electrical has quit IRC20:30
*** simonmcc has quit IRC20:30
*** fkautz has quit IRC20:30
*** ogelbukh has joined #openstack-infra20:30
*** zxiiro has joined #openstack-infra20:30
*** NobodyCam has joined #openstack-infra20:30
*** mgagne has joined #openstack-infra20:30
*** igorbelikov has joined #openstack-infra20:30
*** mgagne is now known as Guest2045420:30
*** bryan_att has joined #openstack-infra20:30
mtreinishjeblair: ok cool, thanks20:30
*** fbo has joined #openstack-infra20:30
*** dimtruck has joined #openstack-infra20:30
*** timsim has joined #openstack-infra20:30
*** tonytan4ever has quit IRC20:30
*** mwhahaha has quit IRC20:30
*** dougwig has quit IRC20:30
jeblairmtreinish: apparently the limit is 999 :)20:31
*** isviridov_away has joined #openstack-infra20:31
*** tpsilva has quit IRC20:31
*** mattfarina has quit IRC20:31
*** patrickeast has joined #openstack-infra20:31
mtreinishjeblair: haha, I was expecting a power of 220:31
*** itisha has joined #openstack-infra20:31
*** Guest13191 is now known as melwitt20:31
*** DuncanT has joined #openstack-infra20:31
*** comstud has joined #openstack-infra20:32
*** zaro has joined #openstack-infra20:32
*** patricku has joined #openstack-infra20:32
*** mwhahaha has joined #openstack-infra20:33
fungihello arbitrary limit?20:33
*** tpsilva has joined #openstack-infra20:34
*** Daviey has joined #openstack-infra20:34
*** lbragstad_ is now known as lbragstad20:35
*** rfolco has quit IRC20:35
*** teran has quit IRC20:35
mordredfungi: 999 is all you should ever need for anything20:35
*** rfolco has joined #openstack-infra20:35
*** _fortis has quit IRC20:35
*** tonytan4ever has joined #openstack-infra20:36
*** simonmcc has joined #openstack-infra20:36
*** electrical has joined #openstack-infra20:36
rbrndtfound an error in openstack/requirements that is blocking CI jobs for at least monasca and magnum20:37
fungirbrndt: ooh, an error?20:38
rbrndtfix has been posted, could someone review it please?
*** jamielennox has joined #openstack-infra20:38
fungirbrndt: also the newly formed requirements team hangs out in #openstack-requirements now according to
rbrndtoh, thanks. didn't know about that channel20:39
fungiyeah, we can finally stop telling people *shrug* when they ask how to get attention from requirements reviewers20:39
*** lewo has joined #openstack-infra20:40
*** jamesmca_ has joined #openstack-infra20:40
mordredfungi: what if we want to tell them *shrug* anyway?20:41
fungii think that's still entirely within the realm of possibility20:41
*** ewindisch has joined #openstack-infra20:41
*** fkautz has joined #openstack-infra20:42
fungipabelanger: i incorporated your thin=>true request into 333065 and i have it passing all its jobs now as well, if you're interested in taking another peek20:42
*** mattfarina has joined #openstack-infra20:42
pabelangerfungi: sure! and thanks20:42
fungialso it comes with a bonus history lesson in the form of a depends-on link20:43
*** teran has joined #openstack-infra20:43
*** gluytium has joined #openstack-infra20:43
*** gouthamr has quit IRC20:44
*** jamesmcarthur has quit IRC20:44
*** vhosakot has joined #openstack-infra20:44
*** kzaitsev_mb has quit IRC20:45
Shrewsmordred: jeblair: fyi, looking at adding use of atomic_move to the async core modules now20:46
jeblairShrews: woot!  i'll work on prepping things so we can deploy that20:47
*** weshay has quit IRC20:47
*** dougwig has joined #openstack-infra20:47
*** jbryce has joined #openstack-infra20:48
*** ayoung has quit IRC20:51
*** e0ne has quit IRC20:51
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Allow users to be filtered by openid and email
*** ajmiller has joined #openstack-infra20:52
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Allow users to be filtered by openid and email
*** hashar has quit IRC20:53
*** _ari_ has quit IRC20:54
*** ajmiller_ has quit IRC20:55
*** ffrank has joined #openstack-infra20:55
openstackgerritJames E. Blair proposed openstack/gertty: Fix "too many SQL variables" error
jeblairmtreinish: your choice of that change or  should fix the problem20:56
mtreinishjeblair: ok, I'll give them both a try20:57
jeblairmtreinish: (they are mutually exclusive, but i don't know which is better)20:58
jeblair(i can get worst-case timing for mine easily, the other one is harder)20:58
*** kzaitsev_mb has joined #openstack-infra20:58
mtreinishjeblair: sure, I meant try them independently20:59
*** ajmiller has quit IRC20:59
*** eharney has quit IRC21:00
*** akshai has quit IRC21:01
*** woodster_ has joined #openstack-infra21:01
jeblairmordred: what do i need to do if i'm pip installing ansible from a working directory?21:01
jeblairmordred: do i need to install multiple repos, or just 'ansible'?21:01
mordredjeblair: if you submodule update to get the submodules, it should just work21:02
clarkbjeblair: you have to init the subrepos21:02
*** rfolco has quit IRC21:03
*** inc0 has quit IRC21:04
TheJuliafungi: it largely seemed like sip clients, I don't think anyone who dialed in actually mentioned any issues. :(21:05
*** amitgandhinz has quit IRC21:06
*** haaj has joined #openstack-infra21:07
ianwjeblair: tbh i don't see why the "zuul" user is any different to all the sysadmin users, and the jenkins user really.  fundamentally it depends on info in heira (the pub key coming from pupept) so we're just making work for ourselves communicating that out-of-band21:07
ianwi don't really care if someone wants to go and re-write all the system-config puppet in ansible, or whatever21:07
jeblairianw: the jenkins user will be removed.  and perhaps we should stop installing our own keys as well.21:08
ianwbut just doing that for the sake of doing that seems like there's more fun things to do21:08
*** gouthamr has joined #openstack-infra21:08
*** rhallisey has quit IRC21:08
*** akshai has joined #openstack-infra21:08
jeblairianw: it's not for the sake of doing it, it's for the sake of extricating the disk images we use for running tests from our long running systems, which at this point, have very little in common, yet the fact that they are so closely tied makes things very difficult21:08
fungiTheJulia: thanks. pabelanger noticed a codec mismatch in the logs right around that time too21:09
fungiso that would potentially make sense21:09
jeblairianw: most of all of this dates from when our test machines *were* long running systems21:10
ianwjeblair: so how do you want dib to get the public key for the zuul user?21:10
*** thorst_away has quit IRC21:10
njohnstonSo in discussions yesterday I think it was ascertained that accessbot is not approaching the max-channels limit.  If so, can be approved so I can get #openstack-fwaas logged?21:10
jeblairianw: i do not have an immediate solution, i merely argue that we should proceed in developing one :)21:10
*** trown is now known as trown|outtypewww21:10
*** psilvad_ has quit IRC21:11
*** gouthamr_ has joined #openstack-infra21:11
fungiianw: we can put that just about anywhere. public keys are public. we can bake it into public configuration/parameters if it helps21:11
clarkbianw: I would have devuser write it and an easy way to do that is put the public key in nodepool.yaml with the env vars for the images21:11
*** ajmiller has joined #openstack-infra21:11
*** haaj has quit IRC21:12
funginjohnston: right, accessbot will actually never hit that limit because it doesn't join any channels, just queries chanserv21:12
ianwclarkb: devuser is the wrong thing really, it was designed to stick one key in for a developer -- comments in
ianwclarkb: the env var into nodepool.yaml was what i suggested in the comment to.  i guess i can do that21:12
njohnstonfungi: Is my information correct, then, that accessbot is the mechanism by which channels get logged to ?21:12
clarkbI am not sure I agree about the devuser element21:12
clarkbit does exactly what we need, create a user and give it a key21:13
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Implement gate-puppet-openstack-spec-helper-lint
funginjohnston: no, accessbot checks whether we have control over channels. meetbot does teh logging (i agree, teh name is confusing)21:13
*** thorst_away has joined #openstack-infra21:13
*** ajmiller_ has joined #openstack-infra21:13
ianwclarkb: well, it creates one user, and overwrites the developer if they're trying to use it (e.g. me) and it grabs the keys from their ~/.ssh/authorized_keys ... i mean why can "jenkins" log in as "nodepool" on nodepool now...21:13
*** gouthamr has quit IRC21:14
*** yaume has quit IRC21:14
clarkbianw: what do you mean by overwrites the developer? you tell it "make a zuul user and give it this public key"21:14
njohnstonok, so it sounds like accessbot should still be there, but meetbot is the important thing.  All right, I'll put in a change to add the channel to that config as well.21:14
clarkbianw: I use it all the time21:14
*** akshai has quit IRC21:14
clarkbworks fine21:14
*** thorst_a_ has joined #openstack-infra21:15
*** shamail has left #openstack-infra21:15
funginjohnston: yeah, and you have it in the list for statusbot in 329570, you can just update that proposed change. the meetbot list is in the same file21:15
*** rdx_matt has joined #openstack-infra21:16
*** maishsk_ has joined #openstack-infra21:16
ianwclarkb: heh, you have your keys in the image from puppet and probably log in to the final image as yourself.  i build and use a devuser to login with my key, but the openstack element overwrites my devuser to "zuul"21:16
jeblairhrm.  i just installed ansible devel on zl01 and it seems to be unable to connect to any host21:16
clarkbianw: I am saying I build dib images all the time using devuser, I don't use our puppet stuff21:16
funginjohnston: statusbot is the thing that echoes urgent infrastructure announcements into the channel, and also implements the #success logging21:16
clarkbianw: I do not have any issues using devuser at all21:16
njohnstonfungi: Is there anything that needs to change in project-config?21:16
*** ajmiller has quit IRC21:17
*** ldnunes has quit IRC21:17
ianwclarkb: i am not saying devuser doesn't work.  i am saying that using devuser to deploy the zuul user isn't the right thing, because it *overrides* any devuser a developer might have setup21:17
funginjohnston: nope, looks like you already added your channel to so that's sufficient for project-config21:17
clarkbianw: you are saying that you can only have a single devuser?21:17
*** thorst_away has quit IRC21:17
ianwclarkb: yes21:17
*** maishsk has quit IRC21:17
*** maishsk_ is now known as maishsk21:17
clarkbianw: seems like that would be easy to address in dib rather than deciding not to do it at all?21:17
funginjohnston: so just update 329570 to also add your channel to the meetbot list and you're all set21:17
clarkbianw: anyways what jeblair is saying is to have a single user, remove any othre special users that map to humans21:18
clarkbianw: then for you or I working locally we would just tell zuul user to auth with a different public key21:18
ianwclarkb: where does dib get the keys?21:18
clarkbthen the only difference between production and dev is what key is in place21:18
openstackgerritNate Johnston proposed openstack-infra/system-config: Add #openstack-fwaas to accessbot channels
clarkbianw: so this shouldn't be an issue for our use case21:18
njohnstonfungi: Thanks, updated! ^^21:18
jeblairi have downgraded ansible to on zl0121:19
clarkbianw: you could enumerate per user vars if you wanted to keep a similar interface21:19
jeblairit seems to be doing things again21:19
clarkbianw: I don't have the answers off the top of my head but it is certainly doable21:19
*** thorst_a_ has quit IRC21:19
funginjohnston: also your commit message is a little off. you're adding to statusbot's and meetbot's channel lists, not accessbot's (you already took care of that in 326821).21:20
*** aysyd has quit IRC21:20
openstackgerritNate Johnston proposed openstack-infra/system-config: Add #openstack-fwaas to statusbot/meetbot
*** vhosakot has quit IRC21:21
njohnstonfungi: Thanks, fixed.21:21
*** robot has quit IRC21:22
openstackgerritRick Aulino proposed openstack-infra/project-config: Upgrade experimental searchlight ES 2.3.2
clarkbanother possible option would be to have the user element work like the pkgmap stuff and colate a bunch of user spec files21:22
ianwclarkb: well it's software, anything is doable.  but it seems like the solution is to take data from puppet, write it into the nodepool config file, pass it through environment variables to a hand-written dib element that adds users ... to avoid using puppet directly which has the key info and someone else has already debugged the orchestration parts of adding users instead of my crappy shell script21:22
Shrewsjeblair: oh? what version was it at?21:23
jeblairShrews: tip of devel21:23
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Implement gate-puppet-openstack-spec-helper-syntax
Shrewsjeblair: and what b0rked with the new version?21:23
jeblairShrews: i'm seeing if i can reproduce locally21:23
jeblairShrews: it just said unable to connect for every host21:23
*** ajmiller_ has quit IRC21:23
Shrewsjeblair: oh, that's a new feature that's coming in 2.2. ansible will only run locally21:23
clarkbianw: I think what I am suggesting is rather than having multiple user elemnts have one that works21:24
Shrewsbecause networking is hard21:24
clarkbthe current existing element is called devuser21:24
clarkbanyways I don't relaly have a dog in this fight, just wanted to point out that devuser works well, dib users know to use devuser, so seems like a good fit21:25
jeblairShrews:  :)21:25
TheJuliafungi: I started trying other codecs to lower bandwidth utilization on my end, it was still choppy for mat128 last night regardless of the codec.  My client showed 0-4% loss on inbound from the sip server, but nothing else21:25
pabelangerianw: I'm working on an ansible-playbook for the zuul-worker element.  I should have something ready for tomorrow21:25
Shrewsjeblair: so, here is a possibly atomic-move-enabled version of async_wrapper:
fungiTheJulia: oh, trying additional codecs might account for the error pabelanger saw. if so, not a result of the issue but rather just a result of you trying workarounds for the issue21:26
Shrewsjeblair: i'm not sure how to test that other than just try it21:26
jeblairShrews, pabelanger: i think i can reproduce locally -- but there's a slight difference -- i get an ssh-askpass dialog asking if i want to trust the worker's host key.  i say 'yes' and it fails with fatal: [node]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh.", "unreachable": true}21:26
fungiTheJulia: in that case, could have been network issues in rackspace or something21:26
pabelangerfungi: TheJulia: Yes, I see seen gsm as a codec for sure. In fact, I am not sure which low-bandwidth codec we have enabled21:26
ianwpabelanger: ok, well that's fine ... still how is it going to get the keys?21:26
jeblairShrews, pabelanger: so i assume there's something up with our known_hosts enforcement21:26
Shrewsjeblair: i'm going to try that with a simple playbook first21:27
*** esberglu has quit IRC21:27
pabelangerianw: in this case, key should be stored in a playbook variable21:27
TheJuliafungi: that or between rackspace and centurylink last night, I know on monday there were a few people reporting choppy audio on the stream at times and I think there were two major carriers with significant packet loss21:27
fungiTheJulia: makes sense. internet gonna internet21:27
pabelangerianw: something like:
*** jamesmca_ has quit IRC21:27
TheJuliafungi: exactly21:28
*** ajmiller has joined #openstack-infra21:28
pabelangerianw: or, we can read it in from a file21:28
*** ddieterly is now known as ddieterly[away]21:28
clarkbsip is udp too right?21:28
clarkbya so any unreliability in the path will be very noticeable21:29
pabelangerwe can do TCP21:29
ianwpabelanger: well storing the public key in the element seems wrong, so yeah, it has to read from an external file ... which is provided by puppet?21:29
pabelangerbut RTP will still be over udp21:29
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Allow users to be filtered by openid and email
pabelangerianw: where would you like the key stored, that's likely the easy thing for me to ask21:30
ianwpabelanger: well, as  suggests I think that leaving it in puppet is fine :)21:31
ianwit seems like the right tool to be adding at user at this point in time21:32
pabelangerjeblair: Just catching up, there is problems with ssh_known_host?21:32
jeblairianw: did this conversation just go in circles?21:32
fungiianw: i agree that leaving it in puppet for now is fine, though we also want to stop running puppet in our image setup at some point so we can stop preinstalling it21:32
jeblairwe agreed like 2 years ago to stop doing this21:33
pabelangerianw: I think the main sticking point, is we want to stop using puppet in DIBs. That was my initial reason for using devuser elemtn21:33
fungiand continuing to put the new account in puppet seems like prolonging the situation21:33
fungisince we still want to move it out of puppet21:34
pabelangerat the same token, I wouldn't like to see us using DIB elements to do puppet things or bash scripts. Which leave an ansible-playbook, which seems to be okay atm21:34
jeblairelements can inherit other elements, right?21:34
ianwjeblair: not really?  they can require21:34
ianwyes, i think my main issue is rewriting stuff like "add a user" in crap shell scripts in dib21:35
openstackgerritStephanie Miller proposed openstack/diskimage-builder: Make svc-map support overridden elements
ianwstuff that puppet/ansible/etc has had figured out just fine21:35
jeblairi can't keep repeating myself21:36
jeblairi want to help find a solution here21:36
jeblairthere's a world of possibilities that could satisfy all of us21:37
roaetis there a command for recheck but for the gate?21:37
clarkbroaet: its the same command, our zuul config requires that check pass before going into the gate so yo urecheck, it goes through check, if it has approval it then goes to gate if check apsses21:37
jeblairianw: but we need to at least agree on the premise -- that we should not run system-config puppet in dib21:38
*** thorst_away has joined #openstack-infra21:38
*** sdake has quit IRC21:38
roaetclarkb: having a weird thing where the recheck passes but the gate fails21:38
roaetbut it is queued21:39
roaetso maybe it'll fail again21:39
ianwjeblair: i feel like we might be at the point that a user-deployment spec might be useful ... cause it's going to touch nodepool, system-config and dib21:39
clarkbroaet: python27 job faile,d you can look at the logs to try and determine why21:39
jeblairianw: how is this possibly that hard?21:39
roaetclarkb: aye, but it didn't fail on the validate. just weird21:39
*** abregman has quit IRC21:40
*** rbrndt has quit IRC21:40
clarkbroaet: you likely have flaky tests then?21:40
roaetclarkb: concur21:40
*** rbrndt_ has joined #openstack-infra21:41
jeblairianw: do you have any suggestions about how someone who is not us -- so they have no system-config puppet -- should configure nodepool to create an image with dib that has a user whose authorized_keys corresponds to the key that nodepool and zuul will use?21:41
*** vhosakot has joined #openstack-infra21:41
*** maishsk_ has joined #openstack-infra21:41
fungiroaet: seems to have worked that time. it's merged now21:42
roaetfungi: aye. I'm telling the dev to make sure his test is deterministic21:42
* clarkb wonders where that jira lives but probably doesn't want to know21:42
*** sdake has joined #openstack-infra21:42
*** thorst_away has quit IRC21:43
*** claudiub has quit IRC21:43
*** maishsk has quit IRC21:43
*** maishsk_ is now known as maishsk21:43
*** thorst_away has joined #openstack-infra21:43
roaetclarkb: heh, ya21:43
ianwjeblair: i would think they'd get the key from meta-data (configdrive in the glean case)21:43
ianwrather than baking it in21:43
jeblairianw: how does zuul use it?21:43
*** armax has left #openstack-infra21:43
jeblairianw: and does glean create users?21:44
*** matt-borland has quit IRC21:44
fungiglean does not add users afaik21:44
fungithat's a cloud-init-ism21:44
ianwglean does not, cloud-init does21:44
*** njohnston is now known as njohnston|afk21:44
fungiand part of why we bake credentials into our image builds21:44
clarkbthe other part is that metadata services have been unreliable21:45
jeblairright, so perhaps we can agree that the goal here is still to bake a user with associated information into our image builds?21:45
clarkbconfig drive seems to be much more reliable21:45
pabelangerjeblair: I think so, it seems to have worked well for us so far21:45
jeblairpabelanger: the converse is also true -- not baking things into our images has worked very poorly for is.21:46
zigoWhat is the ready-script: that I can see at ?21:46
*** mriedem has quit IRC21:46
clarkbzigo: it is a script that nodepool runs on every test node in order to make it "ready"21:46
jeblairzigo: it is found here:
pabelangerjeblair: right21:47
clarkbzigo: once that script completes successfully nodepool assigns the instance to a "target" so that target can run jobs21:47
*** thorst_away has quit IRC21:47
*** esberglu has joined #openstack-infra21:48
zigoI was hoping to find some convenient stuff to configure mirrors depending on the provider ...21:48
*** rdx_matt has quit IRC21:48
zigoI know that at least OVH has a Debian mirror.21:48
zigoWould it make sense to make that script decide one or another hostname for the mrrors, depending on the hostname?21:48
jeblairianw: you seem to have strong opinions on this, and you know the space.  so i'm hoping if we can agree on the premise, you can help come up with a solution.  so what i wrote above is what we should be developing for:21:49
pabelangerzigo: we've had issues with provider mirrors in the past. Thats the reason we configure our own AFS mirrors21:49
clarkbzigo: you can actually query the provider and region directly, they are in /etc/nodepool somewhere. That said not sure how people feel about that, is the debian mirror we are using unreliable?21:49
clarkbalso what pabelanger said21:50
ianwjeblair: my opinion is mostly around avoiding turning dib into a bespoke and completely crap version of puppet21:50
jeblairianw, pabelanger: the problem statement is basically: someone with no system-config puppet repo should be able to configure nodepool to create an image with dib that has a user whose authorized_keys corresponds to the key that nodepool and zuul will use.21:50
zigoclarkb: The Debian mirror is the, so it's just a 302 redirect type of mirror.21:50
zigoThat works well, it's better than a DNS redirector, because more dynamic.21:51
clarkboh I thought we stopped doing that and hard coded it to a specific one21:51
zigoBut it adds the cost of the redirection.21:51
clarkbsince httpredir was failing us21:51
fungiclarkb: i think zigo is saying the mirror he _wants_ to use is that way21:51
funginot ours21:51
clarkbzigo: ya for a time httpredir would either fail to redirect or redirect to mirrors that were down21:51
zigoWell, has the redirector.21:51
jeblairian, pabelanger: i don't really care how that happens.  if the dib best practice is to have an element that runs puppet or ansible, that's fine*.  it needs to be completely self-contained though, and not depend on any external infrastructure -- it especially should not assume anything about the configuration management of the nodepool server.21:51
jeblairianw: ^ sorry dropped the w21:51
clarkband the plan I thought was to point at a specific mirror. But then I moved and stopped computering for a while and am still catching up21:52
fungiclarkb: zigo: oh, weird. i guess we didn't set up an actual debian package mirror yet?21:52
clarkbjeblair: ianw pabelanger keep in mind that we also need to configure dns and firewalls which puppet is doing for those instances21:53
jeblairianw, pabelanger: * (i will say that if dib best practice to create a user is to invoke puppet, i may be underwhelmed)21:53
zigofungi: Right.21:53
clarkb(this may or may not influence the decision here)21:53
zigofungi: That's fine for the moment though.21:53
jeblairclarkb: yes, there is more work to be done.21:53
zigoI had no issue with the redirector yet.21:53
jeblairclarkb: however, pabelanger took a small step in the direction we have wanted to move for a long time21:53
clarkbzigo: we had a lot of issues with it doing image builds21:53
zigoOh ok.21:54
jeblairclarkb: and i would like him to be able to continue21:54
*** tonytan4ever has quit IRC21:54
*** dingyichen has joined #openstack-infra21:54
jeblairclarkb: moreover *we* need to configure dns and firewalls, but not everyone using nodepool and zuul do.  *everyone* using nodepool and zuul needs a user.21:55
ianwclarkb: yeah, which is why i brought this up because turning these from puppet modules into bespoke scripts seems like a regression ... i mean puppet was supposed to be the end of bespoke scripts21:55
clarkbjeblair: ya21:55
pabelangerRight, I think having a DIB element to run ansible (or puppet) could be the first step to address some of these issue.  The location of say roles or manifests would then not live in system-config21:55
jeblairpabelanger: that is an option21:56
jeblairpabelanger: i will look upon dib very poorly if that is the best option21:56
greghaynesjeblair: I assure you that there are no dib best practices which involve invoking puppet, although some people do that21:56
pabelangerianw: right, personally I don't want to write bash scripts for any of this.21:56
ianwpabelanger / jeblair : using ansible to do things like deploy users and setup firewalls seems like a step in the right direction, over making it up by hand21:56
fungibut i also don't want us dragging in the openstack-infra/system-config openstack_project puppet module to do our image builds21:57
ianwi do feel like we've *already* solved this in puppet, and have to re-write it in ansible, but ... ok21:57
greghaynesI get the impression that the argument against using dib devuser and friends is really a tool preference argument...21:57
*** baoli has quit IRC21:57
jeblairianw: i will be quite happy for openstack to congfigure its firewalls via either ansible or puppet if it wants (as long as it isn't system-config).  the user thing has to be solved without thinking about openstack's peculiarities.21:57
fungithe argument against using seems to primarily be that someone might also want to have their own devuser element and use nodepool21:57
pabelangergreghaynes: I think that is part of it21:57
fungigreghaynes: ^21:57
*** roxanaghe has joined #openstack-infra21:58
clarkbfungi: yes and we can have devuser support that21:58
clarkbwould not be hard21:58
fungiand per dib there can be only one devuser21:58
ianwfungi: and that it's grabbing the users ~/.ssh/authorized_keys, rather than deploying from a given list21:58
pabelangergreghaynes: For me, the line is grey. Where do image builds stop and your cfgmgt tool of choice starts21:58
greghaynesfungi: ah. So for that case I'd just make your own element. We could extend devuser to be smart, but makng an element that makes a speical user is 1 dir and 1 file in that dir21:58
*** pilgrimstack has joined #openstack-infra21:58
greghaynesso its hard to argue that we need to make some fancy system when that is the workaround21:58
ianwgreghaynes: well, this "add user" element really needs to take key data from puppet21:59
ianwand handle adding multiple users21:59
ianwso it probably wants to parse a json type config file21:59
fungican't we just put the key in a variable? why does it need to come "from puppet"?21:59
jeblairianw: it does not need to take key data from puppet21:59
fungiit's just a blob of entirely public data we want put on disk21:59
greghaynesalso, adding multiple users is just adduser foo; adduser bar21:59
jeblairianw: the key can even live *in* the element22:00
ianwfungi / jeblair: but then when we want to change it, it's baked in all these different places22:00
jeblairianw: *we* will put it in puppet certainly22:00
ianwthis seems like a regression from having the data kept in once place and handed out22:00
*** ddieterly[away] is now known as ddieterly22:00
fungithat's more of an argument for ldap then22:01
ianwjeblair: right -- *we* need to get it from puppet into a format that dib understands.  so we need to write the dib element with some interface ... say it reads a json file with a list of users and keys22:01
greghaynesI don't know all the context, but you can use an env var (as mentioned) or extra-data.d to run commands outside the chroot that grab the data. I hope you can either input the data or retrieve it with command though otherwise I'm not sure its a dib design issue.22:02
pabelangerI don't know, talking about updating devuser element to add more support for user management doesn't seem to be the best use of time, IMO22:02
fungiseems like the concern over that is having the public key stored in more than one git repo22:02
jeblairgraghaynes: the problem statement is basically: someone with no system-config puppet repo should be able to configure nodepool to create an image with dib that has a user whose authorized_keys corresponds to the key that nodepool and zuul will use.22:03
fungii agree needing to replace that key in multiple repos or even multiple files in one repo is an annoyance. not sure how much of one and whether it's significant enough to avoid simplicity22:03
pabelangerjeblair: right, but duplicated data isn't really a new problem for us today22:03
jeblairfungi: that's an annoyance to be solved at a higher level with configuration management.  for us, that's puppet (for now).22:04
*** dimtruck is now known as zz_dimtruck22:04
ianwwell i'm also thinking about the keys of all the sysadmin users we put in there today22:04
jeblairianw: discard them22:04
greghaynesjeblair: ah. So we template out our nodepool config, correct? We could inject the key via env var that way.22:04
clarkbgreghaynes: I don't even think we need to template it, just hard code it in there22:04
greghaynesor that22:04
jeblairianw: that's "extra stuff that openstack does" and we do not need to worry about it here.  we can continue doing that in puppet as long as we want.  and some day we can stop.22:05
fungithe keys we put on those images today are also keys which have access to the zuul launchers. we can hop through to add our keys as needed22:05
clarkbsince the reast of the config is pretty us specific and so on22:05
greghaynesthat was my way to work around duplication22:05
*** e0ne has joined #openstack-infra22:05
*** gluytium has quit IRC22:05
pabelangerthis is how we are doing it today, with our zuul-worker element:
*** e0ne has quit IRC22:05
*** robot has joined #openstack-infra22:05
pabelangerwe're getting the data from the local nodepool.o.o filesystem with devuser22:06
*** ajmiller has quit IRC22:07
*** xyang1 has quit IRC22:07
openstackgerritAnita Kuno proposed openstack-infra/project-config: Remove gate-storyboard-js-integration to experimental
openstackgerritAnita Kuno proposed openstack-infra/project-config: Move gate-storyboard-js-integration to experimental
ianwpabelanger/jeblair: well, discarding a more generic deployment mechanism, we might as well just hack in a one-off environment variable for zuul-worker to consume and do an "adduser" and forget about this22:08
*** denisra has joined #openstack-infra22:09
ianwhack the variable into nodepool.conf ... populated by puppet (for us)22:09
pabelangerisn't that basically what devuser element does today?22:10
*** signed8bit is now known as signed8bit_Zzz22:11
*** derekh has quit IRC22:11
ianwpabelanger: almost ... let's make something that doesn't conflict with existing users of devuser, and only deploys the one key, instead of grabbing anything in /home/nodepool/.ssh/authorized_keys22:11
jeblairokay.  we'd probably actually need to have the nodepool config point to a file on disk instead of template it in (because we don't and can't template stuff into that file since it's in project-config)22:11
jeblairbut the actual key can be put into place by system-config via puppet+hiera22:12
*** davidlenwell has quit IRC22:12
Shrewsjeblair: question... is it possible that the repeatedly failing jobs have more output than other jobs?22:12
clarkbShrews: in the case of the puppet job at least I think that is the case, since it spews a lot of verbose puppet things then a lot of verbose tempest things whereas our normal teempest just are relatively short devstack related stuff then tempest22:13
*** wznoinsk is now known as wznoinsk_22:13
clarkbShrews: but I would have to double check against a sample to be sure22:13
ianwjeblair / pabelanger : ok, i can get something going in that direction.  pabelanger -- your ansible idea is much more the type of thing if we're going to deploy all of the sysadmin users, and do firewall setup, etc etc, so worth fleshing out22:13
jeblairShrews: but ansible module output? no -- zuul_runner returns a very simple fixed size structure22:13
*** davidlenwell has joined #openstack-infra22:13
jeblairianw: ++22:13
jeblairclarkb:  ^^22:13
clarkbI still say devuser is what I would do...22:14
Shrewsjeblair: not ansible module output, but the job itself (the stdout/stderr)22:14
*** cody-somerville has quit IRC22:14
clarkbbut don't feel so stronly to demand it22:14
jeblairShrews: okay, but ansible doesn't see that, it goes to a file in /tmp22:14
pabelangerjeblair: right. And that goes back to my comment about the grey line22:14
pabelangerI think it would be good to list out our minimal things we need to do to get nodepool / DIB launched.22:15
pabelangerAs I understand it now, it is create zuul user add SSH key22:15
jeblairShrews: (remember, we're not running 'shell' which does return stdout; we're running zuul_runner which essentially only returns success/failure)22:15
greghaynesre: devuser, if you only need 1 user that works. Its basically the 'hello, world!' of element's which is why I mentioned making another - if it isnt going to just work then its probably just as easy to make your own.22:15
Shrewsjeblair: right. k22:15
ianwgreghaynes: yep, just do the adduser by hand in zuul-worker.  devuser is fine for exactly what it says, a developer adding a user to just quickly log in and test22:16
*** ajmiller has joined #openstack-infra22:16
jeblairShrews, pabelanger: also, i hand-reverted the known_hosts stuff, and i'm still getting the connection errors.  -vvv and debug=1 are giving me nothing useful to go on22:16
*** edmondsw has quit IRC22:16
pabelangerjeblair: sure, I am fine with the revert. When you have spare time I would be interested in looking at the logs if possible22:17
Shrewsjeblair: well, i'd never suggest running from HEAD anyway22:17
clarkbgreghaynes: ya we onlt need one user "zuul"22:17
*** matt6434 is now known as mattoliverau22:18
jeblairpabelanger: no -- i'm not reverting it -- i'm trying to see what's causing ansible HEAD to fail22:18
jeblairShrews: okay, should we abandon this and just apply your patch to
Shrewsjeblair: my patch does not work22:18
*** ddieterly has quit IRC22:18
Shrewsjeblair: still trying to figure out wth is going on22:18
*** mdrabe has quit IRC22:19
jeblair(i was trying to be a good ansible citizen and help test, but if it's more like "it's usually broke don't bother" that's cool)22:19
pabelangerjeblair: oh, confused, sorry. reverted locally22:19
jeblairpabelanger: right.  and it still failed, so that seems unlikely to be the problem22:19
Shrewsjeblair: we have shade tests against ansible HEAD to help them catch their breaking changes  :)22:19
*** gluytium has joined #openstack-infra22:20
clarkbShrews: those probably all delegate to localhost and use a local connection so don't catch ssh problems. :)22:20
*** pradk has quit IRC22:21
Shrewsclarkb: yup22:21
pabelanger127.0.0.2 :)22:21
*** wznoinsk has joined #openstack-infra22:21
Shrewsclarkb: point being, HEAD breaks a lot22:21
*** wznoinsk_ has quit IRC22:22
Shrewsjeblair: ah, ok. i think i may have a working something. gimme a bit22:23
*** roxanaghe has quit IRC22:24
*** acoles_ is now known as acoles22:24
*** rbrndt_ has quit IRC22:24
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Don't try to get stack details if the stack doesn't exist
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Stop piping yes to Heat
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion in periodic job
*** pilgrimstack has quit IRC22:25
*** rlandy has quit IRC22:25
*** erlon has quit IRC22:26
Shrewsjeblair: is updated22:26
Shrewsjeblair: it uses os.rename() instead of the atomic_move() func b/c we cannot import that without breaking async22:27
Shrewsjeblair: bcoca suggested that22:27
*** cody-somerville has joined #openstack-infra22:27
clarkbShrews: does async_wrapper run on windows too? if so rename isn't a thing iirc22:28
clarkbShrews: so you might need to special case windows22:28
*** tosky has quit IRC22:28
Shrewsclarkb: i have no idea22:29
*** Sukhdev has quit IRC22:29
Shrewsclarkb: if this solves the immediate problem, we can worry about it then22:29
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Don't try to get stack details if the stack doesn't exist
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Stop piping yes to Heat
clarkbShrews: ++22:30
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion in periodic job
*** Sukhdev has joined #openstack-infra22:30
*** roxanaghe has joined #openstack-infra22:34
Shrewsjeblair: you got a patch for the suggested debug line in the executor?22:34
*** irdkehn_ has joined #openstack-infra22:34
*** Sukhdev has quit IRC22:35
*** irdkehn_ is now known as dkehn22:35
*** Sukhdev has joined #openstack-infra22:35
*** javeriak has joined #openstack-infra22:35
jeblairShrews: no -- but i think we should just interpolate it into the failure message for our purposes.  that's not what they want upstream, but i don't want to run with debug=1 -- it's *way* too much data.  so since we're local-patching anyway, let's just do that.22:36
jeblairclarkb: after i checkout v2.1.0.0, how do i make the submodules be in the correct state?22:37
Shrewsjeblair: git submodule update --init --recursive22:37
Shrewsi think22:37
jeblairoh ok, --init doesn't do what i thought it does then :)22:37
*** tpsilva has quit IRC22:37
clarkbjeblair: git submodule update oh ya shrews has it22:38
jeblairyep that did it, thx22:39
*** vhosakot has quit IRC22:39
*** javeriak_ has joined #openstack-infra22:39
*** javeriak has quit IRC22:40
*** Sukhdev has quit IRC22:40
*** dkehn has quit IRC22:41
*** irdkehn has joined #openstack-infra22:41
*** yamahata has joined #openstack-infra22:41
jeblairShrews: your patch is on the devel branch, right?22:41
Shrewsjeblair: no22:41
Shrewsjeblair: async_atomic_move22:42
jeblairokay, let me try harder to github22:42
Shrewsjeblair: easier to copy pasta22:42
jeblairShrews: no i mean it was *based* on devel?22:42
Shrewsjeblair: oh, yeah. up-to-date devel22:42
clarkbjeblair: be careful doing that22:43
jeblairright, i tried to pull it onto 2100 and it conflicted22:43
jeblairi will just pull it into a devel branch then cherry pick22:43
Shrewsgah. must have been changes b/w 2.1 and HEAD22:43
Shrewsjeblair: sorry, didn't consider that22:43
jeblairno prob, it cherry-picks cleanly22:43
Shrewsit's a small diff22:43
*** dkehn_ has joined #openstack-infra22:44
*** rdx_matt has joined #openstack-infra22:45
*** roxanaghe has quit IRC22:46
jeblairbtw, this would be much easier if we had afs on all the hosts22:48
clarkbjeblair: this?22:48
jeblairpushing a git commit onto a repo on all the launchers22:48
jeblairpulling is easy, pushing is harder22:49
jeblairbut pulling from file:///afs/ is easy22:49
mordredjeblair: rsync?22:50
clarkbI tend to use patch files but there are proably better ways22:51
jeblairi can easily use ansible to run a command on all the hosts22:51
jeblairbut if i want to push something to them, i'm into playbook territory22:51
*** Sukhdev has joined #openstack-infra22:54
Shrewslet's hope we don't have to revert22:55
jeblairno kidding :)22:55
Shrewsi'm pretty sure it doesn't BREAK anything, it's the fixing aspect where i have doubts22:56
jeblairokay, i made temporary branches in both ansible and the core submodule; i pushed those in a bash for-loop to each host, then switched back to ansible to do the checkouts and submodule inits22:57
jeblairthe repo in zl01 looks to be in the correct state22:58
clarkbjeblair: double check the submodules as those can be twitchy22:58
jeblairi will manually install it now only on that host and see if it fails to break things22:58
clarkbits one of the ways you can lose data with git22:58
jeblairclarkb: yeah, git log in the core submodule looks correct22:58
jeblairi see Shrews commit at the top22:58
*** Sukhdev has quit IRC22:59
*** gordc has quit IRC22:59
anteayamorning jhesketh22:59
*** salv-orl_ has joined #openstack-infra22:59
jeblairoh, pip install . isn't going to do anything22:59
*** sdague has quit IRC22:59
jeblairpip install -U . is going to break because it can't compile something22:59
jeblairmaybe i should pip uninstall ansible; pip install .; and hope/assume that jobs launched in the interim will get kicked back for rescheduling23:00
jeblairno one has screamed, so i will do that.23:01
clarkbI am quickly running out of stema. was up at 3am to do an airport run today23:01
*** salv-orlando has quit IRC23:02
jeblairnope, turns out that was impactful23:02
jeblairlots of POST_FAILURES23:02
* Shrews hands clarkb more stema23:02
jeblairnext time we do this we may need to do a zuul-launcher stop to abort all the jobs23:02
clarkbjeblair: stop, update ansible, start?23:03
jeblairor does anyone know of a way to force pip to "re-install" a version?23:03
*** Sukhdev has joined #openstack-infra23:03
jeblairclarkb: yeah; failing that ^23:03
*** ams__ has quit IRC23:03
clarkb--force-reinstall but that seems to be a type of --upgrade23:04
jeblairthat's clear as mud23:04
jeblairyeah, and i suspect upgrade will break23:04
clarkbyou could try --no-deps23:04
clarkbto keep those fixed but update ansible23:04
jeblairi will try that on zl0223:05
openstackgerritMikhail S Medvedev proposed openstack-infra/puppet-mysql_backup: Use puppetlabs-mysql to install mysql client package
*** tphummel has joined #openstack-infra23:07
*** Sukhdev has quit IRC23:07
*** Sukhdev has joined #openstack-infra23:11
jeblairokay, that worked, but basically did the same thing as on zl01.  just slightly faster23:12
jeblairstill killed a bunch of jobs though.23:12
jeblairi'm hard stopping zl0323:13
jeblairreinstalled and started23:15
jeblairi'll repeat that on 04-0723:15
pabelangerthe hope was to keep current ansible-playbook jobs working?23:15
*** bhavik has quit IRC23:16
jeblairpabelanger: yeah, but the reinstall is too slow + too many jobs running23:16
pabelangerya, too bad. would have been nice23:16
openstackgerritStephanie Miller proposed openstack/diskimage-builder: Make svc-map support overridden elements
*** rcernin has joined #openstack-infra23:16
*** zqfan has joined #openstack-infra23:16
rcerninGot d/c anyone for Klingon?23:18
*** rdx_matt has quit IRC23:19
openstackgerritMerged openstack-infra/tripleo-ci: UPDATE REPO FOR DELOREAN_REPO_URL
*** dprince has quit IRC23:19
jeblairall launchers updated23:20
jeblairi'm going to delete the held nodes now so we get a new batch on the next failures23:20
*** sdague has joined #openstack-infra23:20
*** hongbin has quit IRC23:21
*** vhosakot has joined #openstack-infra23:22
jeblair#status log zuul launchers are all running locally patched ansible (source in ~root/ansible) to correct and/or further debug async timeout issue23:22
openstackstatusjeblair: finished logging23:22
jeblairinfra-root, EmilienM, Shrews: ^23:22
clarkbjeblair: maybe link to the patch you are testing too?23:22
*** aeng has joined #openstack-infra23:22
jeblair#status log
openstackstatusjeblair: finished logging23:23
jeblair(plus also a local-only debug info patch)23:23
Shrewsoh good23:24
clarkbShrews: doing quick code review on that change, does the rename at the end of your diff want to be in a finally block?23:26
anteayaclarkb: your parents went home today?23:26
clarkbanteaya: yes23:26
anteayaI'm glad they were able to help you with the move23:26
Shrewsclarkb: hrm, i think both the close and the rename should be, yes23:27
anteayaI'll add to Shrews offer of stema23:27
Shrewsclarkb: good spot23:27
Shrewsjeblair: so....23:27
pabelangerjeblair: Shrews: great work23:27
Shrewsclarkb: ideally, that's not the code path we're hitting anyway23:28
clarkbShrews: ya the old code seemed to lack it as well23:28
Shrewsclarkb: jeblair: i'm going to update the patch to include the finally clause. likely not needed right away, but should update it if this seems to fix things23:30
*** SumitNaiksatam has quit IRC23:30
*** acoles is now known as acoles_23:30
clarkbsounds good23:30
jeblairShrews: sounds good.  i won't apply it now since it doesn't seem to be critical, but if we have to update again, we'll fold it in23:30
*** sdake_ has joined #openstack-infra23:31
*** banix has quit IRC23:32
*** bswartz has quit IRC23:32
*** zz_dimtruck is now known as dimtruck23:33
*** sdake has quit IRC23:35
Shrewsjeblair: clarkb:
Shrewsconfirmed that patch doesn't break a simple playbook either23:40
jeblairif only we could run simple playbooks ;)23:42
Shrewsjeblair: finally going to EOD. ping me if needed23:42
Shrewsmoar scotch time23:43
*** hichihara has quit IRC23:46
*** dmsimard is now known as dmsimard|pto23:48
*** sarob_ has quit IRC23:48
*** amotoki has quit IRC23:48
*** ajmiller has quit IRC23:48
*** sdake_ has quit IRC23:53
*** daemontool has joined #openstack-infra23:53
*** javeriak_ has quit IRC23:54
*** mriedem has joined #openstack-infra23:55
*** gluytium has quit IRC23:55
*** sdake has joined #openstack-infra23:58
*** xarses has quit IRC23:58

Generated by 2.14.0 by Marius Gedminas - find it at!