| *** gokrokve has quit IRC | 00:00 | |
| *** ZZelle_ has quit IRC | 00:00 | |
| clarkb | ianw: https://jenkins03.openstack.org/job/check-tempest-dsvm-centos7/6/ | 00:03 |
|---|---|---|
| ianw | clarkb: hmm, merged a change to fix up the epel repo ... checking | 00:04 |
| *** msabramo has quit IRC | 00:06 | |
| *** VijayTripathi has quit IRC | 00:06 | |
| *** Svedrin has quit IRC | 00:07 | |
| *** guibuild has quit IRC | 00:07 | |
| clarkb | maybe we just need to rebuild images? | 00:09 |
| *** VijayTripathi has joined #openstack-infra | 00:09 | |
| *** nijaba has quit IRC | 00:12 | |
| *** derekh has quit IRC | 00:18 | |
| *** arnaud__ has quit IRC | 00:20 | |
| *** ZZelle has quit IRC | 00:20 | |
| *** pcrews has quit IRC | 00:20 | |
| *** yamahata has quit IRC | 00:20 | |
| *** camunoz has quit IRC | 00:20 | |
| *** HeOS has quit IRC | 00:20 | |
| *** erlon has quit IRC | 00:20 | |
| *** Ryan_Lane has quit IRC | 00:20 | |
| *** reed has quit IRC | 00:20 | |
| *** jamielennox has quit IRC | 00:20 | |
| *** dmsimard_away has quit IRC | 00:20 | |
| *** Kiall has quit IRC | 00:20 | |
| *** imcsk8 has quit IRC | 00:20 | |
| *** vladan has quit IRC | 00:20 | |
| *** kashyap has quit IRC | 00:20 | |
| *** marun has quit IRC | 00:20 | |
| *** sabeen1 has quit IRC | 00:20 | |
| *** packet has quit IRC | 00:20 | |
| *** ChuckC has quit IRC | 00:20 | |
| *** davidhadas_ has quit IRC | 00:20 | |
| *** dangers_away has quit IRC | 00:20 | |
| *** bcourt has quit IRC | 00:20 | |
| *** roaet_ has quit IRC | 00:20 | |
| *** mtreinish has quit IRC | 00:20 | |
| *** leifmadsen has quit IRC | 00:20 | |
| *** ashp has quit IRC | 00:20 | |
| *** ianw has quit IRC | 00:20 | |
| *** fungi has quit IRC | 00:20 | |
| *** pabelanger has quit IRC | 00:20 | |
| *** msuriar has quit IRC | 00:20 | |
| *** vishy has quit IRC | 00:20 | |
| *** JayF has quit IRC | 00:20 | |
| *** adam_g has quit IRC | 00:20 | |
| *** s0nea has quit IRC | 00:20 | |
| *** ctracey has quit IRC | 00:20 | |
| *** Mithrandir has quit IRC | 00:20 | |
| *** jraim has quit IRC | 00:20 | |
| *** rakhmerov has quit IRC | 00:20 | |
| *** mordred has quit IRC | 00:20 | |
| *** zigo has quit IRC | 00:20 | |
| *** zaphod42 has quit IRC | 00:20 | |
| *** 14WAADG9C has quit IRC | 00:20 | |
| *** r-daneel has quit IRC | 00:20 | |
| *** Adri2000 has quit IRC | 00:20 | |
| *** HenryG has quit IRC | 00:20 | |
| *** ttx has quit IRC | 00:20 | |
| *** dtroyer has quit IRC | 00:20 | |
| *** jaypipes has quit IRC | 00:20 | |
| *** marcusvrn_ has quit IRC | 00:20 | |
| *** AJaeger has quit IRC | 00:20 | |
| *** zz_gondoi has quit IRC | 00:20 | |
| *** Vek has quit IRC | 00:20 | |
| *** jbryce has quit IRC | 00:20 | |
| *** jmeridth has quit IRC | 00:20 | |
| *** comstud has quit IRC | 00:20 | |
| *** apmelton has quit IRC | 00:20 | |
| *** mestery has quit IRC | 00:20 | |
| *** jvrbanac has quit IRC | 00:20 | |
| *** wendar has quit IRC | 00:20 | |
| *** rainya has quit IRC | 00:20 | |
| *** mrodden has quit IRC | 00:20 | |
| *** rossella_s has quit IRC | 00:20 | |
| *** StevenK has quit IRC | 00:20 | |
| *** finch has quit IRC | 00:20 | |
| *** stevebaker has quit IRC | 00:20 | |
| *** zhiyan has quit IRC | 00:20 | |
| *** zz_johnthetubagu has quit IRC | 00:20 | |
| *** joa has quit IRC | 00:20 | |
| *** jhesketh has quit IRC | 00:20 | |
| *** paul-- has quit IRC | 00:20 | |
| *** boris-42 has quit IRC | 00:20 | |
| *** armax has quit IRC | 00:20 | |
| *** rwsu has quit IRC | 00:20 | |
| *** larsks has quit IRC | 00:20 | |
| *** greghaynes has quit IRC | 00:20 | |
| *** waynr has quit IRC | 00:20 | |
| *** sandywalsh has quit IRC | 00:20 | |
| *** pfallenop has quit IRC | 00:20 | |
| *** harlowja has quit IRC | 00:20 | |
| *** jeblair has quit IRC | 00:20 | |
| *** timfreund has quit IRC | 00:20 | |
| *** lxsli has quit IRC | 00:20 | |
| *** sbadia has quit IRC | 00:20 | |
| *** dosaboy has quit IRC | 00:20 | |
| *** vipul has quit IRC | 00:20 | |
| *** alaski has quit IRC | 00:20 | |
| *** annegentle has quit IRC | 00:20 | |
| *** jgrimm has quit IRC | 00:20 | |
| *** bswartz has quit IRC | 00:20 | |
| *** cody-somerville has quit IRC | 00:20 | |
| *** Hefeweizen has quit IRC | 00:20 | |
| *** zul has quit IRC | 00:20 | |
| *** michchap has quit IRC | 00:20 | |
| *** tchaypo has quit IRC | 00:20 | |
| *** Mmike has quit IRC | 00:20 | |
| *** dansmith has quit IRC | 00:20 | |
| *** anteaya has quit IRC | 00:20 | |
| *** ruhe has quit IRC | 00:20 | |
| *** SpamapS has quit IRC | 00:20 | |
| *** dhellmann has quit IRC | 00:20 | |
| *** jlk has quit IRC | 00:20 | |
| *** EntropyWorks has quit IRC | 00:20 | |
| *** mjturek has quit IRC | 00:20 | |
| *** yjiang5 has quit IRC | 00:20 | |
| *** shashankhegde has quit IRC | 00:20 | |
| *** 7GHAAEFQK has quit IRC | 00:20 | |
| *** garyh has quit IRC | 00:20 | |
| *** jamespd has quit IRC | 00:20 | |
| *** doude has quit IRC | 00:20 | |
| *** sressot has quit IRC | 00:20 | |
| *** jkt has quit IRC | 00:20 | |
| *** morganfainberg has quit IRC | 00:20 | |
| *** BadCub has quit IRC | 00:20 | |
| *** d0ugal has quit IRC | 00:20 | |
| *** smoser has quit IRC | 00:20 | |
| *** spiffxp has quit IRC | 00:20 | |
| *** emagana has quit IRC | 00:20 | |
| *** katyafervent_awa has quit IRC | 00:20 | |
| *** ivar-lazzaro has quit IRC | 00:20 | |
| *** rkukura has quit IRC | 00:20 | |
| *** mriedem has quit IRC | 00:20 | |
| *** mugsie has quit IRC | 00:20 | |
| *** dkehnm has quit IRC | 00:20 | |
| *** jpeeler has quit IRC | 00:20 | |
| *** tcammann has quit IRC | 00:20 | |
| *** hogepodge has quit IRC | 00:20 | |
| *** bnemec has quit IRC | 00:20 | |
| *** skraynev has quit IRC | 00:20 | |
| *** gilliard has quit IRC | 00:20 | |
| *** ctlaugh_ has quit IRC | 00:20 | |
| *** abramley has quit IRC | 00:20 | |
| *** spiffxp- has quit IRC | 00:20 | |
| *** jesusaurus has quit IRC | 00:20 | |
| *** asadoughi has quit IRC | 00:20 | |
| *** jcooley has quit IRC | 00:20 | |
| *** vponomaryov has quit IRC | 00:20 | |
| *** darvon has quit IRC | 00:20 | |
| *** bodepd_ has quit IRC | 00:20 | |
| *** ekarlso has quit IRC | 00:20 | |
| *** LinuxJedi has quit IRC | 00:20 | |
| *** dstufft has quit IRC | 00:20 | |
| *** schang has quit IRC | 00:20 | |
| *** rodrigods has quit IRC | 00:20 | |
| *** sc68cal has quit IRC | 00:20 | |
| *** saper has quit IRC | 00:20 | |
| *** sdad has quit IRC | 00:20 | |
| *** jgriffith has quit IRC | 00:20 | |
| *** gpocentek has quit IRC | 00:20 | |
| *** VijayTripathi has quit IRC | 00:20 | |
| *** dims_ has quit IRC | 00:20 | |
| *** gyee has quit IRC | 00:20 | |
| *** mmaglana has quit IRC | 00:20 | |
| *** openstackgerrit has quit IRC | 00:20 | |
| *** plars has quit IRC | 00:20 | |
| *** pbelanyi has quit IRC | 00:20 | |
| *** juice has quit IRC | 00:20 | |
| *** davidlenwell has quit IRC | 00:20 | |
| *** bdpayne_ has quit IRC | 00:20 | |
| *** amcrn has quit IRC | 00:20 | |
| *** mwagner_lap has quit IRC | 00:20 | |
| *** mancdaz_away has quit IRC | 00:20 | |
| *** weshay has quit IRC | 00:20 | |
| *** rcarrillocruz has quit IRC | 00:20 | |
| *** dkehn has quit IRC | 00:20 | |
| *** simonmcc has quit IRC | 00:20 | |
| *** achuprin_ has quit IRC | 00:20 | |
| *** funzo has quit IRC | 00:20 | |
| *** lifeless has quit IRC | 00:20 | |
| *** rushiagr_away has quit IRC | 00:20 | |
| *** dougwig has quit IRC | 00:20 | |
| *** toabctl has quit IRC | 00:20 | |
| *** kevinbenton has quit IRC | 00:20 | |
| *** ryanpetrello has quit IRC | 00:20 | |
| *** stevemar has quit IRC | 00:20 | |
| *** masayukig has quit IRC | 00:20 | |
| *** freyes has quit IRC | 00:20 | |
| *** sileht has quit IRC | 00:20 | |
| *** rfolco has quit IRC | 00:20 | |
| *** slagle has quit IRC | 00:20 | |
| *** krtaylor has quit IRC | 00:20 | |
| *** jroll has quit IRC | 00:20 | |
| *** mgagne has quit IRC | 00:20 | |
| *** zz_dimtruck has quit IRC | 00:20 | |
| *** mdbooth has quit IRC | 00:20 | |
| *** dmitryme has quit IRC | 00:20 | |
| *** loquacities has quit IRC | 00:20 | |
| *** jamespage has quit IRC | 00:20 | |
| *** flaper87 has quit IRC | 00:20 | |
| *** lbragstad has quit IRC | 00:20 | |
| *** d34dh0r53 has quit IRC | 00:20 | |
| *** BobBall has quit IRC | 00:20 | |
| *** serverascode__ has quit IRC | 00:20 | |
| *** persia has quit IRC | 00:20 | |
| *** redrobot has quit IRC | 00:20 | |
| *** jd__ has quit IRC | 00:20 | |
| *** DuncanT- has quit IRC | 00:20 | |
| *** mattoliverau has quit IRC | 00:20 | |
| *** EmilienM has quit IRC | 00:20 | |
| *** sweston has quit IRC | 00:20 | |
| *** Hunner has quit IRC | 00:20 | |
| *** Ng has quit IRC | 00:20 | |
| *** maurosr has quit IRC | 00:20 | |
| *** gothicmindfood has quit IRC | 00:20 | |
| *** gus has quit IRC | 00:20 | |
| *** cyeoh has quit IRC | 00:20 | |
| *** dolphm has quit IRC | 00:20 | |
| *** ameade has quit IRC | 00:20 | |
| *** russellb has quit IRC | 00:20 | |
| *** erw has quit IRC | 00:20 | |
| *** SergeyLukjanov has quit IRC | 00:20 | |
| *** tsufiev has quit IRC | 00:20 | |
| *** ildikov has quit IRC | 00:20 | |
| *** tnurlygayanov has quit IRC | 00:20 | |
| *** mgrygoriev has quit IRC | 00:20 | |
| *** markvan has quit IRC | 00:20 | |
| *** kmartin has quit IRC | 00:20 | |
| *** sdake has quit IRC | 00:20 | |
| *** mfisch has quit IRC | 00:20 | |
| *** GheRivero has quit IRC | 00:20 | |
| *** zaro has quit IRC | 00:20 | |
| *** clarkb has quit IRC | 00:20 | |
| *** alazarev has quit IRC | 00:20 | |
| *** ptoohill has quit IRC | 00:20 | |
| *** DinaBelova has quit IRC | 00:20 | |
| *** thingee has quit IRC | 00:20 | |
| *** antonym has quit IRC | 00:20 | |
| *** yamahata_ has quit IRC | 00:20 | |
| *** devananda has quit IRC | 00:20 | |
| *** bogdando has quit IRC | 00:20 | |
| *** timrc has quit IRC | 00:20 | |
| *** phschwartz has quit IRC | 00:20 | |
| *** uvirtbot has quit IRC | 00:20 | |
| *** SlickNik has quit IRC | 00:20 | |
| *** mwhudson has quit IRC | 00:20 | |
| *** soren has quit IRC | 00:20 | |
| *** notmyname has quit IRC | 00:20 | |
| *** niska has quit IRC | 00:20 | |
| *** r1chardj0n3s has quit IRC | 00:20 | |
| *** therve has quit IRC | 00:20 | |
| *** gema has quit IRC | 00:20 | |
| *** zz_naotok has quit IRC | 00:20 | |
| *** gmurphy has quit IRC | 00:20 | |
| *** NikitaKonovalov has quit IRC | 00:20 | |
| *** vhoward has quit IRC | 00:20 | |
| *** hdd has quit IRC | 00:20 | |
| *** patrickeast has quit IRC | 00:20 | |
| *** primeministerp has quit IRC | 00:20 | |
| *** mbacchi has quit IRC | 00:20 | |
| *** skolekonov has quit IRC | 00:20 | |
| *** gary-smith has quit IRC | 00:20 | |
| *** chuckC_ has quit IRC | 00:20 | |
| *** dteselkin has quit IRC | 00:20 | |
| *** hemnass has quit IRC | 00:20 | |
| *** NobodyCam has quit IRC | 00:20 | |
| *** Shrews has quit IRC | 00:20 | |
| *** Nikerabbit has quit IRC | 00:20 | |
| *** krotscheck has quit IRC | 00:20 | |
| *** charz has quit IRC | 00:20 | |
| *** crinkle has quit IRC | 00:20 | |
| *** cinerama has quit IRC | 00:20 | |
| *** jogo has quit IRC | 00:20 | |
| *** grantbow has quit IRC | 00:20 | |
| *** tteggel has quit IRC | 00:20 | |
| *** eglynn-officeafk has quit IRC | 00:20 | |
| *** chmouel has quit IRC | 00:20 | |
| *** winston-d has quit IRC | 00:20 | |
| *** blkperl has quit IRC | 00:20 | |
| *** nibalizer has quit IRC | 00:20 | |
| *** Guest20522 has quit IRC | 00:20 | |
| *** odyi has quit IRC | 00:20 | |
| *** esmute has quit IRC | 00:20 | |
| *** niedbalski has quit IRC | 00:20 | |
| *** pleia2 has quit IRC | 00:20 | |
| *** radez_g0n3 has quit IRC | 00:20 | |
| *** Vivek has quit IRC | 00:20 | |
| *** wolsen has quit IRC | 00:20 | |
| *** dguerri has quit IRC | 00:20 | |
| *** ChanServ has quit IRC | 00:20 | |
| *** mmedvede has quit IRC | 00:20 | |
| *** HeOS_ has quit IRC | 00:20 | |
| *** sunil_ has quit IRC | 00:20 | |
| *** asselin has quit IRC | 00:20 | |
| *** che-arne has quit IRC | 00:20 | |
| *** fabo has quit IRC | 00:20 | |
| *** gnuoy has quit IRC | 00:20 | |
| *** TheJulia has quit IRC | 00:20 | |
| *** mkerrin has quit IRC | 00:20 | |
| *** shardy has quit IRC | 00:20 | |
| *** dvorak has quit IRC | 00:20 | |
| *** morgabra has quit IRC | 00:20 | |
| *** psedlak has quit IRC | 00:20 | |
| *** hamzy has quit IRC | 00:20 | |
| *** dkliban has quit IRC | 00:20 | |
| *** bradm has quit IRC | 00:20 | |
| *** mikal has quit IRC | 00:20 | |
| *** tristanC has quit IRC | 00:20 | |
| *** gaelL has quit IRC | 00:20 | |
| *** beecee has quit IRC | 00:20 | |
| *** dstanek has quit IRC | 00:20 | |
| *** mkoderer has quit IRC | 00:20 | |
| *** isviridov_away has quit IRC | 00:20 | |
| *** asettle has quit IRC | 00:20 | |
| *** kevinbenton has joined #openstack-infra | 00:27 | |
| *** toabctl has joined #openstack-infra | 00:27 | |
| *** dougwig has joined #openstack-infra | 00:27 | |
| *** rushiagr_away has joined #openstack-infra | 00:27 | |
| *** lifeless has joined #openstack-infra | 00:27 | |
| *** funzo has joined #openstack-infra | 00:27 | |
| *** simonmcc has joined #openstack-infra | 00:27 | |
| *** dkehn has joined #openstack-infra | 00:27 | |
| *** rcarrillocruz has joined #openstack-infra | 00:27 | |
| *** weshay has joined #openstack-infra | 00:27 | |
| *** mancdaz_away has joined #openstack-infra | 00:27 | |
| *** mwagner_lap has joined #openstack-infra | 00:27 | |
| *** bdpayne_ has joined #openstack-infra | 00:27 | |
| *** msabramo has joined #openstack-infra | 00:27 | |
| *** achuprin_ has joined #openstack-infra | 00:27 | |
| *** asettle has joined #openstack-infra | 00:27 | |
| *** unicell has joined #openstack-infra | 00:27 | |
| *** nijaba_ has joined #openstack-infra | 00:27 | |
| *** Svedrin has joined #openstack-infra | 00:27 | |
| *** VijayTripathi has joined #openstack-infra | 00:27 | |
| *** jamielennox has joined #openstack-infra | 00:27 | |
| *** mmedvede has joined #openstack-infra | 00:27 | |
| *** erlon has joined #openstack-infra | 00:27 | |
| *** marun has joined #openstack-infra | 00:27 | |
| *** yamahata has joined #openstack-infra | 00:27 | |
| *** hdd has joined #openstack-infra | 00:27 | |
| *** dims_ has joined #openstack-infra | 00:27 | |
| *** bswartz has joined #openstack-infra | 00:27 | |
| *** cody-somerville has joined #openstack-infra | 00:27 | |
| *** camunoz has joined #openstack-infra | 00:27 | |
| *** mjturek has joined #openstack-infra | 00:27 | |
| *** sabeen1 has joined #openstack-infra | 00:27 | |
| *** yjiang5 has joined #openstack-infra | 00:27 | |
| *** packet has joined #openstack-infra | 00:27 | |
| *** ChuckC has joined #openstack-infra | 00:27 | |
| *** gary-smith has joined #openstack-infra | 00:27 | |
| *** HeOS has joined #openstack-infra | 00:27 | |
| *** gyee has joined #openstack-infra | 00:27 | |
| *** katyafervent_awa has joined #openstack-infra | 00:27 | |
| *** boris-42 has joined #openstack-infra | 00:27 | |
| *** shashankhegde has joined #openstack-infra | 00:27 | |
| *** ildikov has joined #openstack-infra | 00:27 | |
| *** arnaud__ has joined #openstack-infra | 00:27 | |
| *** ivar-lazzaro has joined #openstack-infra | 00:27 | |
| *** zigo has joined #openstack-infra | 00:27 | |
| *** davidhadas_ has joined #openstack-infra | 00:27 | |
| *** ryanpetrello has joined #openstack-infra | 00:27 | |
| *** patrickeast has joined #openstack-infra | 00:27 | |
| *** Ryan_Lane has joined #openstack-infra | 00:27 | |
| *** zaphod42 has joined #openstack-infra | 00:27 | |
| *** primeministerp has joined #openstack-infra | 00:27 | |
| *** 7GHAAEFQK has joined #openstack-infra | 00:27 | |
| *** 14WAADG9C has joined #openstack-infra | 00:27 | |
| *** dangers_away has joined #openstack-infra | 00:27 | |
| *** stevemar has joined #openstack-infra | 00:27 | |
| *** reed has joined #openstack-infra | 00:27 | |
| *** mmaglana has joined #openstack-infra | 00:27 | |
| *** ZZelle has joined #openstack-infra | 00:27 | |
| *** armax has joined #openstack-infra | 00:27 | |
| *** bcourt has joined #openstack-infra | 00:27 | |
| *** pcrews has joined #openstack-infra | 00:27 | |
| *** masayukig has joined #openstack-infra | 00:27 | |
| *** rkukura has joined #openstack-infra | 00:27 | |
| *** mriedem has joined #openstack-infra | 00:27 | |
| *** r-daneel has joined #openstack-infra | 00:27 | |
| *** dmsimard_away has joined #openstack-infra | 00:27 | |
| *** freyes has joined #openstack-infra | 00:27 | |
| *** mbacchi has joined #openstack-infra | 00:27 | |
| *** HeOS_ has joined #openstack-infra | 00:27 | |
| *** rwsu has joined #openstack-infra | 00:27 | |
| *** larsks has joined #openstack-infra | 00:27 | |
| *** sunil_ has joined #openstack-infra | 00:27 | |
| *** mugsie has joined #openstack-infra | 00:27 | |
| *** sileht has joined #openstack-infra | 00:27 | |
| *** openstackgerrit has joined #openstack-infra | 00:27 | |
| *** mgrygoriev has joined #openstack-infra | 00:27 | |
| *** skolekonov has joined #openstack-infra | 00:27 | |
| *** Adri2000 has joined #openstack-infra | 00:27 | |
| *** garyh has joined #openstack-infra | 00:27 | |
| *** jamespd has joined #openstack-infra | 00:27 | |
| *** greghaynes has joined #openstack-infra | 00:27 | |
| *** markvan has joined #openstack-infra | 00:27 | |
| *** Kiall has joined #openstack-infra | 00:27 | |
| *** plars has joined #openstack-infra | 00:27 | |
| *** HenryG has joined #openstack-infra | 00:27 | |
| *** imcsk8 has joined #openstack-infra | 00:27 | |
| *** dkehnm has joined #openstack-infra | 00:27 | |
| *** roaet_ has joined #openstack-infra | 00:27 | |
| *** asselin has joined #openstack-infra | 00:27 | |
| *** Hefeweizen has joined #openstack-infra | 00:27 | |
| *** kmartin has joined #openstack-infra | 00:27 | |
| *** ttx has joined #openstack-infra | 00:27 | |
| *** sdake has joined #openstack-infra | 00:27 | |
| *** waynr has joined #openstack-infra | 00:27 | |
| *** dtroyer has joined #openstack-infra | 00:27 | |
| *** vladan has joined #openstack-infra | 00:27 | |
| *** sandywalsh has joined #openstack-infra | 00:27 | |
| *** jaypipes has joined #openstack-infra | 00:27 | |
| *** rfolco has joined #openstack-infra | 00:27 | |
| *** zul has joined #openstack-infra | 00:27 | |
| *** slagle has joined #openstack-infra | 00:27 | |
| *** marcusvrn_ has joined #openstack-infra | 00:27 | |
| *** pbelanyi has joined #openstack-infra | 00:27 | |
| *** che-arne has joined #openstack-infra | 00:27 | |
| *** doude has joined #openstack-infra | 00:27 | |
| *** AJaeger has joined #openstack-infra | 00:27 | |
| *** krtaylor has joined #openstack-infra | 00:27 | |
| *** sressot has joined #openstack-infra | 00:27 | |
| *** jpeeler has joined #openstack-infra | 00:27 | |
| *** fabo has joined #openstack-infra | 00:27 | |
| *** mfisch has joined #openstack-infra | 00:27 | |
| *** chuckC_ has joined #openstack-infra | 00:27 | |
| *** zz_gondoi has joined #openstack-infra | 00:27 | |
| *** juice has joined #openstack-infra | 00:27 | |
| *** jhesketh has joined #openstack-infra | 00:27 | |
| *** pfallenop has joined #openstack-infra | 00:27 | |
| *** jmeridth has joined #openstack-infra | 00:27 | |
| *** jroll has joined #openstack-infra | 00:27 | |
| *** jbryce has joined #openstack-infra | 00:27 | |
| *** Vek has joined #openstack-infra | 00:27 | |
| *** tcammann has joined #openstack-infra | 00:27 | |
| *** gnuoy has joined #openstack-infra | 00:27 | |
| *** mgagne has joined #openstack-infra | 00:27 | |
| *** comstud has joined #openstack-infra | 00:27 | |
| *** zz_dimtruck has joined #openstack-infra | 00:27 | |
| *** apmelton has joined #openstack-infra | 00:27 | |
| *** kashyap has joined #openstack-infra | 00:27 | |
| *** mtreinish has joined #openstack-infra | 00:27 | |
| *** michchap has joined #openstack-infra | 00:27 | |
| *** mdbooth has joined #openstack-infra | 00:27 | |
| *** paul-- has joined #openstack-infra | 00:27 | |
| *** leifmadsen has joined #openstack-infra | 00:27 | |
| *** jkt has joined #openstack-infra | 00:27 | |
| *** tchaypo has joined #openstack-infra | 00:27 | |
| *** davidlenwell has joined #openstack-infra | 00:27 | |
| *** ashp has joined #openstack-infra | 00:27 | |
| *** harlowja has joined #openstack-infra | 00:27 | |
| *** hogepodge has joined #openstack-infra | 00:27 | |
| *** adam_g has joined #openstack-infra | 00:27 | |
| *** hemnass has joined #openstack-infra | 00:27 | |
| *** Nikerabbit has joined #openstack-infra | 00:27 | |
| *** morganfainberg has joined #openstack-infra | 00:27 | |
| *** mestery has joined #openstack-infra | 00:27 | |
| *** dmitryme has joined #openstack-infra | 00:27 | |
| *** jvrbanac has joined #openstack-infra | 00:27 | |
| *** wendar has joined #openstack-infra | 00:27 | |
| *** rainya has joined #openstack-infra | 00:27 | |
| *** Mmike has joined #openstack-infra | 00:27 | |
| *** bnemec has joined #openstack-infra | 00:27 | |
| *** TheJulia has joined #openstack-infra | 00:27 | |
| *** vhoward has joined #openstack-infra | 00:27 | |
| *** loquacities has joined #openstack-infra | 00:27 | |
| *** ianw has joined #openstack-infra | 00:27 | |
| *** dteselkin has joined #openstack-infra | 00:27 | |
| *** skraynev has joined #openstack-infra | 00:27 | |
| *** mrodden has joined #openstack-infra | 00:27 | |
| *** dansmith has joined #openstack-infra | 00:27 | |
| *** GheRivero has joined #openstack-infra | 00:27 | |
| *** gilliard has joined #openstack-infra | 00:27 | |
| *** jeblair has joined #openstack-infra | 00:27 | |
| *** smoser has joined #openstack-infra | 00:27 | |
| *** mkerrin has joined #openstack-infra | 00:27 | |
| *** shardy has joined #openstack-infra | 00:27 | |
| *** jamespage has joined #openstack-infra | 00:27 | |
| *** dvorak has joined #openstack-infra | 00:27 | |
| *** rossella_s has joined #openstack-infra | 00:27 | |
| *** morgabra has joined #openstack-infra | 00:27 | |
| *** anteaya has joined #openstack-infra | 00:27 | |
| *** zaro has joined #openstack-infra | 00:27 | |
| *** clarkb has joined #openstack-infra | 00:27 | |
| *** NobodyCam has joined #openstack-infra | 00:27 | |
| *** BadCub has joined #openstack-infra | 00:27 | |
| *** alazarev has joined #openstack-infra | 00:27 | |
| *** ctlaugh_ has joined #openstack-infra | 00:27 | |
| *** ptoohill has joined #openstack-infra | 00:27 | |
| *** StevenK has joined #openstack-infra | 00:27 | |
| *** pabelanger has joined #openstack-infra | 00:27 | |
| *** fungi has joined #openstack-infra | 00:27 | |
| *** finch has joined #openstack-infra | 00:27 | |
| *** psedlak has joined #openstack-infra | 00:27 | |
| *** msuriar has joined #openstack-infra | 00:27 | |
| *** vishy has joined #openstack-infra | 00:27 | |
| *** stevebaker has joined #openstack-infra | 00:27 | |
| *** zhiyan has joined #openstack-infra | 00:27 | |
| *** d0ugal has joined #openstack-infra | 00:27 | |
| *** flaper87 has joined #openstack-infra | 00:27 | |
| *** Shrews has joined #openstack-infra | 00:27 | |
| *** zz_johnthetubagu has joined #openstack-infra | 00:27 | |
| *** jgrimm has joined #openstack-infra | 00:27 | |
| *** annegentle has joined #openstack-infra | 00:27 | |
| *** alaski has joined #openstack-infra | 00:27 | |
| *** vipul has joined #openstack-infra | 00:27 | |
| *** dosaboy has joined #openstack-infra | 00:27 | |
| *** lxsli has joined #openstack-infra | 00:27 | |
| *** sbadia has joined #openstack-infra | 00:27 | |
| *** timfreund has joined #openstack-infra | 00:27 | |
| *** lbragstad has joined #openstack-infra | 00:27 | |
| *** JayF has joined #openstack-infra | 00:27 | |
| *** hamzy has joined #openstack-infra | 00:27 | |
| *** d34dh0r53 has joined #openstack-infra | 00:27 | |
| *** BobBall has joined #openstack-infra | 00:27 | |
| *** s0nea has joined #openstack-infra | 00:27 | |
| *** DinaBelova has joined #openstack-infra | 00:27 | |
| *** ruhe has joined #openstack-infra | 00:27 | |
| *** abramley has joined #openstack-infra | 00:27 | |
| *** spiffxp- has joined #openstack-infra | 00:27 | |
| *** thingee has joined #openstack-infra | 00:27 | |
| *** SpamapS has joined #openstack-infra | 00:27 | |
| *** serverascode__ has joined #openstack-infra | 00:27 | |
| *** ctracey has joined #openstack-infra | 00:27 | |
| *** persia has joined #openstack-infra | 00:27 | |
| *** Mithrandir has joined #openstack-infra | 00:27 | |
| *** jraim has joined #openstack-infra | 00:27 | |
| *** rodrigods has joined #openstack-infra | 00:27 | |
| *** rakhmerov has joined #openstack-infra | 00:27 | |
| *** redrobot has joined #openstack-infra | 00:27 | |
| *** spiffxp has joined #openstack-infra | 00:27 | |
| *** dkliban has joined #openstack-infra | 00:27 | |
| *** dhellmann has joined #openstack-infra | 00:27 | |
| *** jd__ has joined #openstack-infra | 00:27 | |
| *** krotscheck has joined #openstack-infra | 00:27 | |
| *** antonym has joined #openstack-infra | 00:27 | |
| *** DuncanT- has joined #openstack-infra | 00:27 | |
| *** bradm has joined #openstack-infra | 00:27 | |
| *** mattoliverau has joined #openstack-infra | 00:27 | |
| *** charz has joined #openstack-infra | 00:27 | |
| *** tnurlygayanov has joined #openstack-infra | 00:27 | |
| *** crinkle has joined #openstack-infra | 00:27 | |
| *** mikal has joined #openstack-infra | 00:27 | |
| *** EmilienM has joined #openstack-infra | 00:27 | |
| *** sweston has joined #openstack-infra | 00:27 | |
| *** jesusaurus has joined #openstack-infra | 00:27 | |
| *** Hunner has joined #openstack-infra | 00:27 | |
| *** asadoughi has joined #openstack-infra | 00:27 | |
| *** jcooley has joined #openstack-infra | 00:27 | |
| *** cinerama has joined #openstack-infra | 00:27 | |
| *** vponomaryov has joined #openstack-infra | 00:27 | |
| *** darvon has joined #openstack-infra | 00:27 | |
| *** Ng has joined #openstack-infra | 00:27 | |
| *** jogo has joined #openstack-infra | 00:27 | |
| *** mordred has joined #openstack-infra | 00:27 | |
| *** bodepd_ has joined #openstack-infra | 00:27 | |
| *** yamahata_ has joined #openstack-infra | 00:27 | |
| *** devananda has joined #openstack-infra | 00:27 | |
| *** ekarlso has joined #openstack-infra | 00:27 | |
| *** tristanC has joined #openstack-infra | 00:27 | |
| *** jlk has joined #openstack-infra | 00:27 | |
| *** maurosr has joined #openstack-infra | 00:27 | |
| *** grantbow has joined #openstack-infra | 00:27 | |
| *** bogdando has joined #openstack-infra | 00:27 | |
| *** ameade has joined #openstack-infra | 00:27 | |
| *** tteggel has joined #openstack-infra | 00:27 | |
| *** LinuxJedi has joined #openstack-infra | 00:27 | |
| *** EntropyWorks has joined #openstack-infra | 00:27 | |
| *** gaelL has joined #openstack-infra | 00:27 | |
| *** dstufft has joined #openstack-infra | 00:27 | |
| *** gothicmindfood has joined #openstack-infra | 00:27 | |
| *** timrc-afk has joined #openstack-infra | 00:27 | |
| *** schang has joined #openstack-infra | 00:27 | |
| *** gus has joined #openstack-infra | 00:27 | |
| *** beecee has joined #openstack-infra | 00:27 | |
| *** cyeoh has joined #openstack-infra | 00:27 | |
| *** sc68cal has joined #openstack-infra | 00:27 | |
| *** dstanek has joined #openstack-infra | 00:27 | |
| *** phschwartz has joined #openstack-infra | 00:27 | |
| *** r1chardj0n3s has joined #openstack-infra | 00:27 | |
| *** dolphm has joined #openstack-infra | 00:27 | |
| *** saper has joined #openstack-infra | 00:27 | |
| *** russellb has joined #openstack-infra | 00:27 | |
| *** erw has joined #openstack-infra | 00:27 | |
| *** SergeyLukjanov has joined #openstack-infra | 00:27 | |
| *** mkoderer has joined #openstack-infra | 00:27 | |
| *** dguerri has joined #openstack-infra | 00:27 | |
| *** wolsen has joined #openstack-infra | 00:27 | |
| *** Vivek has joined #openstack-infra | 00:27 | |
| *** radez_g0n3 has joined #openstack-infra | 00:27 | |
| *** wolfe.freenode.net sets mode: +o mordred | 00:27 | |
| *** pleia2 has joined #openstack-infra | 00:27 | |
| *** niedbalski has joined #openstack-infra | 00:27 | |
| *** esmute has joined #openstack-infra | 00:27 | |
| *** odyi has joined #openstack-infra | 00:27 | |
| *** Guest20522 has joined #openstack-infra | 00:27 | |
| *** nibalizer has joined #openstack-infra | 00:27 | |
| *** blkperl has joined #openstack-infra | 00:27 | |
| *** winston-d has joined #openstack-infra | 00:27 | |
| *** chmouel has joined #openstack-infra | 00:27 | |
| *** eglynn-officeafk has joined #openstack-infra | 00:27 | |
| *** tsufiev has joined #openstack-infra | 00:27 | |
| *** uvirtbot has joined #openstack-infra | 00:27 | |
| *** mwhudson has joined #openstack-infra | 00:27 | |
| *** sdad has joined #openstack-infra | 00:27 | |
| *** isviridov_away has joined #openstack-infra | 00:27 | |
| *** SlickNik has joined #openstack-infra | 00:27 | |
| *** jgriffith has joined #openstack-infra | 00:27 | |
| *** gpocentek has joined #openstack-infra | 00:27 | |
| *** joa has joined #openstack-infra | 00:27 | |
| *** soren has joined #openstack-infra | 00:27 | |
| *** notmyname has joined #openstack-infra | 00:27 | |
| *** niska has joined #openstack-infra | 00:27 | |
| *** therve has joined #openstack-infra | 00:27 | |
| *** gema has joined #openstack-infra | 00:27 | |
| *** zz_naotok has joined #openstack-infra | 00:27 | |
| *** gmurphy has joined #openstack-infra | 00:27 | |
| *** NikitaKonovalov has joined #openstack-infra | 00:27 | |
| *** ChanServ has joined #openstack-infra | 00:27 | |
| *** wolfe.freenode.net sets mode: +o ChanServ | 00:27 | |
| *** ChuckC has quit IRC | 00:28 | |
| *** yamamoto has joined #openstack-infra | 00:28 | |
| openstackgerrit | A change was merged to openstack/requirements: Add manila to projects.txt https://review.openstack.org/129536 | 00:29 |
| openstackgerrit | A change was merged to openstack/requirements: Require Jinja2>=2.6 https://review.openstack.org/126349 | 00:29 |
| openstackgerrit | A change was merged to openstack/requirements: Allow documentation to create sequence diagrams https://review.openstack.org/123031 | 00:29 |
| *** yamamoto has quit IRC | 00:33 | |
| *** marun has quit IRC | 00:33 | |
| *** dims_ has quit IRC | 00:36 | |
| *** dims has joined #openstack-infra | 00:36 | |
| *** spzala has joined #openstack-infra | 00:39 | |
| *** rmcall has quit IRC | 00:40 | |
| *** aysyd has joined #openstack-infra | 00:40 | |
| *** dims has quit IRC | 00:40 | |
| *** cody-somerville has quit IRC | 00:41 | |
| *** yamamoto has joined #openstack-infra | 00:42 | |
| anteaya | jaypipes: thank you | 00:42 |
| *** zaphod42_ has joined #openstack-infra | 00:43 | |
| *** hdd has quit IRC | 00:46 | |
| *** zaphod42 has quit IRC | 00:46 | |
| jeblair | fungi, zaro, nibalizer, clarkb, anteaya, mordred, SergeyLukjanov: we're all going to be able to watch a streaming terminal session for the afs demonstration on friday! | 00:47 |
| jeblair | i just figured out how to do it without needing shell access to the machine using "script" and "tcpserver" :) | 00:47 |
| *** zaphod42_ has quit IRC | 00:47 | |
| *** gyee has quit IRC | 00:48 | |
| fungi | jeblair: oh, neat! | 00:48 |
| *** bknudson has joined #openstack-infra | 00:48 | |
| fungi | that sounds like a great trick | 00:49 |
| jeblair | "script -c bash -f /tmp/script" and "tcpserver 0.0.0.0 8888 tail -f /tmp/script" | 00:49 |
| fungi | concise | 00:49 |
| jeblair | then viewers just "telnet $hostname 8.8.8.8" | 00:49 |
| *** sdake_ has joined #openstack-infra | 00:50 | |
| JayF | You can do all that with floobits as well | 00:50 |
| JayF | Interactive, as well | 00:50 |
| anteaya | JayF: awesome | 00:50 |
| JayF | we did that with some CoreOS guys to collaborate over a serial over lan | 00:50 |
| fungi | never heard o floobits | 00:50 |
| JayF | I can figure out how it was done if you guys are interested, I think it was written down | 00:50 |
| JayF | fungi: startup that allows shared collaborative *code* editing, that has early support for sharing terminals | 00:51 |
| jeblair | JayF: though one goal of this is _not_ to give people interactive control of my superuser shell :) | 00:51 |
| JayF | oh, don't be so nervous ;) | 00:51 |
| JayF | what's the sudo worst that could sudo happen | 00:51 |
| fungi | JayF: free/open source? | 00:51 |
| JayF | fungi: I honestly don't know; Knowing the folks who run it probably but IMBW | 00:52 |
| JayF | but it's like, a hosted service | 00:52 |
| fungi | oh | 00:52 |
| clarkb | so they get root too ;) | 00:52 |
| JayF | haha | 00:53 |
| JayF | I was just letting you guys know something else existed :) | 00:53 |
| *** bknudson has quit IRC | 00:53 | |
| JayF | it's a startup started by some former rackers ... 2+ yrs ago at this point | 00:53 |
| jeblair | JayF: no! you were performing a social engineering attack to get root! | 00:53 |
| jeblair | JayF: (jk, cool, thanks ;) | 00:53 |
| JayF | jeblair: I think you guys figured out the social engineering attack to get root | 00:53 |
| fungi | script, tcpserver and telnet have the fact that they're pretty standard tools with all free implementations going for them | 00:54 |
| jeblair | JayF: yeah, review some patches, we give you root | 00:54 |
| JayF | jeblair: be so helpful it can't be denied | 00:54 |
| JayF | hah, exactly | 00:54 |
| JayF | that'd make a hilarious session at a summit | 00:54 |
| JayF | "how to social engineer your way to root on openstack infra" | 00:54 |
| jeblair | JayF: hah! excellent idea! :) | 00:54 |
| JayF | then have it be a howto of how to become a helpful gate troubleshooter and infra person | 00:54 |
| nibalizer | jeblair: schweeeet! | 00:55 |
| fungi | indeed, sounds like an awesome summit session idea | 00:55 |
| JayF | I have no desire whatsoever to have the access you guys do. I know the trick (as do most ops/sysadmins): access leads to responsiblity, responsibility leads to alerting and alerting leads to paiiinnnn | 00:55 |
| nibalizer | ill offer twitch streaming as yet another way to do it | 00:55 |
| fungi | JayF: i am hardly ever alert | 00:56 |
| nibalizer | actually twitch streamibg is a terrible idea | 00:56 |
| jeblair | JayF: that's why we -2 alerting patches :) | 00:56 |
| fungi | i must be doing it wrong | 00:56 |
| JayF | Hehe :) | 00:56 |
| *** doug-fish has joined #openstack-infra | 00:56 | |
| jeblair | if anyone wants to alpha test: telnet afstest.openstack.org 8888 | 00:57 |
| nibalizer | kk sec | 00:58 |
| nibalizer | i think im connected? dont see anything | 00:59 |
| fungi | pretty awesome | 00:59 |
| jeblair | should see an ls of / | 00:59 |
| fungi | ansi color and all | 00:59 |
| nibalizer | nvm working perfectly | 01:00 |
| jeblair | nibalizer: for some reason, the time for connect to ok for you was a bit slow... | 01:00 |
| jeblair | i wonder if it's reverse dns? | 01:00 |
| *** yamamoto has quit IRC | 01:00 | |
| jeblair | yeah, let me restart without that... 1 sec | 01:00 |
| nibalizer | im using connectbot on my phone, which is a telnet client | 01:00 |
| * fungi has a ptr and saw no delay | 01:00 | |
| fungi | so likely | 01:00 |
| *** patrickeast has quit IRC | 01:00 | |
| nibalizer | so probably just derp | 01:00 |
| asselin | fungi, I'm reverting that JJB patch I mentioned this morning to get my jobs loaded in my ci env. you mentioned there was a fix in progress. do you have the review url? | 01:01 |
| jeblair | restarted it | 01:01 |
| jeblair | so i'm using emacs now... how's that look? :) | 01:02 |
| anteaya | I'm watching | 01:02 |
| *** bdpayne_ has quit IRC | 01:02 | |
| nibalizer | works great | 01:02 |
| anteaya | looks fine | 01:02 |
| anteaya | ha ha ha | 01:03 |
| jeblair | top look okay? | 01:03 |
| anteaya | never took you for vi chanlleged | 01:03 |
| anteaya | I can see 15 | 01:03 |
| fungi | asselin: 127370 | 01:03 |
| anteaya | 16 now | 01:03 |
| fungi | jeblair: looks great | 01:03 |
| jeblair | cool, anything else we should check? | 01:03 |
| nibalizer | super slick | 01:03 |
| nibalizer | top worked fine for me | 01:04 |
| anteaya | me too | 01:04 |
| asselin | fungi, thanks testing now | 01:04 |
| jeblair | ok, i'm going to shut it down then | 01:05 |
| nibalizer | i cant think of anything else to test | 01:05 |
| fungi | again that's an awesome trick. i'm going to have to steal it | 01:05 |
| jeblair | fungi: yeah, i don't know why everyone doesn't do it all the time | 01:06 |
| jeblair | so final version of command: tcpserver -H -v 0.0.0.0 8888 tail -f /tmp/script | 01:06 |
| jeblair | the -H avoids dns lookups | 01:06 |
| *** spzala has quit IRC | 01:06 | |
| *** bknudson has joined #openstack-infra | 01:06 | |
| fungi | even better | 01:06 |
| *** packet has quit IRC | 01:08 | |
| *** doug-fish has left #openstack-infra | 01:10 | |
| *** gokrokve has joined #openstack-infra | 01:10 | |
| *** jgrimm is now known as zz_jgrimm | 01:10 | |
| *** david-lyle has joined #openstack-infra | 01:11 | |
| *** packet has joined #openstack-infra | 01:12 | |
| *** amotoki has joined #openstack-infra | 01:14 | |
| *** gokrokve has quit IRC | 01:16 | |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 01:17 | |
| *** baoli has joined #openstack-infra | 01:17 | |
| *** Ryan_Lane has quit IRC | 01:18 | |
| *** shashankhegde has quit IRC | 01:18 | |
| jeblair | http://amo-probos.org/post/17 | 01:22 |
| jeblair | posted what we just learned! ^ | 01:23 |
| *** packet has quit IRC | 01:23 | |
| *** bdpayne has joined #openstack-infra | 01:24 | |
| *** jerryz has joined #openstack-infra | 01:24 | |
| anteaya | very nice | 01:27 |
| *** gokrokve has joined #openstack-infra | 01:27 | |
| *** sarob has joined #openstack-infra | 01:27 | |
| *** sarob has quit IRC | 01:31 | |
| jerryz | jeblair: do i have to get a public signed CA certificate for the nodepool endpoint? | 01:34 |
| *** aysyd has quit IRC | 01:35 | |
| *** aysyd has joined #openstack-infra | 01:36 | |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Add support for "Override Build Parameters" option in EnvInject https://review.openstack.org/129189 | 01:36 |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Add tests for properties inject https://review.openstack.org/129502 | 01:36 |
| anteaya | hey check is down to 127 | 01:37 |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 01:38 | |
| *** david-lyle has quit IRC | 01:39 | |
| *** r-daneel has quit IRC | 01:40 | |
| *** nosnos has joined #openstack-infra | 01:40 | |
| *** gokrokve_ has joined #openstack-infra | 01:43 | |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Added description for node attribute in documentation. https://review.openstack.org/129287 | 01:43 |
| *** sdake_ has quit IRC | 01:45 | |
| *** gokrokve has quit IRC | 01:46 | |
| *** tomoe has joined #openstack-infra | 01:47 | |
| tchaypo | on https://review.openstack.org/#/c/129533/1/zuul/layout.yaml, AJaeger asked me to "use the check-tripleo-jobs template" | 01:48 |
| tchaypo | but I'm not sure what that means, would appreciate any pointers | 01:48 |
| *** stevemar has quit IRC | 01:49 | |
| *** stevemar has joined #openstack-infra | 01:50 | |
| anteaya | tchaypo: hi | 01:51 |
| anteaya | http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n473 | 01:51 |
| anteaya | in the zuul/layout.yaml file, at the top there are zuul/layout templates | 01:51 |
| anteaya | this groups jobs together | 01:52 |
| anteaya | since some jobs occur in groups | 01:52 |
| anteaya | the check tripleo jobs tend to occur as a group | 01:52 |
| anteaya | so to be most efficient that template was created | 01:52 |
| anteaya | you are being encouraged to use it in your patch | 01:52 |
| anteaya | so remove lines 4602-4605 | 01:54 |
| anteaya | and add check-tripleo-jobs to the list of templates | 01:54 |
| tchaypo | anteaya: thank you :) | 01:56 |
| anteaya | welcome | 01:57 |
| anteaya | nice asking | 01:57 |
| anteaya | :D | 01:57 |
| StevenK | tchaypo: Templates are excellent things, and you should use them | 01:57 |
| *** aysyd has quit IRC | 02:00 | |
| *** wenlock has joined #openstack-infra | 02:02 | |
| *** yaguang has joined #openstack-infra | 02:03 | |
| *** gokrokve_ has quit IRC | 02:05 | |
| *** changbl has joined #openstack-infra | 02:07 | |
| *** sdake_ has joined #openstack-infra | 02:08 | |
| ianw | clarkb / jeblair : https://review.openstack.org/130077 (https://review.openstack.org/#/c/130077/) fixes centos7 job (https://jenkins04.openstack.org/job/check-tempest-dsvm-centos7/13/console). updating rax images would also do it i think | 02:21 |
| anteaya | ianw: you linked to the same patch twice | 02:22 |
| anteaya | did you mean to do that? | 02:22 |
| clarkb | let me rebuilt centos7 images now because they are ancient | 02:22 |
| ianw | anteaya: no, sorry, meant to post the description (Always install latest EPEL) | 02:23 |
| anteaya | ah | 02:23 |
| ianw | also rax have fixed their image not to pre-install cloud-init with pip, so we can fixup some devstack work-arounds | 02:24 |
| anteaya | ianw: have to tested these commmands? | 02:24 |
| clarkb | image builds are running in rax | 02:25 |
| ianw | anteaya: sorry, which commands? | 02:25 |
| *** hdd has joined #openstack-infra | 02:26 | |
| anteaya | sorry the ones in the patch, but looking closer they are straightforward | 02:26 |
| anteaya | but this throws me off [epel-bootstrap] | 02:26 |
| *** arnaud__ has quit IRC | 02:27 | |
| ianw | anteaya: just wanted to change that to something more unique | 02:28 |
| *** SumitNaiksatam has joined #openstack-infra | 02:29 | |
| anteaya | sorry [] is an evaluation in bash | 02:29 |
| anteaya | reading about it now | 02:29 |
| *** erlon has quit IRC | 02:29 | |
| ianw | anteaya: in that patch, it's part of a heredoc writing out a .repo file ... it's just a little trick to install the latest EPEL | 02:30 |
| anteaya | cool | 02:31 |
| *** zz_dimtruck is now known as dimtruck | 02:31 | |
| anteaya | on the man page for yum I don't see -bootstrap: http://linux.die.net/man/8/yum | 02:34 |
| ianw | anteaya: the highlighting might have got you: it's just enabling that repo explicitly to install the epel-release rpm "sudo yum --enablerepo=epel-bootstrap -y install epel-release" | 02:35 |
| *** mriedem has quit IRC | 02:35 | |
| ianw | anteaya: the epel-release rpm then installs it's own .repo file for epel | 02:36 |
| *** mmedvede has quit IRC | 02:36 | |
| ianw | so then we delete our "bootstrap" repo | 02:36 |
| anteaya | ah | 02:37 |
| anteaya | yes it looks like there is a whitespace between epel and -bootstrap | 02:38 |
| *** rmcall has joined #openstack-infra | 02:38 | |
| anteaya | okay thanks for your patience ianw | 02:39 |
| ianw | who could get impatient when the giants are up by 6 :) | 02:42 |
| anteaya | oh good | 02:42 |
| anteaya | go giants | 02:42 |
| *** harlowja is now known as harlowja_away | 02:43 | |
| *** fandi has joined #openstack-infra | 02:43 | |
| *** hdd has quit IRC | 02:49 | |
| *** bdpayne has quit IRC | 02:51 | |
| *** annegentle has quit IRC | 02:56 | |
| *** fandi has quit IRC | 02:56 | |
| *** marun has joined #openstack-infra | 03:00 | |
| pleia2 | giants \o/ | 03:00 |
| mtreinish | have you guys seen this? https://jenkins03.openstack.org/job/check-tempest-dsvm-ironic-pxe_ssh-postgres-nv/259/console | 03:01 |
| clarkb | pleia2: did they manage to beat the royals? | 03:01 |
| mtreinish | I've seen the same sort of git failures on a couple of jobs that I started recently | 03:02 |
| clarkb | I don't think anyone has managed that for a while | 03:02 |
| clarkb | mtreinish: looking | 03:02 |
| pleia2 | clarkb: 7-0 giants in the bottom of the 7th, so.. :) | 03:02 |
| clarkb | nice | 03:02 |
| clarkb | mtreinish: wow | 03:02 |
| pleia2 | well, now it's 7-1 | 03:02 |
| clarkb | mtreinish: so the way to test that is to check each of the backends directly. I will do that really quickly | 03:03 |
| mtreinish | clarkb: ok, thanks | 03:03 |
| *** pcrews has quit IRC | 03:04 | |
| clarkb | well I can clone system-config from all 5 backends directly | 03:05 |
| clarkb | so its not completely fubar | 03:06 |
| mtreinish | clarkb: hmm, well looking at the zuul status page it's happening on quite a few jobs | 03:06 |
| clarkb | ok | 03:07 |
| clarkb | in other news http://logs.openstack.org/74/128974/1/check/check-grenade-dsvm/213927e/ passed which means I think juno to master is now working | 03:08 |
| clarkb | neutron full failed though for some reason... | 03:08 |
| mtreinish | I was going to suggest maybe something with the d-g repo, but I'm saw a pep8 job fail with it too | 03:08 |
| * clarkb goes back to digging into git | 03:08 | |
| mtreinish | clarkb: hmm I'll look at that fail :) | 03:08 |
| clarkb | cacti makes it look like possibly git04 is having a hard time | 03:10 |
| clarkb | load average: 2.94, 12.34, 26.30 | 03:10 |
| anteaya | hmmm | 03:10 |
| clarkb | compared to load average: 1.43, 2.28, 3.86 on git01 | 03:11 |
| anteaya | big difference | 03:11 |
| clarkb | there are git operations using a ton of memory | 03:12 |
| jeblair | Oct 22 03:00:03 git04 kernel: git-upload-pack invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 | 03:14 |
| mtreinish | well, that would probably mess something up | 03:14 |
| jeblair | also | 03:15 |
| jeblair | Oct 22 02:59:42 git04 git-daemon[1396]: Too many children, dropping connection | 03:15 |
| jeblair | so that and graphite tell us it was very busy with a lot of connections at that time. but i don't know if that's cause or effect. | 03:15 |
| clarkb | there is no lsof centos6 | 03:15 |
| clarkb | * no lsof on centos6 | 03:16 |
| clarkb | I am trying to figure out which repo results in such large git processes | 03:17 |
| jeblair | clarkb: /var/log/messages has the pid list from the oom killer at the time | 03:18 |
| jeblair | clarkb: you could grab that, sort by rss, see how may big processes there were and check their pids in the log to see if they reference a particular repo | 03:18 |
| jeblair | clarkb: if you do that for all processes, you could get an idea of what kind of requests were happening at the moment the oom killer was invoked | 03:19 |
| mtreinish | there's no lsof on centos6? | 03:19 |
| clarkb | mtreinish: apparently its a package you don't just get | 03:19 |
| *** weshay has quit IRC | 03:19 | |
| *** koolhead17 has joined #openstack-infra | 03:19 | |
| jeblair | (i think the processes running at the time of the event might be more interesting than the ones running now) | 03:19 |
| *** ivar-laz_ has joined #openstack-infra | 03:20 | |
| mtreinish | clarkb: I guess just another reason I'm not going to use it | 03:20 |
| *** ivar-laz_ has quit IRC | 03:20 | |
| jeblair | clarkb: according to the lb, git04 had 1.68x the average number of requests of the other servers during the spike | 03:21 |
| clarkb | we may have a DOSer that is getting funnelled to one node? | 03:22 |
| clarkb | my brain is stuck on printer drivers at the moment and completely failing right now | 03:22 |
| *** ivar-lazzaro has quit IRC | 03:23 | |
| clarkb | /var/log/messages doesn't have anything for 19010 prior to OOMing (that was the process selected for death | 03:23 |
| jeblair | clarkb: i don't think the oom process is interesting, that's almost random in this case | 03:24 |
| jeblair | (it killed a bunch of git procs) | 03:24 |
| *** yamahata has quit IRC | 03:25 | |
| clarkb | it was puppet agent that tripped us over the threashold but that is likely just a bad canary | 03:26 |
| clarkb | we were already well under memory pressure | 03:26 |
| jeblair | can always count on it to use lots of ram | 03:26 |
| clarkb | jeblair: pid 4885 has just been hanging out | 03:27 |
| * anteaya notes clarkb and jeblair seem to have this well in hand, can do nothing of use anyway and is about to turn into a pumpkin | 03:27 | |
| anteaya | g'night | 03:27 |
| clarkb | but that may be unrelated | 03:28 |
| StevenK | It always amuses me when the OOM killer kills the wrong process group and leaves the system under pressure | 03:28 |
| jeblair | clarkb: i have the ip addresses during the recent spike sorted by number of requests | 03:30 |
| *** sarob has joined #openstack-infra | 03:30 | |
| jeblair | the top 2, at 493 requests, i am unable to correlate with nodepool because they are ipv6 (doh) | 03:31 |
| jeblair | #3 at 318 is an hpcloud nodepool node; i'm seeing what it was up to | 03:31 |
| *** dimtruck is now known as zz_dimtruck | 03:31 | |
| jeblair | http://logs.openstack.org/44/129244/1/check/check-grenade-dsvm/57ebc46/logs/devstack-gate-setup-workspace-new.txt.gz | 03:32 |
| jeblair | maybe that's just how many git requests a grenade job does? | 03:33 |
| clarkb | could be | 03:33 |
| clarkb | and maybe our images are stale leading to more requests? tht has happened in the past | 03:33 |
| clarkb | wow ya rax is really old. what happened there | 03:34 |
| jeblair | that explains the particularly high number from ipv6 addrs | 03:35 |
| *** sarob has quit IRC | 03:35 | |
| clarkb | I am going to do devstack-trusty builds manually in rax just because we need them and I want ot see if we get errors | 03:35 |
| clarkb | jeblair: would you rathe I wait to aid in debugging? | 03:36 |
| crinkle | clarkb: 129505 now seems to be the only job stuck in check for more than two hours... | 03:36 |
| jeblair | clarkb: nah, i say go for it | 03:36 |
| clarkb | hrm wow | 03:38 |
| clarkb | it looks like nodepool insists on using dib for rax then fails horribly | 03:38 |
| clarkb | probably because the name collides? I may need to drink now | 03:38 |
| crinkle | you weren't drinking before? :) | 03:38 |
| clarkb | and then glance says 413 | 03:38 |
| clarkb | also jenkins client says something about being unable to check jenkins04 | 03:39 |
| clarkb | which appears ot be up and happy now | 03:39 |
| openstackgerrit | Elizabeth K. Joseph proposed a change to openstack-infra/project-config: Point all puppet projects at puppet-system.config https://review.openstack.org/130109 | 03:40 |
| clarkb | ya I think this is a bug in nodepool | 03:43 |
| clarkb | https://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/nodepool.py#n1212 is naive | 03:44 |
| clarkb | it assumes that a label has a singular source | 03:44 |
| clarkb | seriously tempted to revert the dib stuff in nodepool.yaml right now | 03:45 |
| *** Longgeek has joined #openstack-infra | 03:45 | |
| jeblair | clarkb: wfm | 03:45 |
| *** reed has quit IRC | 03:45 | |
| jeblair | clarkb: so i don't see anything about the rate of connections that lead me to believe that's the cause... new connection rates seem normal | 03:45 |
| clarkb | jeblair: ok | 03:46 |
| jeblair | clarkb: so i suspect that means that connections were sticking around longer (and using more memory, etc) | 03:46 |
| clarkb | ya | 03:46 |
| jeblair | clarkb: either because they had the back luck to be really big connections (a bunch of nova operations?), or the system was unable to keep up (bad neighbor?) | 03:46 |
| jeblair | s/back/bad/ | 03:47 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/system-config: Revert "Switch hpcloud devstack-trusty to dib image" https://review.openstack.org/130112 | 03:49 |
| *** koolhead17 has quit IRC | 03:50 | |
| clarkb | jeblair: there are hardware IRQ portions of the cpu graph | 03:50 |
| *** koolhead17 has joined #openstack-infra | 03:50 | |
| *** koolhead17 has joined #openstack-infra | 03:50 | |
| clarkb | jeblair: ^ we may not want to fast approve that | 03:50 |
| jeblair | clarkb: oh why? | 03:50 |
| clarkb | well mostly because I am not sure I will be around to see it shuffle images | 03:50 |
| clarkb | I guess thats not a huge deal | 03:50 |
| clarkb | oh I have the wrong image selected too | 03:51 |
| clarkb | let me fix that really quickly | 03:51 |
| *** Longgeek has quit IRC | 03:51 | |
| jeblair | true, we are not completely broken now. though suspect your hpythesis about the stale cache is a contributing factor, so i think we want to get it it soon. | 03:51 |
| *** AaronGr has joined #openstack-infra | 03:52 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/system-config: Revert "Switch hpcloud devstack-trusty to dib image" https://review.openstack.org/130112 | 03:53 |
| *** bdpayne has joined #openstack-infra | 03:53 | |
| clarkb | jeblair: ok | 03:53 |
| clarkb | that should be ready now (it has the same trusty image we use for the non dib'd hpcloud image builds) | 03:53 |
| jeblair | fungi: you may want to aprv https://review.openstack.org/130112 first thing | 03:56 |
| jeblair | clarkb: perhaps we should consider enabling swap on the git servers. | 03:56 |
| clarkb | that may be a good idea if it can get us past puppet without OOMing | 03:57 |
| jeblair | and possibly through some load spikes. | 03:58 |
| *** jamespage has quit IRC | 04:00 | |
| *** gnuoy has quit IRC | 04:00 | |
| *** gnuoy has joined #openstack-infra | 04:00 | |
| *** arnaud__ has joined #openstack-infra | 04:00 | |
| *** jamespage has joined #openstack-infra | 04:00 | |
| jeblair | clarkb: the network traffic indicates an unusually high number of outbound packets along with inbound data, with some increase in outbound data. | 04:02 |
| jeblair | and a huge about of io reads | 04:03 |
| clarkb | ya that corresponds with xvda IO device reads | 04:03 |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 04:06 | |
| *** yaguang is now known as yaguang_afk | 04:06 | |
| clarkb | I wonder if that could be related to fallback http protocol | 04:06 |
| clarkb | because it uses many more inefficient operations | 04:07 |
| clarkb | (but I am grasping) | 04:07 |
| jeblair | yeah, maybe? | 04:07 |
| *** yamahata has joined #openstack-infra | 04:08 | |
| clarkb | I am going to need to step away for food shortly. somehow I neglected dinner | 04:08 |
| clarkb | anything else you think we should do/try before I go do that? | 04:08 |
| *** fifieldt has joined #openstack-infra | 04:08 | |
| jeblair | clarkb: nah, looking at the graphs, this sort of thing has been happening periodically for a while. i think we should continue to track it down, but we're not broken right now, and we've lived through it before. i don't think it's critical. | 04:09 |
| clarkb | kk | 04:09 |
| *** garyh has quit IRC | 04:09 | |
| *** koolhead17 has quit IRC | 04:12 | |
| *** Ryan_Lane has joined #openstack-infra | 04:12 | |
| *** nosnos has quit IRC | 04:12 | |
| openstackgerrit | Colleen Murphy proposed a change to openstack-infra/project-config: Run lint and syntax stackforge/puppet-* checks on trusty https://review.openstack.org/130115 | 04:13 |
| *** nosnos has joined #openstack-infra | 04:13 | |
| crinkle | clarkb: ^ | 04:13 |
| *** nosnos has quit IRC | 04:17 | |
| *** VijayTripathi has quit IRC | 04:17 | |
| jeblair | clarkb: i've looked at the longest connections and the connections that transferred the most data, and i'm not seeing anything that stands out relative to other servers around the same time. i'm starting to lean toward chalking this up to the somewhat increased load from our own servers (esp due to stale images) and bad neighbors. | 04:22 |
| jeblair | clarkb: i think in the long run, enabling swap and thinking about adding another server are in order | 04:22 |
| jeblair | for x in `grep git04 /tmp/log|awk '{print $10}'|cut -f 3 -d '/'|sort -n|tail -20`; do grep $x log; done | 04:22 |
| jeblair | clarkb, fungi: ^ i used that to find the log entries for the longest connections | 04:23 |
| jeblair | for x in `grep git04 log|awk '{print $11}'|sort -n|tail -20`; do grep $x log; done | 04:23 |
| clarkb | on git.o.o? | 04:23 |
| jeblair | clarkb, fungi: and that for the connections that transferred the most data | 04:23 |
| jeblair | clarkb: yes | 04:24 |
| jeblair | /tmp/log just has 1 hour of haproxy.log data i pre-grepped for speed | 04:24 |
| *** shashankhegde has joined #openstack-infra | 04:24 | |
| jeblair | in "0/1/187921 118998977" 187921 is the total connection time and 118998977 is bytes transferred | 04:24 |
| *** yfried_ has quit IRC | 04:25 | |
| jeblair | from looking at it, every server had a handful of slow full clones | 04:25 |
| jeblair | so i don't think that's unusual | 04:26 |
| jeblair | okay, i'm really eoding now :) | 04:26 |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Add support for "Override Build Parameters" option in EnvInject https://review.openstack.org/129189 | 04:28 |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Add tests for properties inject https://review.openstack.org/129502 | 04:28 |
| *** loki184 has joined #openstack-infra | 04:33 | |
| *** sabeen1 has quit IRC | 04:33 | |
| *** chandankumar has joined #openstack-infra | 04:34 | |
| *** harlowja_at_home has joined #openstack-infra | 04:36 | |
| *** hdd has joined #openstack-infra | 04:36 | |
| *** marun has quit IRC | 04:42 | |
| *** plars has quit IRC | 04:43 | |
| *** plars has joined #openstack-infra | 04:44 | |
| *** baoli has quit IRC | 04:47 | |
| *** Longgeek has joined #openstack-infra | 04:47 | |
| *** marun has joined #openstack-infra | 04:49 | |
| *** armax has quit IRC | 04:51 | |
| *** Longgeek has quit IRC | 04:52 | |
| *** chandankumar has quit IRC | 04:53 | |
| *** KanagarajM has joined #openstack-infra | 04:54 | |
| *** marun has quit IRC | 04:54 | |
| *** nosnos has joined #openstack-infra | 04:58 | |
| *** marun has joined #openstack-infra | 05:00 | |
| *** KanagarajM has quit IRC | 05:01 | |
| *** cody-somerville has joined #openstack-infra | 05:03 | |
| *** marun has quit IRC | 05:04 | |
| ianw | clarkb: you going to work on the label / image build issue? | 05:05 |
| ianw | i can take a look tomorrow .au time, if someone doesn't get to it first | 05:06 |
| *** _nadya_ has joined #openstack-infra | 05:10 | |
| *** bdpayne has quit IRC | 05:11 | |
| *** gokrokve has joined #openstack-infra | 05:11 | |
| *** ildikov has quit IRC | 05:12 | |
| *** chandankumar has joined #openstack-infra | 05:15 | |
| crinkle | AJaeger: clark recommended https://review.openstack.org/#/c/130115 might fix 129505 if you want to take a look, I don't think rechecking is having any effect | 05:15 |
| *** harlowja_at_home has quit IRC | 05:18 | |
| *** hdd has quit IRC | 05:18 | |
| *** loki184 has quit IRC | 05:20 | |
| *** gokrokve has quit IRC | 05:21 | |
| *** talluri has joined #openstack-infra | 05:23 | |
| *** chandankumar has quit IRC | 05:27 | |
| *** sarob has joined #openstack-infra | 05:28 | |
| *** wenlock has quit IRC | 05:32 | |
| *** sarob has quit IRC | 05:32 | |
| *** chandankumar has joined #openstack-infra | 05:35 | |
| *** k4n0 has joined #openstack-infra | 05:36 | |
| *** rmcall has quit IRC | 05:38 | |
| *** yamamoto_ has joined #openstack-infra | 05:38 | |
| *** emagana has joined #openstack-infra | 05:39 | |
| *** yfried_ has joined #openstack-infra | 05:39 | |
| *** rlandy has joined #openstack-infra | 05:42 | |
| *** doug-fish has joined #openstack-infra | 05:42 | |
| *** ihrachyshka has joined #openstack-infra | 05:43 | |
| *** berendt has joined #openstack-infra | 05:44 | |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 05:47 | |
| *** Longgeek has joined #openstack-infra | 05:48 | |
| *** afazekas has joined #openstack-infra | 05:48 | |
| *** chandankumar has quit IRC | 05:50 | |
| *** doug-fish has quit IRC | 05:51 | |
| *** Longgeek has quit IRC | 05:52 | |
| *** dvorak has quit IRC | 05:56 | |
| *** mmaglana has quit IRC | 05:59 | |
| AJaeger | crinkle, I did the recheck in the hope to get it out of the queue... | 06:02 |
| AJaeger | crinkle, it's still in the check queue. Could you just resubmit the patch so that it gets queued again, please? Just change the commit message... | 06:02 |
| *** dvorak has joined #openstack-infra | 06:03 | |
| crinkle | AJaeger: sure | 06:03 |
| AJaeger | crinkle, 130115 looks fine | 06:06 |
| *** sc68cal has quit IRC | 06:10 | |
| *** jamielennox has quit IRC | 06:10 | |
| *** ildikov has joined #openstack-infra | 06:10 | |
| crinkle | AJaeger: 129505 resubmitted | 06:11 |
| *** gokrokve has joined #openstack-infra | 06:11 | |
| *** BharatK has joined #openstack-infra | 06:11 | |
| *** yolanda has joined #openstack-infra | 06:12 | |
| *** gokrokve has quit IRC | 06:13 | |
| *** gokrokve has joined #openstack-infra | 06:13 | |
| *** ihrachyshka has quit IRC | 06:16 | |
| crinkle | AJaeger: still seems to be stuck on the same two checks | 06:17 |
| *** gokrokve has quit IRC | 06:18 | |
| AJaeger | crinkle, thanks - let's see. Let's clarkb or review this and 130115 once they're awake again... | 06:23 |
| *** ildikov has quit IRC | 06:23 | |
| crinkle | mmk | 06:24 |
| *** oramraz has joined #openstack-infra | 06:26 | |
| *** emagana has quit IRC | 06:27 | |
| *** emagana has joined #openstack-infra | 06:28 | |
| *** talluri has quit IRC | 06:28 | |
| *** jamielennox has joined #openstack-infra | 06:30 | |
| *** emagana_ has joined #openstack-infra | 06:31 | |
| *** emagana has quit IRC | 06:32 | |
| *** chandankumar has joined #openstack-infra | 06:33 | |
| *** talluri has joined #openstack-infra | 06:37 | |
| AJaeger | crinkle, sorry - I need a second core for this - and meant "clarkb or fungi" above | 06:40 |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 06:43 | |
| *** bo_sh has joined #openstack-infra | 06:47 | |
| bo_sh | fungi: mordred: clarkb: does anyone know what does "output-host" mean in the worker.yaml configuration file? | 06:48 |
| bo_sh | could it be the logstash host? | 06:48 |
| *** HeOS has quit IRC | 06:50 | |
| *** katyafervent_awa is now known as katyafervent | 06:52 | |
| *** dmsimard_away has quit IRC | 06:53 | |
| *** dmsimard_away has joined #openstack-infra | 06:53 | |
| *** dmsimard_away is now known as dmsimard | 06:54 | |
| *** ihrachyshka has joined #openstack-infra | 06:58 | |
| *** mmaglana has joined #openstack-infra | 06:59 | |
| openstackgerrit | Angus Lees proposed a change to openstack-infra/project-config: Promote gate-neutron-pylint check to voting https://review.openstack.org/129762 | 07:02 |
| *** ihrachyshka has quit IRC | 07:03 | |
| *** sc68cal has joined #openstack-infra | 07:04 | |
| *** mmaglana has quit IRC | 07:05 | |
| *** Ryan_Lane has quit IRC | 07:11 | |
| *** jcoufal has joined #openstack-infra | 07:11 | |
| *** gokrokve has joined #openstack-infra | 07:14 | |
| *** jlibosva has joined #openstack-infra | 07:17 | |
| *** luqas has joined #openstack-infra | 07:18 | |
| *** gokrokve has quit IRC | 07:18 | |
| *** MaxV has joined #openstack-infra | 07:21 | |
| *** IvanBerezovskiy has joined #openstack-infra | 07:22 | |
| *** che-arne has quit IRC | 07:25 | |
| *** Longgeek has joined #openstack-infra | 07:28 | |
| *** talluri has quit IRC | 07:34 | |
| *** arnaud__ has quit IRC | 07:39 | |
| *** e0ne has joined #openstack-infra | 07:39 | |
| *** stevemar has quit IRC | 07:40 | |
| *** shashankhegde has quit IRC | 07:40 | |
| *** davidhadas_ has quit IRC | 07:44 | |
| *** e0ne has quit IRC | 07:45 | |
| *** arxcruz has joined #openstack-infra | 07:48 | |
| *** andreykurilin_ has joined #openstack-infra | 07:49 | |
| *** Guest53198 has joined #openstack-infra | 07:50 | |
| *** mancdaz_away is now known as mancdaz | 07:50 | |
| *** koolhead17 has joined #openstack-infra | 07:50 | |
| *** pblaho has joined #openstack-infra | 07:55 | |
| openstackgerrit | Zhidong Yu proposed a change to openstack/requirements: Sahara's CDH plugin requires this library to communicate with CM. https://review.openstack.org/130153 | 07:56 |
| *** jpich has joined #openstack-infra | 07:56 | |
| SergeyLukjanov | jeblair, cool | 07:59 |
| *** HeOS has joined #openstack-infra | 08:00 | |
| *** rcarrillocruz has left #openstack-infra | 08:00 | |
| *** koolhead17 has quit IRC | 08:00 | |
| *** mmaglana has joined #openstack-infra | 08:01 | |
| *** jistr has joined #openstack-infra | 08:02 | |
| *** mmaglana has quit IRC | 08:05 | |
| *** derekh has joined #openstack-infra | 08:10 | |
| *** alexpilotti has joined #openstack-infra | 08:11 | |
| *** gokrokve has joined #openstack-infra | 08:12 | |
| *** jedimike has joined #openstack-infra | 08:13 | |
| *** jamielennox has quit IRC | 08:14 | |
| *** kim___ has joined #openstack-infra | 08:15 | |
| kim___ | I have a question | 08:15 |
| kim___ | Does anyone got a timeout when he run a jb with heat enable? | 08:16 |
| *** hashar has joined #openstack-infra | 08:16 | |
| *** yamamoto_ has quit IRC | 08:16 | |
| *** gokrokve has quit IRC | 08:16 | |
| *** emagana_ has quit IRC | 08:19 | |
| *** emagana has joined #openstack-infra | 08:20 | |
| *** yamahata has quit IRC | 08:20 | |
| *** isviridov_away is now known as isviridov | 08:22 | |
| *** emagana has quit IRC | 08:24 | |
| *** SumitNaiksatam has quit IRC | 08:25 | |
| *** andreykurilin_ has quit IRC | 08:26 | |
| *** SumitNaiksatam has joined #openstack-infra | 08:29 | |
| *** mpaolino has joined #openstack-infra | 08:32 | |
| *** jamielennox has joined #openstack-infra | 08:33 | |
| *** jamielenz has joined #openstack-infra | 08:34 | |
| *** talluri has joined #openstack-infra | 08:34 | |
| *** talluri has quit IRC | 08:39 | |
| *** fandi has joined #openstack-infra | 08:43 | |
| *** Longgeek has quit IRC | 08:45 | |
| *** jp_at_hp has joined #openstack-infra | 08:47 | |
| *** koolhead17 has joined #openstack-infra | 08:50 | |
| *** talluri has joined #openstack-infra | 08:50 | |
| *** emagana has joined #openstack-infra | 08:51 | |
| *** nijaba_ is now known as nijaba | 08:55 | |
| *** Longgeek has joined #openstack-infra | 08:55 | |
| *** nijaba has quit IRC | 08:55 | |
| *** nijaba has joined #openstack-infra | 08:55 | |
| *** emagana has quit IRC | 08:57 | |
| *** e0ne has joined #openstack-infra | 08:59 | |
| *** mmaglana has joined #openstack-infra | 09:02 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/gearman-plugin: Script to easily build a package https://review.openstack.org/130166 | 09:04 |
| *** ociuhandu has joined #openstack-infra | 09:06 | |
| *** jcoufal has quit IRC | 09:06 | |
| *** mmaglana has quit IRC | 09:06 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 09:09 | |
| *** gokrokve has joined #openstack-infra | 09:11 | |
| *** teran has quit IRC | 09:12 | |
| *** gokrokve has quit IRC | 09:12 | |
| *** gokrokve has joined #openstack-infra | 09:13 | |
| *** koolhead17 has quit IRC | 09:15 | |
| *** ildikov has joined #openstack-infra | 09:15 | |
| *** gokrokve has quit IRC | 09:17 | |
| *** ildikov has quit IRC | 09:21 | |
| *** andreaf has joined #openstack-infra | 09:22 | |
| *** _nadya_ has quit IRC | 09:23 | |
| *** yamamoto_ has joined #openstack-infra | 09:30 | |
| *** yfried_ is now known as yfried|afk | 09:31 | |
| *** katyafervent is now known as katyafervent_awa | 09:33 | |
| *** yfried|afk is now known as yfried_ | 09:33 | |
| *** Longgeek has quit IRC | 09:34 | |
| *** Longgeek has joined #openstack-infra | 09:36 | |
| *** _nadya_ has joined #openstack-infra | 09:37 | |
| *** katyafervent_awa is now known as katyafervent | 09:38 | |
| *** tnurlygayanov has quit IRC | 09:43 | |
| *** alexpilotti has quit IRC | 09:43 | |
| *** emagana has joined #openstack-infra | 09:46 | |
| *** bo_sh has left #openstack-infra | 09:48 | |
| *** alexpilotti has joined #openstack-infra | 09:48 | |
| *** emagana has quit IRC | 09:51 | |
| openstackgerrit | Yair Fried proposed a change to openstack-infra/project-config: Reduce the number of instances in large-ops https://review.openstack.org/130178 | 09:51 |
| *** ihrachyshka has joined #openstack-infra | 09:52 | |
| *** teran has joined #openstack-infra | 09:54 | |
| *** teran_ has joined #openstack-infra | 09:56 | |
| *** bradjones has joined #openstack-infra | 09:58 | |
| *** ociuhandu has quit IRC | 09:59 | |
| *** jp_at_hp has quit IRC | 09:59 | |
| *** teran__ has joined #openstack-infra | 09:59 | |
| *** unicell1 has joined #openstack-infra | 09:59 | |
| *** jp_at_hp has joined #openstack-infra | 09:59 | |
| *** teran has quit IRC | 09:59 | |
| *** ociuhandu has joined #openstack-infra | 10:01 | |
| *** pelix has joined #openstack-infra | 10:01 | |
| *** unicell has quit IRC | 10:01 | |
| *** teran__ has quit IRC | 10:01 | |
| *** ociuhandu has quit IRC | 10:01 | |
| *** teran has joined #openstack-infra | 10:01 | |
| *** e0ne_ has joined #openstack-infra | 10:02 | |
| yfried_ | clarkb: could you please take a look? https://review.openstack.org/#/c/130178/ | 10:02 |
| *** teran_ has quit IRC | 10:02 | |
| *** mmaglana has joined #openstack-infra | 10:02 | |
| *** ociuhandu has joined #openstack-infra | 10:03 | |
| *** e0ne has quit IRC | 10:04 | |
| *** teran has quit IRC | 10:06 | |
| *** mmaglana has quit IRC | 10:07 | |
| *** bradjones has quit IRC | 10:08 | |
| *** bradjones has joined #openstack-infra | 10:09 | |
| *** gokrokve has joined #openstack-infra | 10:11 | |
| *** jcoufal has joined #openstack-infra | 10:14 | |
| *** teran has joined #openstack-infra | 10:14 | |
| yfried_ | fungi: ping | 10:14 |
| *** gokrokve has quit IRC | 10:16 | |
| *** pc_m has joined #openstack-infra | 10:18 | |
| *** pc_m has quit IRC | 10:18 | |
| *** pc_m has joined #openstack-infra | 10:19 | |
| *** teran has quit IRC | 10:19 | |
| *** yamahata has joined #openstack-infra | 10:22 | |
| *** dizquierdo has joined #openstack-infra | 10:24 | |
| *** teran has joined #openstack-infra | 10:26 | |
| *** frank_zhou has joined #openstack-infra | 10:29 | |
| *** apevec has joined #openstack-infra | 10:32 | |
| apevec | hi, stable-maint would appreciate this to be merged: https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:juno-compat,n,z | 10:32 |
| *** _nadya_ has quit IRC | 10:34 | |
| *** yolanda_ has joined #openstack-infra | 10:35 | |
| *** dims has joined #openstack-infra | 10:36 | |
| *** Guest53198 has quit IRC | 10:36 | |
| *** hashar has quit IRC | 10:37 | |
| AJaeger | yfried_, clarkb and fungi are asleep... | 10:38 |
| *** yolanda has quit IRC | 10:39 | |
| AJaeger | apevec, why not give a +1 to those? | 10:39 |
| *** dims has quit IRC | 10:39 | |
| yfried_ | AJaeger: tnx. they are the only core members I know here. looking for eyes on https://review.openstack.org/#/c/130178/ | 10:39 |
| *** hashar has joined #openstack-infra | 10:39 | |
| *** dims has joined #openstack-infra | 10:39 | |
| apevec | well, I propose 2/3 :) | 10:40 |
| *** emagana has joined #openstack-infra | 10:40 | |
| apevec | actaully 3/3 | 10:40 |
| AJaeger | apevec, sorry, didn't check quickly enough | 10:40 |
| *** hashar has quit IRC | 10:40 | |
| AJaeger | yfried_, apevec: I'll grab myself a coffee and will then review... | 10:40 |
| apevec | I amended mtreinish's first patch in the queue too | 10:40 |
| apevec | thanks! | 10:40 |
| yfried_ | AJaeger: enjoy. tnx | 10:41 |
| apevec | stable/juno is already rolling so we need to keep an eye on it :) | 10:41 |
| marcusvrn_ | Hi guys...just a question....all CIs use devstack/master to run? In my CI, I'm using it and it was working well, but after last commit, my CI stop working... | 10:41 |
| marcusvrn_ | "stop working" = returning fail results | 10:43 |
| *** mase_x200 has joined #openstack-infra | 10:43 | |
| *** emagana has quit IRC | 10:44 | |
| *** dmellado has joined #openstack-infra | 10:44 | |
| AJaeger | apevec, do we need stable-compat-jobs on python-saharaclient as well? | 10:47 |
| SergeyLukjanov | AJaeger, apevec, I think so - for stable/juno | 10:47 |
| AJaeger | SergeyLukjanov, will you propose a patch - or should we add this to https://review.openstack.org/#/c/129101 ? | 10:48 |
| SergeyLukjanov | AJaeger, I will - it'll be a good place to discuss | 10:48 |
| AJaeger | Let's do it as separate one - I'll approve the series from apevec ... | 10:49 |
| SergeyLukjanov | AJaeger, ack | 10:49 |
| SergeyLukjanov | AJaeger, I'll upload the change soon | 10:49 |
| AJaeger | apevec, +3s | 10:49 |
| openstackgerrit | A change was merged to openstack-infra/release-tools: Adds stable_freeze.py https://review.openstack.org/125839 | 10:49 |
| *** dizquierdo has quit IRC | 10:51 | |
| apevec | SergeyLukjanov, AJaeger - thanks | 10:51 |
| AJaeger | yfried_, +2 - and I'd like some review by a tempest core on the change | 10:53 |
| yfried_ | AJaeger: I'll try to get one | 10:53 |
| AJaeger | SergeyLukjanov, sorry that I didn't followup last night with you on https://review.openstack.org/129433 - I understand that no further sahara changes are needed, correct? | 10:55 |
| SergeyLukjanov | AJaeger, yup, it looks correct | 10:55 |
| SergeyLukjanov | AJaeger, the question was about sahara-dashboard, yup? | 10:56 |
| openstackgerrit | A change was merged to openstack-infra/project-config: Add juno compat jobs to the tempest check pipeline https://review.openstack.org/128987 | 10:56 |
| AJaeger | SergeyLukjanov, yes. | 10:56 |
| AJaeger | SergeyLukjanov, a +1/2 would be appreciated then to make this clear | 10:56 |
| openstackgerrit | A change was merged to openstack-infra/project-config: Add periodic Juno jobs https://review.openstack.org/128961 | 10:56 |
| *** mwagner_lap has quit IRC | 10:57 | |
| AJaeger | SergeyLukjanov, but nothing urgent... | 10:57 |
| SergeyLukjanov | AJaeger, we only need to run py26 on stable/icehouse for sahara-dashboard, will not have stable/juno for it | 10:57 |
| AJaeger | do we need py26 on master? | 10:57 |
| SergeyLukjanov | AJaeger, nope | 10:57 |
| AJaeger | So, let me change the patch... | 10:58 |
| SergeyLukjanov | AJaeger, sorry about that | 10:58 |
| AJaeger | SergeyLukjanov, no problem | 10:58 |
| openstackgerrit | A change was merged to openstack-infra/project-config: Add Juno stable-compat-jobs for trunk clients and libs https://review.openstack.org/129101 | 10:59 |
| *** amotoki has quit IRC | 10:59 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/project-config: Limit python26 jobs to icehouse/juno for server projects https://review.openstack.org/129433 | 11:00 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/project-config: Remove python26 jobs from all stackforge projects https://review.openstack.org/129434 | 11:00 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/project-config: Remove python26 jobs from various projects https://review.openstack.org/129435 | 11:00 |
| AJaeger | SergeyLukjanov, now you can review ;) | 11:00 |
| SergeyLukjanov | AJaeger, lgtm, thx | 11:02 |
| *** lttrl has joined #openstack-infra | 11:02 | |
| AJaeger | Thanks for the review, SergeyLukjanov ! | 11:02 |
| *** mmaglana has joined #openstack-infra | 11:03 | |
| *** dizquierdo has joined #openstack-infra | 11:06 | |
| *** yaguang_afk has quit IRC | 11:07 | |
| *** fandi has quit IRC | 11:07 | |
| *** mmaglana has quit IRC | 11:07 | |
| *** gokrokve has joined #openstack-infra | 11:11 | |
| *** doug-fish has joined #openstack-infra | 11:14 | |
| *** habib has joined #openstack-infra | 11:15 | |
| *** gokrokve has quit IRC | 11:16 | |
| *** BharatK has quit IRC | 11:17 | |
| *** denis_makogon has joined #openstack-infra | 11:18 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/project-config: Remove requirements job from nova-docker https://review.openstack.org/130192 | 11:23 |
| *** oramraz has left #openstack-infra | 11:23 | |
| *** aysyd has joined #openstack-infra | 11:23 | |
| *** oramraz has joined #openstack-infra | 11:23 | |
| *** hashar has joined #openstack-infra | 11:27 | |
| *** bradjones has quit IRC | 11:30 | |
| *** sdake_ has quit IRC | 11:32 | |
| *** bradjones has joined #openstack-infra | 11:33 | |
| *** emagana has joined #openstack-infra | 11:34 | |
| *** sdake_ has joined #openstack-infra | 11:34 | |
| *** nosnos has quit IRC | 11:37 | |
| *** nosnos has joined #openstack-infra | 11:38 | |
| *** nosnos has quit IRC | 11:38 | |
| *** hashar has quit IRC | 11:38 | |
| *** nosnos has joined #openstack-infra | 11:38 | |
| *** HeOS has quit IRC | 11:38 | |
| *** mwagner_lap has joined #openstack-infra | 11:38 | |
| *** emagana has quit IRC | 11:39 | |
| *** nosnos has quit IRC | 11:39 | |
| *** nosnos has joined #openstack-infra | 11:39 | |
| *** nosnos has quit IRC | 11:40 | |
| *** hashar has joined #openstack-infra | 11:42 | |
| *** yamamoto_ has quit IRC | 11:47 | |
| *** HeOS has joined #openstack-infra | 11:48 | |
| *** baoli has joined #openstack-infra | 11:48 | |
| *** habib has quit IRC | 11:51 | |
| *** koolhead17 has joined #openstack-infra | 11:54 | |
| *** e0ne_ has quit IRC | 11:55 | |
| *** oramraz has quit IRC | 11:56 | |
| *** jcoufal has quit IRC | 11:57 | |
| *** freemanbrandon has joined #openstack-infra | 12:03 | |
| *** mmaglana has joined #openstack-infra | 12:04 | |
| *** jcoufal has joined #openstack-infra | 12:05 | |
| openstackgerrit | Christoffer Skeppstedt proposed a change to openstack-infra/jenkins-job-builder: Serialize combine-queued-commits for git-revision https://review.openstack.org/124965 | 12:05 |
| *** kgiusti has joined #openstack-infra | 12:07 | |
| *** dprince has joined #openstack-infra | 12:07 | |
| *** bswartz has quit IRC | 12:07 | |
| *** mmaglana has quit IRC | 12:08 | |
| *** freemanbrandon has quit IRC | 12:09 | |
| *** Guest53198 has joined #openstack-infra | 12:11 | |
| *** gokrokve has joined #openstack-infra | 12:11 | |
| *** dane_leblanc has joined #openstack-infra | 12:13 | |
| *** oramraz has joined #openstack-infra | 12:14 | |
| *** gokrokve has quit IRC | 12:16 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Doc: ref links for "approval" requirement https://review.openstack.org/130202 | 12:22 |
| *** MarkAtwood has joined #openstack-infra | 12:22 | |
| *** weshay has joined #openstack-infra | 12:25 | |
| *** amotoki has joined #openstack-infra | 12:27 | |
| *** emagana has joined #openstack-infra | 12:28 | |
| *** esker has joined #openstack-infra | 12:29 | |
| openstackgerrit | Matthew Gilliard proposed a change to openstack-infra/system-config: Prevents double-execution of ci_page_loaded https://review.openstack.org/125279 | 12:32 |
| *** emagana has quit IRC | 12:33 | |
| *** esker has quit IRC | 12:33 | |
| *** hdd has joined #openstack-infra | 12:33 | |
| *** timrc-afk is now known as timrc | 12:34 | |
| *** miqui has joined #openstack-infra | 12:40 | |
| *** IvanBerezovskiy has quit IRC | 12:43 | |
| *** IvanBerezovskiy has joined #openstack-infra | 12:44 | |
| *** _nadya__ has joined #openstack-infra | 12:45 | |
| *** MarkAtwood has quit IRC | 12:45 | |
| *** mancdaz is now known as mancdaz_away | 12:46 | |
| *** frank_zhou has quit IRC | 12:47 | |
| *** Guest38702 has joined #openstack-infra | 12:47 | |
| *** talluri has quit IRC | 12:48 | |
| *** esker has joined #openstack-infra | 12:50 | |
| *** teran_ has joined #openstack-infra | 12:53 | |
| *** mjturek has quit IRC | 12:54 | |
| *** nosnos has joined #openstack-infra | 12:54 | |
| *** teran has quit IRC | 12:56 | |
| *** mjturek has joined #openstack-infra | 12:57 | |
| *** signed8bit has joined #openstack-infra | 12:59 | |
| *** bswartz has joined #openstack-infra | 12:59 | |
| *** afazekas has quit IRC | 13:01 | |
| *** bknudson has quit IRC | 13:01 | |
| *** teran_ has quit IRC | 13:02 | |
| *** teran has joined #openstack-infra | 13:02 | |
| *** dustins has joined #openstack-infra | 13:03 | |
| *** stevemar has joined #openstack-infra | 13:03 | |
| *** eharney has joined #openstack-infra | 13:04 | |
| *** mmaglana has joined #openstack-infra | 13:05 | |
| *** dmellado has quit IRC | 13:05 | |
| *** nosnos has quit IRC | 13:06 | |
| *** e0ne has joined #openstack-infra | 13:07 | |
| *** dustins has quit IRC | 13:07 | |
| *** nosnos has joined #openstack-infra | 13:07 | |
| *** nosnos has quit IRC | 13:08 | |
| *** dustins has joined #openstack-infra | 13:09 | |
| *** mmaglana has quit IRC | 13:09 | |
| *** dims has quit IRC | 13:09 | |
| *** dims has joined #openstack-infra | 13:10 | |
| *** teran has quit IRC | 13:11 | |
| bcourt | fungi, clarkb, quick question, for nodepool, is it possible to have it use images that are already in openstack or do they have to be uploaded or created via nodepool? | 13:11 |
| *** gokrokve has joined #openstack-infra | 13:11 | |
| *** Adri2000 has quit IRC | 13:11 | |
| *** dkranz has joined #openstack-infra | 13:11 | |
| *** yfried_ has quit IRC | 13:13 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack/requirements: Enforce requirements check only for registered projects https://review.openstack.org/130210 | 13:13 |
| *** HeOS has quit IRC | 13:13 | |
| *** afazekas has joined #openstack-infra | 13:13 | |
| fungi | bcourt: honestly, i haven't tried. it would probably be fairly simple to add if you wanted to dig into the source (after all you're just asking it to do less than usual, not especially new things) | 13:15 |
| fungi | bcourt: i think by default it's going to try to rebuild images you configure. pretty sure it currently lacks a feature to not do that | 13:16 |
| fungi | bcourt: however, you could make your prep script basically a no-op | 13:16 |
| *** gokrokve has quit IRC | 13:16 | |
| *** nosnos has joined #openstack-infra | 13:16 | |
| *** Adri2000 has joined #openstack-infra | 13:17 | |
| *** Adri2000 has quit IRC | 13:17 | |
| *** Adri2000 has joined #openstack-infra | 13:17 | |
| fungi | bcourt: it would effectively bring up your specified base image, do nothing to it other than stash some nodepool-specific metadata in /etc and then shut it down and take a snapshot of that | 13:17 |
| *** mase_x200 has quit IRC | 13:17 | |
| fungi | apevec: taking a look at that patch series now | 13:18 |
| *** MarkAtwood has joined #openstack-infra | 13:18 | |
| fungi | oh, looks like AJaeger took care of it. awesome! | 13:19 |
| AJaeger | good morning, fungi! | 13:19 |
| *** changbl has quit IRC | 13:20 | |
| *** pblaho has quit IRC | 13:20 | |
| *** freemanbrandon has joined #openstack-infra | 13:20 | |
| fungi | thanks AJaeger! seems that way so far at least | 13:20 |
| *** bknudson has joined #openstack-infra | 13:21 | |
| EmilienM | I just found a weird bug in JJB: https://bugs.launchpad.net/openstack-ci/+bug/1384238 | 13:22 |
| uvirtbot | Launchpad bug 1384238 in openstack-ci "JJB: package name does not match and breaks PBR" [Undecided,New] | 13:22 |
| AJaeger | fungi, only problem I'm aware of is 129505 and this might be fixed by https://review.openstack.org/#/c/130115 | 13:22 |
| EmilienM | not sure how to fix properly it though | 13:22 |
| AJaeger | fungi, that one is not finding a job to run - see top of check queue | 13:22 |
| *** mfer has joined #openstack-infra | 13:22 | |
| *** emagana has joined #openstack-infra | 13:22 | |
| *** zul has quit IRC | 13:23 | |
| fungi | neat | 13:23 |
| *** xyang0 has joined #openstack-infra | 13:23 | |
| *** bo_sh has joined #openstack-infra | 13:23 | |
| *** zul has joined #openstack-infra | 13:23 | |
| *** freemanbrandon has quit IRC | 13:24 | |
| bo_sh | fungi: i have some questions regarding the log-jenkins-worker.yaml file. what is the purpose of the job-filter and the tags attributes? | 13:25 |
| *** dims has quit IRC | 13:25 | |
| fungi | AJaeger: very odd... though if that were the cause i'd have expected either failure or not_registered results | 13:25 |
| bo_sh | fungi: are they used for logstash to index | 13:25 |
| fungi | EmilienM: taking a look | 13:25 |
| AJaeger | fungi, ah, I see | 13:26 |
| fungi | bo_sh: yes, those are tags we can use in lucene queries | 13:26 |
| *** HeOS has joined #openstack-infra | 13:26 | |
| *** dims has joined #openstack-infra | 13:26 | |
| bo_sh | fungi: what about the job-filter? | 13:26 |
| *** dims has quit IRC | 13:26 | |
| EmilienM | fungi: I wonder how our CI could have miss that one. Don't we have functional tests? I may miss something though | 13:26 |
| *** emagana has quit IRC | 13:27 | |
| *** dims has joined #openstack-infra | 13:27 | |
| *** Guest53198 has quit IRC | 13:27 | |
| *** dims has quit IRC | 13:27 | |
| openstackgerrit | A change was merged to openstack-infra/system-config: Revert "Switch hpcloud devstack-trusty to dib image" https://review.openstack.org/130112 | 13:27 |
| *** marun has joined #openstack-infra | 13:28 | |
| fungi | EmilienM: is it possible you're running with an old (pre-0.7) setuptools? | 13:28 |
| *** teran has joined #openstack-infra | 13:28 | |
| *** julim has joined #openstack-infra | 13:28 | |
| EmilienM | fungi: oh let me check | 13:29 |
| *** dims has joined #openstack-infra | 13:29 | |
| *** hdd has quit IRC | 13:29 | |
| *** dims has quit IRC | 13:29 | |
| fungi | EmilienM: also possible this is a bug in pbr. looks familiar so i'm hunting for a dupe there | 13:30 |
| fungi | bo_sh: not sure. i'll see if i can figure it out | 13:30 |
| bo_sh | fungi: ok thanks. | 13:30 |
| EmilienM | fungi: Package python-setuptools.noarch 0:0.9.8-3.el7 | 13:30 |
| EmilienM | fungi: running RHEL7 | 13:31 |
| fungi | EmilienM: i think maybe bug 137957 (but it does say setuptools <0.7.2 there) | 13:31 |
| *** mriedem has joined #openstack-infra | 13:31 | |
| *** stevemar has quit IRC | 13:31 | |
| *** Guest53198 has joined #openstack-infra | 13:31 | |
| fungi | er, bug 1379579 | 13:31 |
| uvirtbot | Launchpad bug 1379579 in python-swiftclient "unhandled PBR Exception "Versioning for this project requires either an sdist tarball" with older (<0.7.2) setuptools" [Undecided,Confirmed] https://launchpad.net/bugs/1379579 | 13:31 |
| fungi | EmilienM: same traceback | 13:32 |
| EmilienM | fungi: weird, I'm running 0.9 | 13:32 |
| fungi | EmilienM: i think maybe the bug reporter merely observed that it broke with some older setuptools, worked with 2.2.0, and assumed it was something solved in 0.7.2 | 13:33 |
| *** cdent has joined #openstack-infra | 13:33 | |
| fungi | so we should probably adjust this bug to indicate that it's also observed with 0.9.8 | 13:33 |
| EmilienM | fungi: if I run "PBR_VERSION=whatever jenkins-.... update ..." it works :) | 13:34 |
| EmilienM | yes +1 | 13:34 |
| *** mancdaz_away is now known as mancdaz | 13:34 | |
| fungi | EmilienM: yep, that's the "let me tell you what you really want" envvar for pbr's versioning routines (we;;, one of several anyway) | 13:34 |
| EmilienM | fungi: I wonder if, since we have one python module we could change the package name | 13:35 |
| *** _nadya__ has quit IRC | 13:35 | |
| fungi | EmilienM: that might work, but i think pbr should handle this better | 13:35 |
| fungi | EmilienM: probably worth getting this in front of dhellmann, lifeless and mordred (since maybe it's already fixed in pbr master or something) | 13:36 |
| fungi | that bug hasn't been triaged yet, so it's possible they don't know | 13:36 |
| *** dims has joined #openstack-infra | 13:36 | |
| *** mmedvede has joined #openstack-infra | 13:36 | |
| *** cpowell has joined #openstack-infra | 13:37 | |
| EmilienM | fungi: when I read the doc ( http://docs.openstack.org/developer/pbr/#usage ) I have the feeling we don't use correctly PBR in JJB | 13:38 |
| fungi | EmilienM: it's possible, but if so then this is a pattern also biting at least one other project too | 13:39 |
| fungi | EmilienM: and i suspect newer setuptools solves this which is why it's not been noticed earlier | 13:39 |
| *** bcourt has quit IRC | 13:40 | |
| fungi | i think i'll add pbr to bug 1384238 and mention the two bugs in one another but not mark them as dupes, since the reporter of 1379579 did say they did not observe the problem with 0.7.2 | 13:41 |
| uvirtbot | Launchpad bug 1384238 in openstack-ci "JJB: package name does not match and breaks PBR" [Undecided,New] https://launchpad.net/bugs/1384238 | 13:41 |
| EmilienM | fungi: sounds reasonable | 13:41 |
| fungi | though it's also entirely possible that red hat's setuptools 0.9.8 is not really 0.9.8 | 13:41 |
| fungi | they do like to selectively backport some bits while forward-porting old behaviors into new versions | 13:42 |
| *** afazekas has quit IRC | 13:42 | |
| *** signed8bit has quit IRC | 13:42 | |
| *** mfer has quit IRC | 13:43 | |
| *** signed8bit has joined #openstack-infra | 13:43 | |
| *** zz_jgrimm is now known as jgrimm | 13:44 | |
| *** mfer has joined #openstack-infra | 13:47 | |
| fungi | and launchpad picks this moment to go out to lunch | 13:47 |
| fungi | lunchpad | 13:47 |
| *** Guest53198 has quit IRC | 13:47 | |
| *** armax has joined #openstack-infra | 13:47 | |
| AJaeger | ;) | 13:48 |
| *** sabeen has joined #openstack-infra | 13:50 | |
| *** Guest53198 has joined #openstack-infra | 13:51 | |
| *** dangers_away is now known as dangers | 13:51 | |
| *** jcoufal_ has joined #openstack-infra | 13:52 | |
| *** chandankumar has quit IRC | 13:52 | |
| *** amotoki has quit IRC | 13:52 | |
| *** bknudson has quit IRC | 13:52 | |
| *** berendt has quit IRC | 13:52 | |
| *** jcoufal has quit IRC | 13:53 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack/requirements: Enforce requirements check only for registered projects https://review.openstack.org/130210 | 13:54 |
| *** jcoufal_ is now known as jcoufal | 13:54 | |
| *** bcourt has joined #openstack-infra | 13:55 | |
| *** yamahata has quit IRC | 13:55 | |
| *** yamahata has joined #openstack-infra | 13:55 | |
| *** thedodd has joined #openstack-infra | 13:57 | |
| *** radez_g0n3 is now known as radez | 13:57 | |
| *** berendt has joined #openstack-infra | 13:58 | |
| *** mfer has quit IRC | 13:58 | |
| *** marun has quit IRC | 13:58 | |
| *** julim has quit IRC | 13:58 | |
| *** mfer has joined #openstack-infra | 14:00 | |
| *** mfer has quit IRC | 14:00 | |
| *** Guest38702 is now known as annegentle | 14:00 | |
| *** julim has joined #openstack-infra | 14:00 | |
| *** mfer has joined #openstack-infra | 14:01 | |
| *** berendt has quit IRC | 14:02 | |
| *** bcourt has quit IRC | 14:04 | |
| *** mmaglana has joined #openstack-infra | 14:05 | |
| *** bknudson has joined #openstack-infra | 14:06 | |
| *** talluri has joined #openstack-infra | 14:08 | |
| fungi | bo_sh: aha, so you were asking about the job-filter parameters in http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/logstash/jenkins-log-client.yaml | 14:09 |
| *** mmaglana has quit IRC | 14:10 | |
| fungi | bo_sh: that allows us to only retrieve and index particular files if the jenkins job name matches that pattern | 14:10 |
| openstackgerrit | A change was merged to openstack-infra/project-config: Run lint and syntax stackforge/puppet-* checks on trusty https://review.openstack.org/130115 | 14:11 |
| *** talluri has quit IRC | 14:11 | |
| fungi | bo_sh: so that we can skip indexing those files for other jobs where it would be a waste of resources for some reason (either because the files don't exist or they're not useful in those cases) | 14:11 |
| mtreinish | did the stable juno job patches land? Because I'm seeing the juno jobs on tempest now (but zuul is showing them as "NOT_REGISTERED") | 14:11 |
| *** talluri has joined #openstack-infra | 14:11 | |
| *** gokrokve has joined #openstack-infra | 14:11 | |
| *** talluri has quit IRC | 14:11 | |
| *** marun has joined #openstack-infra | 14:11 | |
| *** talluri has joined #openstack-infra | 14:12 | |
| fungi | mtreinish: https://review.openstack.org/129101 merged at 10:59 utc | 14:12 |
| *** salv-orlando has joined #openstack-infra | 14:13 | |
| fungi | mtreinish: there may be a small race condition on adding new jobs to zuul before any of the jenkins masters have added and registered workers for them | 14:13 |
| *** zz_dimtruck is now known as dimtruck | 14:13 | |
| *** MrTopless has joined #openstack-infra | 14:13 | |
| fungi | mtreinish: at what time were the "not_registered" results triggered? | 14:13 |
| fungi | example link? | 14:13 |
| mtreinish | fungi: nowish, thats what I'm seeing on status.o.o/zuul now | 14:14 |
| fungi | for which change? | 14:14 |
| mtreinish | fungi: 118424 is the most recent tempest change on the check pipeline | 14:14 |
| fungi | haha, i bet i know the problem | 14:15 |
| fungi | just a sec | 14:15 |
| *** nelsnelson has joined #openstack-infra | 14:15 | |
| *** gokrokve has quit IRC | 14:16 | |
| *** nosnos has quit IRC | 14:16 | |
| *** nosnos has joined #openstack-infra | 14:16 | |
| *** emagana has joined #openstack-infra | 14:16 | |
| fungi | nope, that wasn't it | 14:16 |
| *** MrTopless has left #openstack-infra | 14:17 | |
| fungi | none of these have been registered yet | 14:17 |
| *** esker has quit IRC | 14:17 | |
| *** k4n0 has quit IRC | 14:17 | |
| *** koolhead17 has quit IRC | 14:17 | |
| *** esker has joined #openstack-infra | 14:17 | |
| *** bcourt has joined #openstack-infra | 14:18 | |
| *** freemanbrandon has joined #openstack-infra | 14:19 | |
| fungi | cyeoh: i finally got around to moving the openid from your "extra" gerrit account to be associated with the account you're using, and deactivated the unused account. let me know if you spot any problems | 14:19 |
| mtreinish | mordred: I'm also seeing some pbr fails on the stable jobs for tempest: http://logs.openstack.org/06/130106/1/check/check-tempest-dsvm-full-icehouse/c130eed/logs/devstacklog.txt.gz#_2014-10-22_03_40_39_582 | 14:20 |
| dhellmann | is anyone else having trouble loading pages from gerrit? | 14:20 |
| * dhellmann asks as the page finally loads | 14:20 | |
| mtreinish | I'm guessing that it's related to me pushing out the 2 tag yesterday | 14:20 |
| *** msabramo has quit IRC | 14:21 | |
| *** nosnos has quit IRC | 14:21 | |
| *** emagana has quit IRC | 14:21 | |
| fungi | mtreinish: found the problem... jjb is broken (same issue EmilienM reported a little while ago) | 14:23 |
| fungi | dhellmann: bug 1384238 | 14:24 |
| uvirtbot | Launchpad bug 1384238 in openstack-ci "JJB: package name does not match and breaks PBR" [Undecided,Confirmed] https://launchpad.net/bugs/1384238 | 14:24 |
| EmilienM | happy to not be alone | 14:24 |
| mtreinish | fungi: ah, ok | 14:24 |
| *** emagana has joined #openstack-infra | 14:24 | |
| bo_sh | fungi: thanks! | 14:24 |
| *** hashar has quit IRC | 14:25 | |
| *** talluri has quit IRC | 14:26 | |
| *** bo_sh has quit IRC | 14:27 | |
| fungi | i'm momentarily rolling jjb back to the commit before the introduction of the --version flag on our jenkins masters so i can get these pending jobs created | 14:28 |
| *** radez is now known as radez_g0n3 | 14:29 | |
| fungi | then i'll look at proposing a revert of that commit while we figure out what it's doing wrong (or whether it's actually a bug in pbr) | 14:30 |
| *** doude has quit IRC | 14:30 | |
| *** doude_ has joined #openstack-infra | 14:30 | |
| dhellmann | fungi: hrm | 14:30 |
| *** gokrokve has joined #openstack-infra | 14:30 | |
| *** bcourt has quit IRC | 14:31 | |
| dhellmann | fungi: ayoung ran into a similar issue with version detection in python-keystone-middleware the other day | 14:31 |
| *** mmaglana has joined #openstack-infra | 14:31 | |
| morganfainberg | dhellmann, that discovery made me sad. | 14:31 |
| dhellmann | fungi: the fix was to set the name correctly in their __init__ where they get the version details (instead of using __name__ they had to use a literal string) | 14:31 |
| dhellmann | morganfainberg: well, setuptools looks for the dist not the package, so if those names don't match you have to ask for the version of the dist | 14:32 |
| morganfainberg | dhellmann, i know. | 14:32 |
| fungi | EmilienM: sounds like that ^ might take care of it then if you want to propose | 14:32 |
| morganfainberg | dhellmann, does't mean i'm happy about running into that :P | 14:32 |
| *** _nadya_ has joined #openstack-infra | 14:32 | |
| *** ChuckC has joined #openstack-infra | 14:33 | |
| dhellmann | EmilienM: if you propose a patch, I can vote on it | 14:33 |
| dhellmann | oh, well, maybe not if it's in jjb | 14:34 |
| *** emagana has quit IRC | 14:34 | |
| *** hdd has joined #openstack-infra | 14:34 | |
| EmilienM | dhellmann: fungi: Ok, I'm doing a patch now | 14:35 |
| fungi | well, you *can* vote on it | 14:35 |
| *** emagana has joined #openstack-infra | 14:35 | |
| dhellmann | EmilienM, fungi : the other option is to just use pkg_resources to get the version directly -- I don't know why jjb needs to get a dev version string from git through pbr | 14:35 |
| *** markmcclain has joined #openstack-infra | 14:35 | |
| *** msabramo has joined #openstack-infra | 14:35 | |
| dhellmann | unless we're running unreleased versions of jjb somewhere? | 14:35 |
| dhellmann | fungi: yes, well, technically :-) | 14:35 |
| fungi | a lot of people cd it and install with pip, including us | 14:36 |
| *** stevemar has joined #openstack-infra | 14:36 | |
| fungi | install from source with pip i mean | 14:36 |
| dhellmann | ok, so you'd get a version but it would be slightly wrong | 14:36 |
| *** dprince has quit IRC | 14:36 | |
| mtreinish | dhellmann: while your helping with pbr did you see the log link I posted above? :) (different issue) | 14:36 |
| dhellmann | mtreinish: looking | 14:37 |
| dhellmann | mtreinish: that looks like lifeless' semver validation logic | 14:38 |
| clarkb | why would the version be slightly wrong? | 14:38 |
| dhellmann | clarkb: because pkg_resources won't include the dev trailing bits showing which commit is being installed, will it? | 14:39 |
| dhellmann | clarkb: I guess it would, actually | 14:39 |
| mtreinish | dhellmann: yeah that's what it looked like. When I looked at the code it's probably because I pushed the 2 tag out yesteday. But, I'm not sure why it only fails on the stable/icehouse jobs | 14:39 |
| clarkb | I think it does. we do this somewhere else (zuul msybe) and it works as expected | 14:39 |
| mtreinish | I've got the setup.cfg version bump pending, but other issues in the gate keep blocking it... | 14:39 |
| *** garyh has joined #openstack-infra | 14:40 | |
| dhellmann | mtreinish: I'm not sure. lifeless is our main person on that stuff, so the best I can do is direct you to him. :-/ | 14:40 |
| *** emagana has quit IRC | 14:40 | |
| dhellmann | clarkb: yeah, I think it might get confused if you "pip install -e" and then "git pull" later but otherwise it should be right-ish | 14:40 |
| fungi | mtreinish: okay, those jobs are registered now. however they may need to be pinned to devstack-precise nodes (right now they're only being registered for devstack-trusty nodes) | 14:41 |
| fungi | clarkb: ^ ? | 14:41 |
| fungi | we were planning to run juno jobs on precise rather than trusty still, right? | 14:41 |
| clarkb | fungi juno jobs should be trusty | 14:41 |
| ttx | mordred: Are you fine with docs target being optional, or are you going to follow up on https://review.openstack.org/#/c/119875/ ? If the former, please abandon review | 14:41 |
| fungi | clarkb: mtreinish: okay, then should be fine to recheck now | 14:42 |
| clarkb | since second half of cycle was all trusty | 14:42 |
| fungi | works for me | 14:42 |
| *** ZZelle has quit IRC | 14:42 | |
| *** bcourt has joined #openstack-infra | 14:42 | |
| mtreinish | fungi: ok cool thanks. I've got to wait for the results to come back before another recheck will work right? | 14:42 |
| *** tdasilva has joined #openstack-infra | 14:43 | |
| *** ZZelle has joined #openstack-infra | 14:43 | |
| fungi | mtreinish: yeah, unless you make a new patchset by fiddling the commit message or something | 14:43 |
| *** emagana has joined #openstack-infra | 14:43 | |
| *** teran has quit IRC | 14:43 | |
| mtreinish | fungi: oh, yeah that's a good idea, better than waiting ~40min | 14:44 |
| *** wenlock has joined #openstack-infra | 14:45 | |
| *** SumitNaiksatam has quit IRC | 14:46 | |
| mtreinish | fungi: could you promote: https://review.openstack.org/#/c/130099/ I think it fixes the pbr semver issue on the tempest stable jobs I was just asking dhellmann about | 14:46 |
| *** radez_g0n3 is now known as radez | 14:46 | |
| fungi | mtreinish: sure, no problem | 14:47 |
| *** yolanda_ has quit IRC | 14:47 | |
| *** kim___ has quit IRC | 14:47 | |
| mtreinish | fungi: cool, thanks | 14:47 |
| *** yolanda_ has joined #openstack-infra | 14:47 | |
| *** SumitNaiksatam has joined #openstack-infra | 14:47 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack/requirements: Enforce requirements check only for registered projects https://review.openstack.org/130210 | 14:48 |
| fungi | mtreinish: this is causing jobs to fail for other projects, or just for changes proposed to tempest? | 14:49 |
| dims | mtreinish: thanks again for the pointers! | 14:50 |
| mtreinish | dhellmann: oh wait, are we supposed to drop version from setup.cfg (re-reading the pbr code) | 14:50 |
| mtreinish | dims: sure, np | 14:50 |
| fungi | mtreinish: i think if you're using normal postversioning patterns you don't put a version in setup.cfg at all | 14:51 |
| mtreinish | fungi: I've just seen it on the stable tempest jobs on proposed tempest commits | 14:51 |
| mtreinish | fungi: ok, then let me respin that patch to drop the version from setup.cfg | 14:51 |
| clarkb | dims dhellman I am -1 on that approach | 14:51 |
| mtreinish | fungi: because that's different then how it's been done in tempest for a while | 14:51 |
| clarkb | ccomment left in change | 14:51 |
| *** teran has joined #openstack-infra | 14:52 | |
| *** andreaf has quit IRC | 14:52 | |
| *** tonytan4ever has joined #openstack-infra | 14:53 | |
| fungi | dims: the recommendation on the ml was to stop running that job on projects which don't need it | 14:53 |
| fungi | dims: not to make that job a no-op so you can run it and have it do nothing | 14:53 |
| mtreinish | fungi: oh, that's probably because we just copied it from nova which handles releases differently now :) | 14:54 |
| *** katyafervent is now known as katyafervent_awa | 14:54 | |
| dims | fungi: 2 issues, update.py is in dsvm jobs, see example failure - http://logs.openstack.org/65/130065/1/check/check-tempest-dsvm-f20-docker/f9000d4/devstacklog.txt.gz yes, i have another review to remove the gate requirements job | 14:55 |
| fungi | dims: oh, this is to address the behavior in devstack, not to solve for teh requirements enforcement | 14:55 |
| *** teran_ has joined #openstack-infra | 14:55 | |
| dims | fungi: yep | 14:55 |
| clarkb | ya so we should fix in devstack | 14:55 |
| fungi | i thought there was already an envvar in devstack to turn off that requirements rewriting | 14:56 |
| dims | clarkb: yes, there is. question is how do we specify it for say nova-docker | 14:56 |
| fungi | looking for an example now | 14:56 |
| dims | NON_STANDARD_REQS flag | 14:57 |
| dhellmann | fungi: the env var only works if the sync job is not in place. dims' change allows for both, so they can stay up to date on requirements that other projects are using and not fail because they have extras | 14:57 |
| *** teran has quit IRC | 14:57 | |
| clarkb | nononono | 14:57 |
| * dhellmann thinks we're doing requirements enforcement in too many places | 14:57 | |
| clarkb | you cant have both | 14:57 |
| dhellmann | clarkb: why? | 14:57 |
| *** pfallenop has quit IRC | 14:58 | |
| clarkb | because you either get the enforcement or you dont | 14:58 |
| dims | fungi: all i did was check against projects.txt to do the same thing as that flag NON_STANDARD_REQS override | 14:58 |
| clarkb | there is no only enforce things in a subset feature | 14:58 |
| clarkb | the idea here is one global list | 14:58 |
| dhellmann | clarkb: ok, then we need to allow all projects that are going to be tested under devstack configurations, whether integrated or not, to add requirements to that list | 14:59 |
| dims | clarkb: my previous proposal was to allow nova-docker (example) to set NON_STANDARD_REQS in its jobs. then devstack gate wrap to pass that variable onto devstack and then devstack runs update.py with the correct variable setting | 14:59 |
| clarkb | dhellmann no | 14:59 |
| mtreinish | hmm, dropping the version for a commit makes the intermediate version 2.1.XXX but with version = 3 it's 3.dev1.XXXX (which feels like the better version string) | 14:59 |
| clarkb | you just only sync projects in that list | 14:59 |
| mtreinish | fungi: ^^^ so I think promoting that change is fine | 14:59 |
| dhellmann | clarkb: ok, what's the right approach for this project then? | 14:59 |
| fungi | there is requirements _enforcement_ in one place (an optional job that runs when you modify your requirements.txt or test-requirements.txt) | 14:59 |
| dhellmann | fungi: it's also being enforced during the devstack run | 15:00 |
| *** habib has joined #openstack-infra | 15:00 | |
| fungi | what devstack has is a mechanism to make sure that all projects it installs use a common set of versions of requirements because otherwise it would go insane | 15:00 |
| dhellmann | that's a side-effect, but it's happening | 15:00 |
| dhellmann | right | 15:00 |
| clarkb | I think the idea of not updating projects in the list is good | 15:00 |
| *** amotoki has joined #openstack-infra | 15:00 | |
| dims | clarkb: fungi: here's the readme that explicitly says enforcement only if listed in projects.txt - http://git.openstack.org/cgit/openstack/requirements/tree/README.rst#n60 | 15:00 |
| clarkb | its just happening at the wrong layer | 15:00 |
| *** thedodd has quit IRC | 15:00 | |
| *** HeOS has quit IRC | 15:01 | |
| mtreinish | fungi: for the future I just need to remember to bump the version string before tagging :) | 15:01 |
| fungi | dims: we need to update that document. i think that's a hold-over from when we ran a partial pypi mirror | 15:01 |
| clarkb | because if you run a command to do something in the unix world and it doesnt do that thing and it returns 0 that is a bug | 15:02 |
| fungi | dims: oh, it's talking about the requirements check job | 15:02 |
| clarkb | so we should instead simply not run that command if we dont want updates | 15:02 |
| fungi | yes that's an optional job. you either add your project in projects.txt and run the enforcement job or you don't. it's not describing the side effect in devstack under discussiojn | 15:02 |
| dhellmann | clarkb: that's not what he's doing though. He's making it possible to sync the common requirements and ignore extra requirements. Why is that not good? | 15:03 |
| dims | fungi: see http://logs.openstack.org/65/130065/1/check/check-tempest-dsvm-f20-docker/f9000d4/devstacklog.txt.gz | 15:03 |
| clarkb | because we have a single global requirements list | 15:03 |
| *** dane_leblanc has quit IRC | 15:03 | |
| dims | fungi: i need a dsvm job that does not enforce requirements | 15:03 |
| clarkb | its not global + extra | 15:03 |
| dims | since nova-docker is not integrated | 15:03 |
| fungi | dims: i see it... that's not the requirements check job described in the section of the requirements readme you pointed to though | 15:04 |
| clarkb | you are either part of the enforcement or not | 15:04 |
| dims | fungi: i pointed to both | 15:04 |
| dhellmann | clarkb: to which we have told stackforge projects they cannot add anything, and so they have to keep themselves up to date by hand, and this project is now going to be running tests under devstack so we want the requirements to be common so ... | 15:04 |
| dims | https://review.openstack.org/#/c/130192/ is for the gate requirements job | 15:04 |
| *** ilyashakhat has joined #openstack-infra | 15:04 | |
| clarkb | dhellmann no they can have auto updates | 15:04 |
| *** pfallenop has joined #openstack-infra | 15:04 | |
| clarkb | they just have to share the list | 15:04 |
| clarkb | or | 15:04 |
| *** signed8bit has quit IRC | 15:04 | |
| clarkb | they do not share the list | 15:05 |
| dhellmann | clarkb: that was not what we told solum | 15:05 |
| clarkb | what? | 15:05 |
| *** otherwiseguy has joined #openstack-infra | 15:05 | |
| mtreinish | dhellmann: yes it was, the issue there was they wanted extra stuff that had not place in the g-r list | 15:05 |
| clarkb | we tell projects you enforce or dont | 15:05 |
| dhellmann | we told solum they could not add things to the requirements list because they are not integrated | 15:05 |
| dhellmann | right | 15:05 |
| clarkb | thats orthogonal | 15:05 |
| fungi | okay, so the underlying issue is not actually the requirements sync in devstack... it's the extra check which was added to the sync routine to bail if you add a requirement not in the requirements list | 15:05 |
| *** emagana has quit IRC | 15:05 | |
| *** homeless has joined #openstack-infra | 15:05 | |
| dims | correct | 15:05 |
| *** emagana has joined #openstack-infra | 15:06 | |
| *** tomoe_ has joined #openstack-infra | 15:06 | |
| clarkb | solum/nova-docker can either be part of the same stuff that ties everyone together or not | 15:06 |
| clarkb | there is no middle ground this is intentional | 15:06 |
| dhellmann | clarkb: we tell them it's on or off, but then we have rejected adding requirements for some projects because they aren't in the integrated release | 15:06 |
| mtreinish | clarkb: yes, which is what I've been saying. There is just a bug in devstack (or the nova-docker devstack stuff) which is blocking them | 15:07 |
| clarkb | whether or not reqs core approves changes to the global list for them is a different issue | 15:07 |
| dhellmann | and so something that is not in the integrated releaase cannot have the checks on if they need things other parts of the integrated release don't need | 15:07 |
| dhellmann | clarkb: well, no, it's not -- it's all part of the same requirements management system | 15:07 |
| *** IAX has joined #openstack-infra | 15:07 | |
| * dims hangs back to watch :) | 15:07 | |
| clarkb | I have long said add those deps as long as they meet the criteria | 15:08 |
| *** iax7 has joined #openstack-infra | 15:08 | |
| clarkb | but I am in the minority | 15:08 |
| *** hemnass is now known as hemna | 15:08 | |
| *** yolanda_ has quit IRC | 15:08 | |
| dhellmann | clarkb: that is not our current policy, though. I tend to agree with you, but that's not what we're doing. | 15:08 |
| clarkb | I see py3k + license compat + support as criteria | 15:08 |
| clarkb | not is stackforge | 15:08 |
| *** pinajo has joined #openstack-infra | 15:08 | |
| *** pinajo has quit IRC | 15:09 | |
| *** IAX has quit IRC | 15:09 | |
| *** iax7 has quit IRC | 15:09 | |
| *** baoli_ has joined #openstack-infra | 15:09 | |
| fungi | so it looks like openstack/requirements update.py needs a switch around the bit at the end of _sync_requirements_file() which can be flipped on invocation, and then devstack should be flipping that switch if requested | 15:09 |
| *** gokrokve_ has joined #openstack-infra | 15:09 | |
| *** baoli has quit IRC | 15:09 | |
| *** tomoe has quit IRC | 15:09 | |
| dims | fungi: yep | 15:10 |
| clarkb | devstack should just not call update.py if we dont want updates | 15:10 |
| *** pfallenop has quit IRC | 15:10 | |
| *** reed has joined #openstack-infra | 15:10 | |
| dhellmann | clarkb: we do want updates, though | 15:10 |
| dhellmann | clarkb: we need devstack to update everything it is installing, right? | 15:10 |
| fungi | clarkb: er, except that it wants to coinstall your project with the rest of devstack... uninstall loops and so forth await if it does not | 15:10 |
| clarkb | dhellmann no | 15:10 |
| *** IAX has joined #openstack-infra | 15:10 | |
| dhellmann | clarkb: how do you see this working then? | 15:10 |
| *** mrmartin has joined #openstack-infra | 15:10 | |
| *** rmcall has joined #openstack-infra | 15:11 | |
| clarkb | fungi but they are subverting the usefullness of that anyways so it doesnt matter | 15:11 |
| clarkb | as soon as you put deps not in global reqs this safety net has a hole | 15:11 |
| *** dims_ has joined #openstack-infra | 15:11 | |
| *** emagana has quit IRC | 15:11 | |
| *** signed8bit has joined #openstack-infra | 15:11 | |
| *** iax7 has joined #openstack-infra | 15:11 | |
| *** mpaolino_ has joined #openstack-infra | 15:11 | |
| fungi | clarkb: i think it's probably still a valid test of "can your thing be installed and work with the rest of devstack as long as your additional dependencies are also satisfied" | 15:12 |
| *** mpaolino_ has quit IRC | 15:12 | |
| *** gokrokve_ has quit IRC | 15:12 | |
| *** mpaolino has quit IRC | 15:12 | |
| clarkb | if grep $PROJECT projects.txt ; the update.py $PROJECT ; else #noop ; fi | 15:12 |
| clarkb | fungi that is still tru with ^ | 15:12 |
| *** gokrokve_ has joined #openstack-infra | 15:12 | |
| mtreinish | why not just manually install the nova-docker requirements in it's devstack hook and ignore the devstack mechanism for it that has the check? | 15:12 |
| clarkb | but it avoids complicating a simple tool that does one thing | 15:12 |
| dhellmann | dims: where's your original patch to add the things you need to the global requirements list? | 15:12 |
| *** gokrokve has quit IRC | 15:13 | |
| clarkb | mtreinish well that makes it harder for others to install | 15:13 |
| *** pinajo has joined #openstack-infra | 15:13 | |
| *** JayJ has joined #openstack-infra | 15:14 | |
| *** pinajo has quit IRC | 15:14 | |
| *** iax7 has quit IRC | 15:14 | |
| *** dims__ has joined #openstack-infra | 15:14 | |
| fungi | clarkb: oh, i see. when setting up devstack, flatten the reqs of all the enforced projects being installed but don't flatten the reqs of the unofficial one(s) | 15:14 |
| *** IAX has quit IRC | 15:14 | |
| *** david-lyle has joined #openstack-infra | 15:14 | |
| clarkb | yup | 15:14 |
| *** gary-smith has left #openstack-infra | 15:14 | |
| mtreinish | clarkb: yeah it probably would | 15:14 |
| *** IAX has joined #openstack-infra | 15:15 | |
| fungi | so not saying don't run the update routine at all when testing an unofficial project, just don't flatten that particular project's requirements but flatten the rest | 15:15 |
| *** dims has quit IRC | 15:15 | |
| fungi | that seems like a straightforward compromise | 15:15 |
| *** hdd has quit IRC | 15:15 | |
| *** IAX has quit IRC | 15:16 | |
| dims__ | fungi: clarkb: reply on ML please? | 15:16 |
| *** apevec has quit IRC | 15:16 | |
| fungi | dims__: once my next meeting wraps up, can do | 15:16 |
| *** dims_ has quit IRC | 15:16 | |
| dims__ | thanks much! | 15:16 |
| openstackgerrit | Doug Hellmann proposed a change to openstack/requirements: Explicitly state which projects can add requirements https://review.openstack.org/130245 | 15:17 |
| *** gokrokve_ has quit IRC | 15:17 | |
| *** dane_leblanc has joined #openstack-infra | 15:17 | |
| dhellmann | clarkb: ^^ | 15:17 |
| *** Sukhdev has joined #openstack-infra | 15:18 | |
| dims__ | dhellmann: huge +1 from me :) | 15:18 |
| *** pcrews has joined #openstack-infra | 15:18 | |
| dhellmann | dims__: if you unabandon https://review.openstack.org/#/c/128746/1 and rebase on top of ^^ I'll +2 | 15:18 |
| *** unicell1 has quit IRC | 15:18 | |
| dhellmann | and then, assuming the "rule change" is approved we can just drop all of the other changes | 15:19 |
| dims__ | dhellmann: with pleasure | 15:19 |
| *** david-lyle has quit IRC | 15:19 | |
| dhellmann | dims__: I don't know who the PTL is for requirements. Maybe ttx? We should make sure that rule change sees wide review, though. | 15:19 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack/requirements: docker-py dependency for nova-docker https://review.openstack.org/128746 | 15:20 |
| fungi | dhellmann: http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n312 agrees with you | 15:20 |
| *** hdd has joined #openstack-infra | 15:20 | |
| dhellmann | fungi: thanks! my memory worked once today. I hope I have nothing more important to try to remember. | 15:21 |
| mtreinish | dhellmann: doesn't that conflict with the dual purpose of having the reqs be a cannonical list for packagers to have openstack you need g-r | 15:21 |
| *** MarkAtwood has quit IRC | 15:21 | |
| clarkb | no because apparently its never been that list | 15:21 |
| *** pfallenop has joined #openstack-infra | 15:22 | |
| mtreinish | ok, wfm | 15:22 |
| dhellmann | it hasn't been that list, but we were trying to make it be that list, which means yes there is a conflict but we have to pick or allow changes in implementation | 15:22 |
| *** rlandy has quit IRC | 15:23 | |
| dims__ | ttx: please see https://review.openstack.org/#/c/130245/ :) | 15:23 |
| ttx | yeah, we'll need wider review on that one | 15:25 |
| ttx | will add to next week cross-meeting agenda | 15:25 |
| dims__ | ttx: thanks | 15:26 |
| fungi | yeah, i'm personally fine if the community wants global-requirements.txt to be "here's the list of libraries it's okay for official projects to depend on, even if they don't have a need for them at this point in time" and then let unofficial projects request additions to that list | 15:26 |
| clarkb | the reason I am fine with it is we dont really trim the list ever | 15:27 |
| fungi | yep | 15:27 |
| clarkb | so it already contains a superset of the actually needed reqs | 15:27 |
| *** habib has quit IRC | 15:27 | |
| clarkb | so adding a few here and there isnt detrimental | 15:27 |
| clarkb | as long as they meet the general criteria we hold all deps to | 15:27 |
| fungi | and the actual list of what libs you need to support openstack, or some subset thereof, is defined within the projects themselves | 15:28 |
| fungi | but yes, "i need this dependency added to the list even though it doesn't meet criteria for an official project dependency but that's okay because my project isn't official" isn't going to fly | 15:29 |
| fungi | since once something's in the list, our enforcement tooling isn't going to discriminate which projects can or can't start to rely on it | 15:29 |
| *** pfallenop has quit IRC | 15:31 | |
| openstackgerrit | Mikhail S Medvedev proposed a change to openstack-infra/devstack-gate: Rename config => system-config in devstack-gate https://review.openstack.org/130254 | 15:31 |
| *** zaphod42 has joined #openstack-infra | 15:32 | |
| *** jerryz has quit IRC | 15:35 | |
| *** bdpayne has joined #openstack-infra | 15:37 | |
| *** oramraz has quit IRC | 15:37 | |
| *** msabramo has quit IRC | 15:39 | |
| *** krtaylor has quit IRC | 15:40 | |
| *** ayoung is now known as ayoung-afk | 15:40 | |
| *** msabramo has joined #openstack-infra | 15:43 | |
| *** david-lyle has joined #openstack-infra | 15:43 | |
| *** sdake_ has quit IRC | 15:44 | |
| *** pfallenop has joined #openstack-infra | 15:46 | |
| *** pfallenop has joined #openstack-infra | 15:46 | |
| dhellmann | fungi: I was going to send email to the -dev list about python 2.6 support in oslo. We need to keep it until the stable branches drop it, right? What release is that, L or M? | 15:48 |
| * dhellmann is following his "write it down" policy | 15:48 | |
| *** kgiusti has quit IRC | 15:48 | |
| fungi | dhellmann: until juno reaches eol | 15:48 |
| fungi | dhellmann: but more specifically, only thinks juno and icehouse depend on need to be tested against 2.6 | 15:49 |
| fungi | dhellmann: so if you spin out new libs only used by kilo, no need for them to work with 2.6 | 15:49 |
| dhellmann | sure | 15:49 |
| dhellmann | well, unless they're used by the clients | 15:49 |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/jenkins-job-builder: Import import pbr.version in __init__ https://review.openstack.org/130260 | 15:49 |
| EmilienM | dhellmann: not sure I did it correctly ^ | 15:50 |
| *** patrickeast has joined #openstack-infra | 15:51 | |
| *** pfallenop has quit IRC | 15:51 | |
| fungi | dhellmann: mmm... tricky bit there... yeah we did separately say we'd support the clients on 2.6 for the lifetime of juno | 15:52 |
| *** cody-somerville has quit IRC | 15:52 | |
| *** mwagner_lap has quit IRC | 15:55 | |
| dhellmann | fungi: is this right? https://etherpad.openstack.org/p/oslo-26-support-policy | 15:55 |
| *** krtaylor has joined #openstack-infra | 15:56 | |
| *** sdake_ has joined #openstack-infra | 15:56 | |
| *** skolekonov has quit IRC | 15:57 | |
| *** groknix has joined #openstack-infra | 15:57 | |
| *** dustins has quit IRC | 15:58 | |
| *** jlibosva has quit IRC | 15:59 | |
| *** mestery has quit IRC | 15:59 | |
| *** shashankhegde has joined #openstack-infra | 15:59 | |
| *** dustins has joined #openstack-infra | 16:02 | |
| *** larsks has quit IRC | 16:03 | |
| *** larsks has joined #openstack-infra | 16:03 | |
| dhellmann | EmilienM: I left some comments, but I think all you need to do is change the argument to the existing call to VersionInfo in the version module | 16:04 |
| *** mestery has joined #openstack-infra | 16:04 | |
| * dhellmann heads to lunch | 16:04 | |
| marcusvrn_ | Hi guys...just a question....all CIs use devstack/master to run? In my CI, I'm using it and it was working well, but after last merge (yesterday), my CI returns fail results | 16:05 |
| *** IAX has joined #openstack-infra | 16:05 | |
| *** jcoufal has quit IRC | 16:05 | |
| clarkb | marcusvrn_: most jobs do yes | 16:06 |
| *** mrmartin has quit IRC | 16:07 | |
| *** iax7 has joined #openstack-infra | 16:07 | |
| *** iax7 has quit IRC | 16:07 | |
| *** IAX has quit IRC | 16:08 | |
| *** dprince has joined #openstack-infra | 16:08 | |
| *** IAX has joined #openstack-infra | 16:08 | |
| *** IAX has quit IRC | 16:08 | |
| *** mancdaz is now known as mancdaz_away | 16:10 | |
| david-lyle | The Horizon team needs a place to store images (wireframes and visual mocks). Up to this point we've relied on free services like imgur or personal servers; and we've hosted an askbot server for UX. None of these work very well. One big concern is files being removed. We'd like a better long term solution. Do you all have any ideas? | 16:11 |
| *** arnaud__ has joined #openstack-infra | 16:12 | |
| anteaya | david-lyle: hi | 16:12 |
| david-lyle | Something like paste.o.o for images would be awesome, but I've yet to find it | 16:12 |
| david-lyle | hi anteaya | 16:12 |
| anteaya | mordred has been wanting us to set up an instance of seafiles or owncloud or something | 16:12 |
| *** cody-somerville has joined #openstack-infra | 16:13 | |
| anteaya | and I haven't had the time to do it yet | 16:13 |
| anteaya | david-lyle: would either one of them work? | 16:13 |
| anteaya | it isn't publicly accessible via a url like imgur I don't think | 16:13 |
| anteaya | I don't know | 16:13 |
| david-lyle | even swift with public URLs would work | 16:14 |
| *** pfallenop has joined #openstack-infra | 16:14 | |
| anteaya | hey swift with public URLs, what an idea | 16:14 |
| david-lyle | an ideal solution would display the images as well, but I would be happy with just a stable host | 16:14 |
| anteaya | david-lyle: jhesketh has been doing most of our work with swift right now | 16:14 |
| *** bdpayne has quit IRC | 16:14 | |
| anteaya | david-lyle: any chance you could try to find him once he is awake (austraila time) | 16:15 |
| marcusvrn_ | clarkb: hmm...are all CIs OK after last devstack merge? | 16:15 |
| anteaya | david-lyle: he might have some thoughts | 16:15 |
| david-lyle | these links would be something posted in blueprints for Horizon | 16:15 |
| david-lyle | anteaya, I can certainly ping him when he's around | 16:15 |
| dims__ | anteaya: when you get a chance, please take a look at https://review.openstack.org/#/c/130192/ - removing a job from nova-docker | 16:16 |
| clarkb | marcusvrn_: they seem to be. you can check at http://status.openstack.org/zuul | 16:16 |
| anteaya | dims__: kk | 16:16 |
| clarkb | marcusvrn_: what change was made that is breaking you? | 16:16 |
| anteaya | dims__: can you link to the ml discussion in a comment on the patch | 16:17 |
| anteaya | the commit message alludes to it | 16:17 |
| *** emagana has joined #openstack-infra | 16:17 | |
| anteaya | but it would save me time looking | 16:17 |
| * anteaya returns to cinder meeting | 16:17 | |
| *** jistr has quit IRC | 16:18 | |
| *** pfallenop has quit IRC | 16:19 | |
| *** denis_makogon has quit IRC | 16:19 | |
| marcusvrn_ | clarkb: this change 'Merge "Move generation of baremetal VM names to lib/ironic"' is breaking the CI. After this change...all jobs failed and the tempest error is "no valid host was found" | 16:19 |
| *** pfallenop has joined #openstack-infra | 16:19 | |
| *** IAX has joined #openstack-infra | 16:20 | |
| *** gokrokve has joined #openstack-infra | 16:20 | |
| *** Longgeek has quit IRC | 16:20 | |
| *** ZZelle_ has joined #openstack-infra | 16:22 | |
| *** jpich has quit IRC | 16:22 | |
| *** esker has quit IRC | 16:22 | |
| dims__ | anteaya: ack | 16:22 |
| anteaya | dims__: thanks | 16:23 |
| *** esker has joined #openstack-infra | 16:23 | |
| clarkb | FYI I just manually ran manage-projects for stackforge/puppet-sahara as its acls were wrong | 16:24 |
| clarkb | not sure why they are wrong yet. was hoping a manual run would spit out errors but it was fine | 16:24 |
| anteaya | interesting | 16:24 |
| anteaya | the acl .config file was wrong? | 16:24 |
| anteaya | or gerrit didn't have the correct values from the .config file? | 16:25 |
| clarkb | it had the old sahara specific one but was set in projects.yaml to use the generic puppet module one | 16:25 |
| clarkb | everything was fine it just worked | 16:25 |
| clarkb | so not sure where jeepyb broke updating it | 16:25 |
| anteaya | hmmmm | 16:25 |
| anteaya | because pleia2 has a patch up to point more repos at the group puppet.config acl | 16:26 |
| anteaya | wondering if something similar will happen when it merges | 16:26 |
| clarkb | it may. will probably help debugging if it happens | 16:26 |
| clarkb | as we can investigate it as it happens rather than a while later | 16:26 |
| anteaya | do we have enough logging set to collect useful info for debugging? | 16:27 |
| clarkb | I think so, just not this far out into the future | 16:27 |
| *** esker has quit IRC | 16:27 | |
| anteaya | I understand everything before the comma | 16:28 |
| *** MaxV has quit IRC | 16:28 | |
| anteaya | and don't parse the last part | 16:28 |
| clarkb | the sahara puppet thing was updated a while back best I can tell | 16:28 |
| *** radez is now known as radez_g0n3 | 16:28 | |
| *** jp_at_hp has quit IRC | 16:28 | |
| clarkb | I don't think digging for logs from back then will be fruitful | 16:28 |
| anteaya | ah | 16:28 |
| zaro | morning | 16:29 |
| anteaya | okay yes, we can watch as it merges and comb the logs for useful bits as it happens | 16:29 |
| anteaya | morning zaro | 16:29 |
| *** derekh has quit IRC | 16:29 | |
| *** palar has joined #openstack-infra | 16:30 | |
| *** markmcclain has quit IRC | 16:31 | |
| ZZelle_ | Hi everyone, | 16:31 |
| *** mrmartin has joined #openstack-infra | 16:32 | |
| ZZelle_ | Is it normal to get -1 from Jenkins when job state is success or NOT_REGISTERED (https://review.openstack.org/130232) | 16:33 |
| ZZelle_ | ? | 16:33 |
| clarkb | ZZelle_: yes. zuul was unable to run the job so it -1'd the change | 16:33 |
| clarkb | so the -1 aspect is "normal" | 16:33 |
| clarkb | but NOT_REGISTERED isn't super normal. I am looking | 16:34 |
| mtreinish | clarkb, dtroyer: if you've got a sec can you take a look at/update: https://wiki.openstack.org/wiki/QA/releases I'm trying to document the release process we use for all the qa projects | 16:34 |
| clarkb | fungi: mtreinish ^ that looks like the thing you debugged early today | 16:34 |
| mtreinish | clarkb: there was a jjb issue this morning with adding the stable juno jobs | 16:34 |
| clarkb | mtreinish: and it is better now? | 16:34 |
| ZZelle_ | clarkb, it's linked to zuul restart? | 16:34 |
| mtreinish | clarkb: should ber | 16:34 |
| mtreinish | /ber/be | 16:34 |
| cody-somerville | What was the jjb issue this morning? | 16:34 |
| clarkb | ZZelle_: no in this case we didn't add the job to jenkins before trying to run the job | 16:34 |
| clarkb | ZZelle_: its better now | 16:34 |
| *** IAX has quit IRC | 16:34 | |
| ZZelle_ | clarkb, just recheck? | 16:35 |
| clarkb | cody-somerville: change to jjb to add a --version option broke jjb | 16:35 |
| clarkb | ZZelle_: yes | 16:35 |
| mtreinish | ZZelle_: just recheck the job it should work now | 16:35 |
| ZZelle_ | mtreinish, clarkb, thanks for your responses :) | 16:35 |
| clarkb | ZZelle_: do let us know if you see NOT_REGISTERED in the future | 16:35 |
| clarkb | it is likely an issue that needs to be corrected | 16:36 |
| ZZelle_ | clarkb, ok | 16:36 |
| *** arxcruz has quit IRC | 16:36 | |
| fungi | clarkb: ZZelle_: yep, recheck should solve those now | 16:36 |
| *** thedodd has joined #openstack-infra | 16:36 | |
| fungi | though we should avoid approving any more job additions until we either revert the broken commit in jjb or approve EmilienM's fix | 16:36 |
| *** tonytan4ever has quit IRC | 16:37 | |
| clarkb | rgr /me looks for EmilienM's fix | 16:37 |
| *** yolanda has joined #openstack-infra | 16:37 | |
| dtroyer | mtreinish: added to the todo list… | 16:37 |
| clarkb | do we know why CI didn't catch this? I checked the --version change and jjb ran there | 16:37 |
| *** packet has joined #openstack-infra | 16:38 | |
| mtreinish | dtroyer: cool thanks, I had no idea what you do for a devstack release so that section is pretty much blank :) | 16:38 |
| openstackgerrit | Ian Cordasco proposed a change to openstack/requirements: Freeze mccabe's version for all projects https://review.openstack.org/130273 | 16:39 |
| *** r-daneel has joined #openstack-infra | 16:39 | |
| dtroyer | basically just like grenade, or maybe grande is like devstack? | 16:39 |
| fungi | clarkb: i think it has to do with running earlier versions of setuptools maybe | 16:39 |
| clarkb | ohhhh | 16:39 |
| clarkb | yup that sounds likely | 16:40 |
| fungi | clarkb: we're running into it on teh jenkins masters where they have ubuntu precise packaged setuptools/pkg-config | 16:40 |
| fungi | old "distribute" setuptools | 16:40 |
| *** bdpayne has joined #openstack-infra | 16:40 | |
| fungi | runtime pbr with distribute is bad hudu | 16:41 |
| *** patrickeast_ has joined #openstack-infra | 16:41 | |
| *** IAX has joined #openstack-infra | 16:41 | |
| yolanda | clarkb, we had same issue in our systems, i had to comment the buggy line reporting version | 16:41 |
| fungi | hoodoo | 16:41 |
| yolanda | do you want me to help debug? | 16:41 |
| clarkb | mtreinish: the grenade portion of that wiki doc look good | 16:41 |
| mtreinish | dtroyer: probably grenade is just like devstack :) | 16:41 |
| clarkb | yolanda: there is a fix up https://review.openstack.org/#/c/130260/1 but it has a -1 because it isn't quite complete | 16:42 |
| *** bdpayne has quit IRC | 16:42 | |
| fungi | clarkb: should we take over the fix or just revert the --version option commit for now? | 16:43 |
| clarkb | fungi: why don't we take over the fix. | 16:43 |
| *** bradjones has quit IRC | 16:43 | |
| zaro | clarkb: i don't see how --version broke anything | 16:43 |
| *** AJaeger_ has joined #openstack-infra | 16:43 | |
| *** patrickeast has quit IRC | 16:43 | |
| *** patrickeast_ is now known as patrickeast | 16:43 | |
| *** bdpayne has joined #openstack-infra | 16:43 | |
| *** iax7 has joined #openstack-infra | 16:43 | |
| clarkb | zaro: https://review.openstack.org/#/c/130260/1 | 16:43 |
| *** iax7 has quit IRC | 16:44 | |
| *** iax7 has joined #openstack-infra | 16:44 | |
| yolanda | clarkb, fungi, nice to know, we were wondering if problem was in our deployment | 16:44 |
| yolanda | i can help you with testing if needed | 16:44 |
| clarkb | zaro: version.py uses the wrong name when calling VersionInfo | 16:45 |
| *** bdpayne_ has joined #openstack-infra | 16:46 | |
| *** ildikov has joined #openstack-infra | 16:47 | |
| clarkb | I am testing dhellmann's suggestion | 16:47 |
| clarkb | (on new setuptools) | 16:47 |
| clarkb | dhellmann: you are correct that it drops the sha1 data from the version | 16:48 |
| clarkb | I really thought it wouldn't do that so this code is buggy anyways | 16:48 |
| clarkb | and looking at the zuul status page it drops the sha1 too | 16:49 |
| *** bdpayne has quit IRC | 16:49 | |
| fungi | clarkb: are you currently running gertty from a release tag or an untagged commit? if the latter, you can see that same behavior on the help dialog title line | 16:49 |
| clarkb | so while I thought that was doing the other thing it does indeed drop the sha1 | 16:49 |
| clarkb | fungi: oh ok so maybe we can live with this | 16:50 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/jenkins-job-builder: Get verison info for jenkins-job-builder https://review.openstack.org/130260 | 16:50 |
| zaro | i don't even get why the fix is in __init__.py when the same thing is already in version.py? | 16:50 |
| clarkb | there is dhellmann's fix | 16:50 |
| clarkb | zaro: it shouldn't be | 16:50 |
| clarkb | zaro: dhellmann's comments explain all this. the patch I just fixed should be the fix we want according to dhellmann | 16:50 |
| *** teran_ has quit IRC | 16:51 | |
| fungi | i'll test it now | 16:51 |
| zaro | clarkb: ok, that looks better. | 16:53 |
| fungi | seems to solve the issue | 16:53 |
| fungi | i've +2'd. i'll let zaro approve when he's satisfied | 16:53 |
| *** pinajo has joined #openstack-infra | 16:53 | |
| *** mrmartin has quit IRC | 16:54 | |
| fungi | yolanda: ^ feel like testing https://review.openstack.org/130260 patchset 2? | 16:54 |
| *** micah has joined #openstack-infra | 16:54 | |
| micah | i've been trying out gertty, and I've been unable to get it to show me review requests that have already been merged... is there a way to do that? | 16:55 |
| *** talluri has joined #openstack-infra | 16:55 | |
| fungi | micah: it allows you to feed it arbitrary gerrit query strings | 16:55 |
| fungi | micah: ctrl-o | 16:55 |
| *** mwagner_lap has joined #openstack-infra | 16:56 | |
| *** jistr has joined #openstack-infra | 16:56 | |
| clarkb | zaro: and it looks like http://tarballs.openstack.org/ci/gearman-plugin/gearman-plugin-0.1.0.1.fa5f083.hpi is the jenkins plugin version we want with the tempoffline fixes? | 16:56 |
| fungi | micah: for example, you can enter "status:merged project:stackforge/gertty" (without the quotes) and it will give you the changes matching that search | 16:56 |
| clarkb | zaro: I can put 06 and 07 in shutdown mode now if we are ready to do this again | 16:56 |
| zaro | clarkb: lets do it! | 16:57 |
| *** esker has joined #openstack-infra | 16:57 | |
| clarkb | yolanda: in other news. Did you see https://review.openstack.org/#/c/130112/ | 16:57 |
| clarkb | yolanda: we need to make nodepool understand that a label's various underlying images can be built in different ways | 16:57 |
| zaro | +2 for that version thing. should we wait for jenkins results before apprv? | 16:57 |
| clarkb | fungi: any oppposition to shutdown moding jenkins06 and jenkins07 to make another pass at a gearman-plugin upgrade? | 16:58 |
| fungi | zaro: if jobs aren't happy with that change it won't merge anyway | 16:58 |
| clarkb | mordred: jeblair SergeyLukjanov ^ | 16:58 |
| zaro | fungi: good point, i'll just apprv than | 16:58 |
| clarkb | fungi: well we do run the one non voting job but other than that it should be fine | 16:58 |
| yolanda | looking | 16:58 |
| fungi | clarkb: fine by me--you've got the regression identified and solved this time? | 16:59 |
| clarkb | fungi: ya zaro pushed a fix | 16:59 |
| zaro | yolanda: ok will wait for your vote | 16:59 |
| clarkb | fungi: and I will be hyper vigilant for the specific failure mode | 16:59 |
| fungi | clarkb: zaro: and it did the right thing on jenkins-dev? | 16:59 |
| micah | fungi: hm. I do that on my gerrit and I get two results, but in the web interface, there are 5 | 16:59 |
| fungi | if so, fine by me | 16:59 |
| yolanda | zaro, ok, testing in our instance | 16:59 |
| * SergeyLukjanov reading backlog for the last 3 hours | 17:00 | |
| zaro | fungi: i tested on my own jenkins cloud setup, not on jenkins dev. but i'm pretty confident this time. | 17:00 |
| fungi | micah: interesting. what sort of search did you perform? gertty reimplements search parameters itself, so it may not work 1:1 identically to gerrit for some searches (yet anyway) | 17:00 |
| fungi | zaro: sounds fine then--thanks! | 17:00 |
| *** pradk has joined #openstack-infra | 17:00 | |
| micah | fungi: i just did 'status:Merged' while in a project | 17:01 |
| *** andreykurilin_ has joined #openstack-infra | 17:01 | |
| *** gyee has joined #openstack-infra | 17:01 | |
| yolanda | works for me | 17:02 |
| fungi | micah: oh, interesting. i may have just dos'd myself because it seems to have ignored my project parameter and instead shown me every merged change in our gerrit (which is a very, very large number) | 17:02 |
| zaro | yolanda: thanks! | 17:03 |
| clarkb | zaro: any chance you can test it on jenkins-dev? | 17:03 |
| fungi | so it's still busy trying to grind my computer into a fine powder | 17:03 |
| *** pinajo has quit IRC | 17:03 | |
| zaro | clarkb: np, will do now | 17:03 |
| *** iax7 has quit IRC | 17:03 | |
| clarkb | fungi: performance testing of yuor IO capabilities | 17:03 |
| fungi | clarkb: yes! that's totally why i did it | 17:04 |
| yolanda | i left a +12 | 17:04 |
| fungi | completely intentional ;) | 17:04 |
| yolanda | +1 | 17:04 |
| yolanda | woo, i'd like to have +12 :) | 17:04 |
| *** IAX has quit IRC | 17:04 | |
| fungi | you want +13. 12 isn't prime | 17:05 |
| *** bswartz has quit IRC | 17:05 | |
| yolanda | clarkb, so you are reverting dib changes? | 17:05 |
| *** IAX has joined #openstack-infra | 17:05 | |
| micah | fungi: haha sorry about that - we only started using gerrit, so we only have about 10 of those | 17:06 |
| *** ihrachyshka has quit IRC | 17:07 | |
| *** markmcclain has joined #openstack-infra | 17:07 | |
| yolanda | i see that your environment is quite complex, it needs some rewrite of dib concepts, i see | 17:07 |
| fungi | micah: yeah, i killed the process with extreme prejudice ;) | 17:07 |
| *** tonytan4ever has joined #openstack-infra | 17:07 | |
| *** amotoki has quit IRC | 17:07 | |
| yolanda | clarkb, are you working on that change, or have some spec? | 17:07 |
| *** JayJ has quit IRC | 17:08 | |
| clarkb | yolanda: no I don't think we need to revert the change since this is fixable in config | 17:08 |
| *** HeOS has joined #openstack-infra | 17:09 | |
| clarkb | and no that all happened late last night for me so I haven't had time to work on anything beyond the config change | 17:09 |
| clarkb | I think its just a bug not needing a spec | 17:09 |
| clarkb | but I haven't had time to write the bug yet | 17:09 |
| *** cdent has quit IRC | 17:09 | |
| clarkb | yolanda: basically label: foo should be able to have images that are built with snapshot builder and dib builder | 17:09 |
| clarkb | yolanda: right now it must be one or the other | 17:09 |
| openstackgerrit | Ian Cordasco proposed a change to openstack-infra/devstack-gate: Fix broken links to the renamed openstack-infra/config project https://review.openstack.org/130285 | 17:10 |
| *** oramraz has joined #openstack-infra | 17:10 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Get verison info for jenkins-job-builder https://review.openstack.org/130260 | 17:10 |
| *** cpowell has quit IRC | 17:10 | |
| *** cdent has joined #openstack-infra | 17:11 | |
| *** tonytan4ever has quit IRC | 17:12 | |
| *** cdent has quit IRC | 17:12 | |
| *** bswartz has joined #openstack-infra | 17:13 | |
| yolanda | oh, so you need to share both methods, it will need a rewrite of the config file | 17:14 |
| *** Ryan_Lane has joined #openstack-infra | 17:15 | |
| yolanda | why is that reason, because of provider limits? | 17:15 |
| *** yamahata has quit IRC | 17:15 | |
| *** harlowja_away is now known as harlowja | 17:16 | |
| *** winston-d is now known as winston-d_zZZ | 17:17 | |
| *** kgiusti has joined #openstack-infra | 17:17 | |
| *** _nadya_ has quit IRC | 17:17 | |
| *** cpowell has joined #openstack-infra | 17:17 | |
| *** Sukhdev has quit IRC | 17:18 | |
| *** sdake_ has quit IRC | 17:18 | |
| EmilienM | fungi: clarkb: https://review.openstack.org/#/c/130260/ has been merged, while dhellmann had a comment | 17:18 |
| fungi | EmilienM: clarkb updated it to address dhellmann's comment | 17:19 |
| EmilienM | fungi: oh, my bad. | 17:19 |
| EmilienM | clarkb: thx! | 17:19 |
| fungi | EmilienM: it's possibly an additional change is warranted to make it "more correct" but we were in a hurry to make it work again (either by moving forward or moving backward) | 17:20 |
| *** iax7 has joined #openstack-infra | 17:21 | |
| *** IAX has quit IRC | 17:21 | |
| *** iax7 has quit IRC | 17:21 | |
| *** jedimike has quit IRC | 17:22 | |
| *** IAX has joined #openstack-infra | 17:22 | |
| *** IAX has quit IRC | 17:22 | |
| *** arnaud__ has quit IRC | 17:22 | |
| *** sarob has joined #openstack-infra | 17:23 | |
| openstackgerrit | Ian Cordasco proposed a change to openstack/requirements: Freeze mccabe's version for all projects https://review.openstack.org/130273 | 17:25 |
| *** ivar-lazzaro has joined #openstack-infra | 17:25 | |
| openstackgerrit | Ian Cordasco proposed a change to openstack/requirements: Freeze mccabe's version for hacking https://review.openstack.org/130273 | 17:25 |
| *** iax7 has joined #openstack-infra | 17:27 | |
| *** iax7 is now known as IAX | 17:27 | |
| *** sarob has quit IRC | 17:28 | |
| *** sarob_ has joined #openstack-infra | 17:28 | |
| EmilienM | fungi: makes sense | 17:28 |
| zaro | clarkb, fungi : tested the GP fix on jenkins-dev, worked as expected. | 17:33 |
| clarkb | zaro: great thank you | 17:34 |
| clarkb | zaro: fungi I will put 07 and 06 in shutdown mode now | 17:34 |
| fungi | clarkb: thanks | 17:35 |
| *** radez_g0n3 is now known as radez | 17:36 | |
| openstackgerrit | greghaynes proposed a change to openstack/requirements: Fix casing of xstatic-* https://review.openstack.org/130287 | 17:38 |
| *** julim has quit IRC | 17:39 | |
| *** IAX has quit IRC | 17:40 | |
| *** todd_dsm has joined #openstack-infra | 17:40 | |
| anteaya | dims__: I love your patches, they are so nice and tidy | 17:40 |
| clarkb | greghaynes: see comment | 17:41 |
| anteaya | dims__: and if you want to do stuff to your own project that doesn't affect anyone else, who am I to stand in your way? | 17:41 |
| greghaynes | clarkb: fun fun | 17:41 |
| *** changbl has joined #openstack-infra | 17:41 | |
| greghaynes | and I thought I could get away with a simple %s// | 17:41 |
| *** dims__ has quit IRC | 17:45 | |
| *** dims has joined #openstack-infra | 17:45 | |
| openstackgerrit | greghaynes proposed a change to openstack/requirements: Fix casing of xstatic-* https://review.openstack.org/130287 | 17:45 |
| *** dims has quit IRC | 17:46 | |
| *** dims has joined #openstack-infra | 17:46 | |
| dims | anteaya: thanks :) | 17:46 |
| anteaya | :D | 17:47 |
| *** _nadya_ has joined #openstack-infra | 17:48 | |
| *** tonytan4ever has joined #openstack-infra | 17:50 | |
| openstackgerrit | A change was merged to openstack-infra/project-config: Remove requirements job from nova-docker https://review.openstack.org/130192 | 17:50 |
| *** johnthetubaguy is now known as zz_johnthetubagu | 17:52 | |
| clarkb | if at first you don't succeed: recheck | 17:53 |
| clarkb | mtreinish: jogo: I think that the tempest tests are not super healthy right now | 17:53 |
| greghaynes | recheck, recheck again | 17:53 |
| dhellmann | EmilienM, clarkb, fungi: the updated jjb patch lgtm | 17:54 |
| dhellmann | fungi, clarkb: did either of you have comments on https://etherpad.openstack.org/p/oslo-26-support-policy before I send it to the list? | 17:54 |
| clarkb | let me look | 17:55 |
| fungi | dhellmann: oh, sorry, reviewing it now | 17:55 |
| dhellmann | fungi: np, you had other fires to fight | 17:55 |
| *** jamielennox has quit IRC | 17:55 | |
| clarkb | dhellmann: https://etherpad.openstack.org/p/py26-deprecation is the thing I intend on sending soon | 17:55 |
| *** jamielenz is now known as jamielennox | 17:55 | |
| clarkb | but I don't think they conflict | 17:56 |
| clarkb | your etherpad lgtm | 17:56 |
| clarkb | if you send that before I send mine I will refer to your email in mine where I talk about what oslo will be doing | 17:56 |
| jogo | clarkb: we had a good run :/ | 17:57 |
| dhellmann | clarkb: thanks, I've had several questions about 2.6 for oslo so I wanted to get this out soon. | 17:57 |
| jogo | clarkb: taking a look | 17:57 |
| jogo | clarkb: classification rate of 55% | 17:57 |
| clarkb | dhellmann: yup go for it | 17:57 |
| jogo | and yeah http://jogo.github.io/gate/ | 17:57 |
| dhellmann | clarkb: cool, I'll wait for an all-clear from fungi since I asked him, too | 17:57 |
| *** amcrn has joined #openstack-infra | 17:58 | |
| *** dizquierdo has quit IRC | 17:58 | |
| fungi | dhellmann: i think that's got it... if an api client later tries to use a non-2.6-capable oslo lib, we'll catch that in testing the client change (in theory anyway) | 17:58 |
| *** markmcclain has quit IRC | 17:58 | |
| *** rmcall has quit IRC | 17:59 | |
| *** arnaud__ has joined #openstack-infra | 17:59 | |
| dhellmann | fungi: in that case are you talking about a new lib? I guess we'd need to go back and add 2.6 support before the client could use it, in that case? | 17:59 |
| mtreinish | clarkb: do you have any examples, I didn't feel think our failure rate had skyrocketted or anything | 17:59 |
| *** rmcall has joined #openstack-infra | 18:00 | |
| fungi | dhellmann: right | 18:01 |
| dhellmann | ok, I'll include that | 18:01 |
| *** andreykurilin_ has quit IRC | 18:01 | |
| *** jamielennox has quit IRC | 18:01 | |
| clarkb | mtreinish: https://review.openstack.org/#/c/128974/ refuses to pass | 18:01 |
| fungi | dhellmann: basically the risk that we release a lib which doesn't need 2.6 support and then later another lib or client which _does_ still require 2.6 support tries to start using it | 18:01 |
| clarkb | has been rechecked a ton | 18:01 |
| clarkb | mtreinish: jogo's graph are better hard data though | 18:02 |
| fungi | dhellmann: but i do expect we'd catch that in testing the depending change anyway, so should work out fine | 18:02 |
| *** otherwiseguy has quit IRC | 18:02 | |
| *** sdake_ has joined #openstack-infra | 18:03 | |
| *** dane_leblanc has quit IRC | 18:03 | |
| *** _nadya_ has quit IRC | 18:04 | |
| *** eharney has quit IRC | 18:04 | |
| *** vigneshvar has joined #openstack-infra | 18:04 | |
| *** vigneshvar has quit IRC | 18:04 | |
| dhellmann | fungi: is havana still supported, or is it just icehouse and juno now? | 18:05 |
| jogo | mtreinish: http://status.openstack.org/elastic-recheck/index.html#1383347 and http://status.openstack.org/elastic-recheck/index.html#1383439 are pretty bad | 18:05 |
| fungi | dhellmann: havana is dead and buried since more than a month | 18:05 |
| dhellmann | fungi: that's what I thought, thanks | 18:05 |
| fungi | though if you want to fire a few bullets into its rotting carcass feel free | 18:06 |
| *** dane_leblanc has joined #openstack-infra | 18:06 | |
| *** luqas has quit IRC | 18:06 | |
| *** dims has quit IRC | 18:07 | |
| mtreinish | jogo: hmm, well one of the failures on that change I hadn't seen before. Nova absolute limits on image metadata failed to enforce | 18:08 |
| mtreinish | jogo: the grenade one is on a non-voting job | 18:08 |
| *** dims has joined #openstack-infra | 18:08 | |
| mtreinish | the job doesn't work | 18:08 |
| jogo | mtreinish: oh right https://bugs.launchpad.net/grenade/+bug/1383439 is the non-voting one | 18:10 |
| mtreinish | clarkb: hmm both the failures this time look like the dreaded ssh timeout failure | 18:10 |
| uvirtbot | Launchpad bug 1383439 in grenade "KeyError: 'private' in *-grenade-dsvm-neutron javelin logs" [Undecided,New] | 18:10 |
| jogo | mtreinish: that leaves https://bugs.launchpad.net/neutron/+bug/1381617 | 18:10 |
| uvirtbot | Launchpad bug 1381617 in neutron "Check for FIP status in scenario tests cause instability in neutron gate jobs" [Critical,In progress] | 18:10 |
| jogo | salv-orlando: ^ | 18:10 |
| mtreinish | clarkb: hmm on the first 1 it's only trying to ssh 3 times, it normally does more than that | 18:11 |
| jogo | mtreinish: and the fix https://review.openstack.org/#/c/130221/ | 18:11 |
| mtreinish | http://logs.openstack.org/74/128974/1/check/check-tempest-dsvm-full/7acbd15/console.html#_2014-10-22_17_33_34_617 | 18:11 |
| * jogo goes to neutron room to bother them | 18:11 | |
| *** thedodd has quit IRC | 18:12 | |
| *** gokrokve has quit IRC | 18:12 | |
| *** marun has quit IRC | 18:14 | |
| tdasilva | hello...I'm looking for some help regarding stackforge projects. We recently had our project created: https://github.com/stackforge/swiftonfile and our irc channel #swiftonfile now has openstackinfra listed as Founder | 18:15 |
| tdasilva | does anybody know who could give one of us operator privilege? | 18:15 |
| notmyname | tdasilva: you've been absorbed by the borg! | 18:15 |
| dhellmann | fungi, clarkb: http://lists.openstack.org/pipermail/openstack-dev/2014-October/048999.html | 18:15 |
| dhellmann | clarkb: I added that link to your etherpad for easy reference | 18:16 |
| tdasilva | the borg??? | 18:16 |
| *** mrmartin has joined #openstack-infra | 18:16 | |
| fungi | tdasilva: i can op you in there, just a moment | 18:16 |
| tdasilva | fungi: thx | 18:17 |
| mrmartin | re | 18:17 |
| *** andreykurilin_ has joined #openstack-infra | 18:17 | |
| mrmartin | guys, if you have a little time, please review this: https://review.openstack.org/#/c/129610/ | 18:17 |
| *** radez is now known as radez_g0n3 | 18:18 | |
| *** baoli_ has quit IRC | 18:22 | |
| jogo | clarkb: http://logs.openstack.org/22/117922/10/gate/gate-tempest-dsvm-neutron-heat-slow/541580e/console.html#_2014-10-22_11_31_05_639 | 18:22 |
| clarkb | jogo: ya thats an ongoing issue that should be mitigated as we build new images | 18:23 |
| clarkb | jogo: is that a siginificant chunk of the failure? (the failures on my change were not that) | 18:23 |
| jogo | clarkb: is there an open bug for that? | 18:24 |
| fungi | mrmartin: i hadn't seen that one yet, but looks good to me | 18:25 |
| jogo | 103 hits in 24 hours | 18:25 |
| clarkb | jogo: mtreinish may have opened one. it was late last night so I didn't do much | 18:25 |
| clarkb | (mtreinish pointed it out last night) | 18:25 |
| jogo | clarkb: will add a query just to drop those from the unclassified page | 18:25 |
| mtreinish | clarkb, jogo: no I never got around to writing a query for it. It was a bit late when I caught the problem | 18:26 |
| *** jamielenz has joined #openstack-infra | 18:26 | |
| mtreinish | jogo: also never bothered opening a bug for it | 18:26 |
| *** jgrimm is now known as zz_jgrimm | 18:26 | |
| *** jamielenz is now known as jamielennox | 18:26 | |
| jogo | clarkb: filed a bug, can you triage it accordingly: bug 1384373 | 18:27 |
| uvirtbot | Launchpad bug 1384373 in openstack-ci "fatal: Could not read from remote repository" [Undecided,New] https://launchpad.net/bugs/1384373 | 18:27 |
| salv-orlando | jogo: thanks for raising the issue. From what I gather armax is following this bug. | 18:27 |
| *** thedodd has joined #openstack-infra | 18:28 | |
| jogo | salv-orlando: yeah, they are discussing it in the neutron room | 18:28 |
| armax | salv-orlando, jogo: I reported and watching it | 18:28 |
| salv-orlando | and obviously I was not looking at openstack-neutron | 18:28 |
| armax | but currently I don’t have a lot of cycles to make sure this gets closed quickly | 18:28 |
| *** erlon has joined #openstack-infra | 18:29 | |
| clarkb | jogo: thanks. commented | 18:29 |
| salv-orlando | armax: From what I saw last week, this is a regression that has been introduced by a commit occurred on Oct 15th. I did not see exactly which one though. | 18:29 |
| armax | salv-orlando: a tempest test was introduced that checked that the state transition for the FIP was actually occurring | 18:29 |
| *** __TheDodd__ has joined #openstack-infra | 18:30 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1384373 https://review.openstack.org/130305 | 18:30 |
| uvirtbot | Launchpad bug 1384373 in openstack-ci "fatal: Could not read from remote repository" [Undecided,New] https://launchpad.net/bugs/1384373 | 18:30 |
| *** lttrl has quit IRC | 18:30 | |
| *** mrodden has quit IRC | 18:30 | |
| jogo | salv-orlando: this is a case where going over the openstack/openstack commit logs is useful | 18:30 |
| jogo | because this could be a change in tempest or devstack that triggered it etc | 18:30 |
| armax | jogo: we already determined the root cause | 18:31 |
| salv-orlando | armax: I see. So more a case of a bug being uncovered than a regression. | 18:31 |
| armax | salv-orlando: correct | 18:31 |
| jogo | armax: excellent | 18:31 |
| salv-orlando | armax: I see a patch that fixes only the dvr case. Do we need a different one for “traditional” routers? | 18:32 |
| armax | salv-orlando: yes, I made that point on the patch | 18:32 |
| *** markmcclain has joined #openstack-infra | 18:32 | |
| armax | salv-orlando: from logstash DVR is affected more | 18:32 |
| armax | salv-orlando: but the issue manifests itself on the other jobs too | 18:33 |
| armax | salv-orlando: the DVR job is non voting now so it’s sort of ok | 18:33 |
| *** e0ne has quit IRC | 18:33 | |
| salv-orlando | armax: I see. Do you suspect independnet root causes then? I mean for dvr and non-dvr | 18:33 |
| jogo | clarkb: a lot of the gate failures in the past 2 days were git related | 18:33 |
| *** thedodd has quit IRC | 18:33 | |
| armax | salv-orlando: dvr is only slower than the legacy case | 18:33 |
| armax | salv-orlando: as more actions are invovled for the FIP handling | 18:34 |
| *** julim has joined #openstack-infra | 18:34 | |
| armax | salv-orlando: so the fact that the test complains that the state transition hasn’t occurred happens in 100% cases for DVR | 18:34 |
| *** andreykurilin_ has quit IRC | 18:34 | |
| clarkb | jogo: ok I will focus on making sure we have up to date images to reduce that pressure | 18:34 |
| armax | salv-orlando: as the timeout is just not long enough for DVR | 18:34 |
| salv-orlando | armax: so different dynamics, but same root cause. | 18:35 |
| armax | salv-orlando: I suspect that the non-dvr case we’re testing right at the edge | 18:35 |
| *** tsg has joined #openstack-infra | 18:35 | |
| * zaro afk, bb in 2hrs | 18:35 | |
| armax | salv-orlando: in a nutshell, that’s my understanding, yes | 18:35 |
| salv-orlando | Which makes me raise an eyebrow on a proposed fix which targets only dbr | 18:35 |
| *** zz_jgrimm is now known as jgrimm | 18:35 | |
| openstackgerrit | Michael Tupitsyn proposed a change to openstack-infra/jenkins-job-builder: Support additional events for HipChat plugin https://review.openstack.org/129760 | 18:35 |
| *** Guest53198 has quit IRC | 18:35 | |
| armax | salv-orlando: yup, I made that point, it’s sensible to have separate fixes though | 18:35 |
| salv-orlando | armax: why? don’t answer now - save it for when I come back from dinner! | 18:36 |
| armax | I’ll answer now, because I might be to lunch then :) | 18:36 |
| *** andreykurilin_ has joined #openstack-infra | 18:37 | |
| jogo | clarkb: well ER is up to date, and I don't see any hits for a few hours | 18:37 |
| jogo | so that looks promising | 18:37 |
| armax | salv-orlando: it can be two separate fixes because the components being affected may be different, so it’s just logical decomposition, I mean we can target both fixes in the same patch, the patch most likely is going to be bigger | 18:38 |
| pelix | Running into some problems with certain changes on the jenkins slaves, not quite sure how to replicate locally in order to determine how to fix. see http://logs.openstack.org/18/129318/2/check/gate-git-upstream-python27/021eaea/console.html.gz -> does not happen locally and my attempt to guard against the env variable not being set resulted in a change in behaviour -> http://logs.openstack.org/02/130302/1/check/gate-git-upstream-python27/4 | 18:39 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1384373 https://review.openstack.org/130305 | 18:39 |
| uvirtbot | Launchpad bug 1384373 in openstack-ci "fatal: Could not read from remote repository" [Undecided,New] https://launchpad.net/bugs/1384373 | 18:39 |
| pelix | Any suggestions for how to test? Is this just something to do with the minimal environment set up in slaves? | 18:39 |
| jogo | clarkb: this is fun too http://logs.openstack.org/11/124811/4/gate/gate-tempest-dsvm-neutron-heat-slow/2d87a05/logs/devstacklog.txt.gz#_2014-10-21_15_18_24_243 | 18:40 |
| jogo | looks like dl.fedoraproject.org is being slow as molasses | 18:40 |
| anteaya | I haven't been around much today, been doing meetings and errandy things | 18:41 |
| anteaya | now I have to go to town so will be afk for a few hours | 18:41 |
| anteaya | I don't think anyone is waiting on me for anything though | 18:41 |
| *** andreykurilin_ has quit IRC | 18:41 | |
| *** marun has joined #openstack-infra | 18:42 | |
| *** jergerber has joined #openstack-infra | 18:42 | |
| *** emagana has quit IRC | 18:43 | |
| mtreinish | jogo: there's a bug for that one, I hit it yesterday (or mon) | 18:43 |
| *** koolhead17 has joined #openstack-infra | 18:44 | |
| mtreinish | https://bugs.launchpad.net/openstack-ci/+bug/1383928 | 18:44 |
| uvirtbot | Launchpad bug 1383928 in openstack-ci "Heat fedora image download times out job" [Undecided,New] | 18:44 |
| mtreinish | although my guess in the bug turned out to be wrong | 18:44 |
| *** hdd has quit IRC | 18:45 | |
| *** rmcall has quit IRC | 18:50 | |
| *** pelix has quit IRC | 18:50 | |
| *** dane_leblanc has quit IRC | 18:50 | |
| *** rmcall has joined #openstack-infra | 18:51 | |
| *** IAX has joined #openstack-infra | 18:52 | |
| *** IAX has quit IRC | 18:52 | |
| *** rlandy has joined #openstack-infra | 18:52 | |
| *** armax has quit IRC | 18:55 | |
| clarkb | jogo: ok rax devstack-trusty image updates are almost done (they were the most stale) | 18:56 |
| clarkb | and I am doing devstack-trusty rebuilds in hpcloud too just for completeness | 18:56 |
| clarkb | that should cover the majority of our git updates on job runs | 18:56 |
| *** dane_leblanc has joined #openstack-infra | 18:56 | |
| *** mrmartin has quit IRC | 18:57 | |
| clarkb | I am preparing to upgrade the gearman-plugin on 06 and 07 | 18:57 |
| clarkb | they are both done running jobs | 18:57 |
| *** sarob_ has quit IRC | 18:58 | |
| *** markmcclain has quit IRC | 18:58 | |
| *** sarob has joined #openstack-infra | 18:58 | |
| *** eharney has joined #openstack-infra | 19:01 | |
| *** amcrn has quit IRC | 19:01 | |
| *** sarob_ has joined #openstack-infra | 19:02 | |
| *** sarob has quit IRC | 19:02 | |
| jogo | mtreinish: you were right btw https://bugs.launchpad.net/openstack-ci/+bug/1383928 | 19:02 |
| uvirtbot | Launchpad bug 1383928 in openstack-ci "Heat fedora image download times out job" [Undecided,New] | 19:02 |
| *** _nadya_ has joined #openstack-infra | 19:02 | |
| jogo | there are two patches we need to backport | 19:02 |
| jogo | clarkb: thanks | 19:02 |
| *** dustins has quit IRC | 19:02 | |
| *** bswartz1 has joined #openstack-infra | 19:03 | |
| *** otherwiseguy has joined #openstack-infra | 19:04 | |
| *** dustins has joined #openstack-infra | 19:04 | |
| jogo | clarkb: oh rackspace http://logs.openstack.org/44/129744/1/gate/gate-grenade-dsvm-forward/ee04d51/logs/devstack-gate-setup-host.txt.gz#_2014-10-22_03_42_02_536 | 19:05 |
| jogo | fungi: ^ | 19:05 |
| *** clarkb has quit IRC | 19:06 | |
| *** bswartz has quit IRC | 19:07 | |
| *** clarkb has joined #openstack-infra | 19:07 | |
| *** david-lyle_ has joined #openstack-infra | 19:07 | |
| lifeless | mtreinish: whats up ? | 19:07 |
| *** markmcclain has joined #openstack-infra | 19:08 | |
| clarkb | gah what did I do to weechat | 19:08 |
| greghaynes | clarkb: did you update? | 19:08 |
| *** ryanpetrello has quit IRC | 19:09 | |
| clarkb | greghaynes: not that I am aware of | 19:09 |
| *** shashankhegde has quit IRC | 19:09 | |
| clarkb | but its like I am in some super weird mouse mode | 19:09 |
| clarkb | I was hoping a restart would fix because I didn/t /save | 19:09 |
| *** david-lyle has quit IRC | 19:09 | |
| *** ryanpetrello has joined #openstack-infra | 19:09 | |
| clarkb | it might also be screen but this only seems to affect windows with weechat in it | 19:09 |
| greghaynes | I consider XFCE to be a super wierd mouse mode | 19:09 |
| fungi | jogo: what's wrong in that log file you're pointing out? | 19:09 |
| openstackgerrit | A change was merged to openstack/requirements: Bump oslotest minimum requirements to 1.2.0 https://review.openstack.org/130058 | 19:09 |
| clarkb | greghaynes: somehow I enabled the mouse | 19:10 |
| clarkb | greghaynes: /mouse disable has it happy again | 19:10 |
| clarkb | that was fun | 19:10 |
| *** talluri has quit IRC | 19:11 | |
| fungi | jogo: the error which tanked that job seems to be http://logs.openstack.org/44/129744/1/gate/gate-grenade-dsvm-forward/ee04d51/logs/devstack-gate-setup-workspace-new.txt.gz#_2014-10-22_03_45_38_683 | 19:11 |
| *** marun has quit IRC | 19:11 | |
| fungi | jogo: er, same but even earlier... http://logs.openstack.org/44/129744/1/gate/gate-grenade-dsvm-forward/ee04d51/logs/devstack-gate-setup-workspace-new.txt.gz#_2014-10-22_03_42_55_808 | 19:12 |
| openstackgerrit | A change was merged to openstack/requirements: Add enum34 requirement https://review.openstack.org/127829 | 19:13 |
| fungi | jogo: i think a relative path isn't what we think it is when the cd is happening, or else we're accidentally blowing away that directory, because you can see it being created and used earlier in the output | 19:13 |
| mriedem | anyone seen this? http://goo.gl/Tk9bVX | 19:13 |
| mriedem | http://logs.openstack.org/72/128772/3/check/check-tempest-dsvm-full-icehouse/5c16e86/logs/devstacklog.txt.gz#_2014-10-22_00_00_00_997 | 19:14 |
| *** cody-somerville has quit IRC | 19:15 | |
| fungi | mriedem: master tempest uninstallable with the pbr from openstack/requirements stable/icehouse? | 19:15 |
| clarkb | ok now that weechat is happy again I can go back to jenkins07 and 06 | 19:15 |
| *** ociuhandu has quit IRC | 19:15 | |
| clarkb | fungi: ready for me to upgrade them? | 19:15 |
| fungi | clarkb: have at it | 19:15 |
| mriedem | fungi: not sure i know what that means | 19:16 |
| mriedem | mtreinish: have you seen this? http://logs.openstack.org/72/128772/3/check/check-tempest-dsvm-full-icehouse/5c16e86/logs/devstacklog.txt.gz#_2014-10-22_00_00_00_997 | 19:16 |
| fungi | mriedem: oh, wait, this is the issue mtreinish mentioned earlier related to the tempest 2 release tag | 19:16 |
| mriedem | _get_version_from_git_target | 19:17 |
| mriedem | yeah that looks right | 19:17 |
| fungi | mriedem: it's pending tempest landing a change to setup.cfg | 19:17 |
| clarkb | 07 is coming back now | 19:17 |
| mriedem | fungi: have a link? | 19:17 |
| clarkb | zaphod42: ^ | 19:17 |
| clarkb | sorry zaphod42 I meant to ping zaro ^ | 19:17 |
| fungi | mriedem: https://review.openstack.org/130099 | 19:18 |
| fungi | mriedem: seems it's merged now | 19:18 |
| *** david-lyle_ is now known as david-lyle | 19:18 | |
| mtreinish | mriedem: yeah that was because I pushed the tag before updating the version string in setup.cfg | 19:19 |
| mriedem | fungi: thanks - was going to write an e-r query but i guess it only affected the check queue | 19:19 |
| fungi | mriedem: right, only affected tempest itself apparently | 19:19 |
| fungi | so after the tag, nothing could get into the gate | 19:20 |
| fungi | for that one project | 19:20 |
| mtreinish | thanks to the new semver check in pbr it was exploding because it detected the 2 was already tagged so we couldn't be working on a dev version of the same version | 19:20 |
| mtreinish | fungi: the weird thing is it was only on the stable branch jobs | 19:20 |
| mtreinish | master worked fine | 19:20 |
| mtreinish | lifeless: I figured it out: ^^^ | 19:20 |
| clarkb | fungi: zaro I have confirmed that one of the jobs on 07 reported back success to zuul | 19:22 |
| clarkb | so that appears to be working. will keep an eye out for other jobs though | 19:22 |
| *** markmcclain has quit IRC | 19:22 | |
| fungi | awesome | 19:22 |
| clarkb | https://jenkins07.openstack.org/job/gate-oslo.config-python33/112/console that job. you can see it is marked success on zuul status page | 19:23 |
| *** markmcclain has joined #openstack-infra | 19:23 | |
| fungi | nibalizer: any suggestions for a tool which can do searches of the puppet filebucket? | 19:24 |
| *** kgiusti has quit IRC | 19:26 | |
| clarkb | fungi: mordred I am going to beg for reviews on the stack at https://review.openstack.org/#/c/128973/1 | 19:26 |
| clarkb | jenkins is +1 across the board (double check that the greand ebranches do what you expect though) | 19:27 |
| clarkb | but once that is in we should be done with juno grenade stuff for a while | 19:27 |
| dkranz | clarkb: Did you see my comment in https://review.openstack.org/#/c/127561/, and can you push it through? It is needed for the periodic stress job to not fail. | 19:27 |
| nibalizer | fungi: uh no, i don't think anyone wrote a tool like that | 19:28 |
| *** tonytan4ever has quit IRC | 19:28 | |
| *** signed8bit has quit IRC | 19:28 | |
| fungi | nibalizer: thanks! that confirms my findings as well ;) | 19:28 |
| clarkb | dkranz: done | 19:29 |
| dkranz | clarkb: thanks, I will push a new patch to make it configurable | 19:29 |
| nibalizer | i think find . | xargs cat | grep keyword gets you pretty clsoe | 19:29 |
| *** dims has quit IRC | 19:29 | |
| clarkb | nibalizer: grep -R ? | 19:29 |
| nibalizer | clarkb: that also works | 19:30 |
| *** dims has joined #openstack-infra | 19:30 | |
| *** sdake_ has quit IRC | 19:30 | |
| *** e0ne has joined #openstack-infra | 19:30 | |
| openstackgerrit | Jan Kundrát proposed a change to stackforge/gertty: Add support for fetching changesets over SSH protocol https://review.openstack.org/124239 | 19:30 |
| *** arnaud has joined #openstack-infra | 19:31 | |
| clarkb | 07 continues to look happy | 19:32 |
| clarkb | doing 06 now | 19:32 |
| fungi | clarkb: nibalizer: for searching a local filebucket anyway. remotely searching the filebucket is likely going to be intractable without retrieving all the contents i guess | 19:32 |
| mtreinish | BobBall: on xenserver you guys might want to take a look at: https://review.openstack.org/#/c/117649/ I'm guessing it's having issues with the tempest-lib requirement, but the tox failure isn't getting printed in the logs | 19:33 |
| *** kgiusti has joined #openstack-infra | 19:33 | |
| *** shashankhegde has joined #openstack-infra | 19:34 | |
| *** tonytan4ever has joined #openstack-infra | 19:34 | |
| clarkb | and 06 is coming back with new plugin now | 19:34 |
| *** dims has quit IRC | 19:34 | |
| *** jamielennox has quit IRC | 19:36 | |
| *** oramraz has quit IRC | 19:38 | |
| ashp | Argh, this is so weird, everything was working and suddenly: | 19:38 |
| ashp | 2014-10-22 19:38:19,639 ERROR zuul.Gearman: Job <gear.Job 0x7fe7f808cf90 handle: None name: build:gate-horizon-pep8 unique: f0605bb00129412481515122cdd55c24> is not registered with Gearman | 19:38 |
| ashp | my jobs just... seem to have been forgotten | 19:39 |
| clarkb | ashp: do the jobs still exist in jenkins? | 19:39 |
| fungi | ashp: did you restart zuul? | 19:39 |
| ashp | I did, and they do still exist in jenkins | 19:39 |
| ashp | i restarted jenkins and zuul and let it update to the 1.0 of the gearman plugin, even | 19:39 |
| ashp | I can see the jobs listed in zuul.log as existing, so this is confusing | 19:40 |
| clarkb | ashp: oh you don't want that version of the gearman-plugin. its broken :( | 19:40 |
| fungi | restarting zuul ends up restarting the gearman service it provides, which causes it to forget any prior job registrations | 19:40 |
| ashp | clarkb: whoops | 19:40 |
| clarkb | ashp: also if you have no slaves to run the jobs then the will not be registered | 19:40 |
| fungi | however, when jenkins reconnects to the gearman service it should reregister any jobs for which it has configuration and workers | 19:40 |
| ashp | I do have a slave, that's what I was fixing up today, I just trie dto manually run a job to make sure jenkins can see the slave, and it does | 19:41 |
| ashp | downgrading to the previous plugin version first, tho | 19:41 |
| clarkb | ashp: or if you are brave you can try http://tarballs.openstack.org/ci/gearman-plugin/gearman-plugin-0.1.0.1.fa5f083.hpi | 19:41 |
| clarkb | Iam currently upgrading to that on our jenkinses | 19:41 |
| fungi | we have that running in production on two jenkins masters so far | 19:41 |
| ashp | this is probably just my lack of understanding but.. is there an easy way, outside of just typing status to gearman, to see what jobs zuul is waiting for jenkins to pick up and run | 19:42 |
| ashp | because I feel like I should have oustanding jobs that jenkins is just ignoring | 19:42 |
| ashp | oh hmm, status doesn't show any of the jobs, so I guess it really has forgotten about them. | 19:42 |
| fungi | ashp: zuul has a status.json you can retrieve from its rudimentary read-only web api | 19:43 |
| *** dkranz has quit IRC | 19:43 | |
| fungi | ashp: and also a fairly simplistic status page served from the root url | 19:43 |
| fungi | work is underway to extend both of those considerably in coming versions | 19:44 |
| ashp | So I restart zuul, in the zuul.log/debug.log I see my jobs in layout.yaml being loaded, but then status doesn't know about them. That should get populated when jenkins connects and informs it what jobs it has, right? | 19:44 |
| ashp | I might have to try 0.1.0.1 at this rate, as it seems like jenkins has forgotten to bother | 19:44 |
| fungi | ashp: right, they won't show up in the gearman queue until jenkins connects and registers them | 19:45 |
| ashp | I assume jenkins should just constantly be trying to connect? | 19:45 |
| fungi | it should | 19:45 |
| ashp | hmm, the test connection works fine, I wonder why it doesn't | 19:45 |
| *** shashankhegde has quit IRC | 19:45 | |
| fungi | when we restart zuul our jenkins masters all reconnect automatically | 19:45 |
| ashp | I wonder what on earth I did to jenkins to make it so unhappy today | 19:46 |
| fungi | the 1.0 plugin has a bit of an unintended delay on job registration | 19:46 |
| ashp | i'm back on the 0.0.7 plugin, and I can see with tcpdump outside of hitting the test button it doesn't even try to connect to gearman | 19:46 |
| *** oramraz has joined #openstack-infra | 19:47 | |
| fungi | ashp: what version of jenkins are you running? | 19:47 |
| ashp | 1.584 | 19:47 |
| fungi | aha, we're on 1.565.3 (stable train) so it's also possible something new has changed in the dev train | 19:48 |
| ashp | what's weird is this was totally working yesterday and this morning | 19:48 |
| ashp | I was fixing my ec2 slave building steps and it just suddenly stopped | 19:48 |
| ashp | ok, trying another restart, I can see it trying to talk to gearman | 19:49 |
| ashp | hmm, what the hell. | 19:50 |
| fungi | but yeah, the fix on master since 0.1.0 has to do with incorrect reporting of job completion status, shouldn't affect registration at all (i think) | 19:50 |
| *** ChuckC has quit IRC | 19:50 | |
| ashp | http://pastebin.com/JEPE6htC from jenkins.log, not very useful | 19:52 |
| clarkb | it might since its related to registration on offline | 19:52 |
| ashp | but it's still not registering its own jobs | 19:52 |
| *** weshay has quit IRC | 19:52 | |
| *** e0ne has quit IRC | 19:52 | |
| ashp | it's clearly talking to zuul however | 19:53 |
| *** rmcall has quit IRC | 19:53 | |
| ashp | http://pastebin.com/eiQkGumE status just gives me that | 19:53 |
| ashp | i'm going to try 1.0.1.0 just in case | 19:54 |
| *** rmcall has joined #openstack-infra | 19:54 | |
| fungi | is it possible your jenkins master is talking to a different gearman server than the one zuul is using? (like maybe you accidentally ran a standalone gearmand somwhere and are connecting jenkins to that?) | 19:55 |
| ashp | I don't think so, I can see it happily talking over tcpdump when they start up | 19:56 |
| ashp | and there's no other gearman on that box | 19:56 |
| fungi | 0.1.0.1.0.1.0.1 | 19:56 |
| ashp | and confirmed by the fact if I stop zuul running it freaks out | 19:57 |
| *** teran has joined #openstack-infra | 19:57 | |
| *** dims has joined #openstack-infra | 19:57 | |
| *** markmcclain has quit IRC | 19:58 | |
| *** shashankhegde has joined #openstack-infra | 19:58 | |
| fungi | definitely weird | 19:58 |
| ashp | I see both jenkins and the build slave in workers :/ | 19:58 |
| *** MaxV has joined #openstack-infra | 19:58 | |
| *** teran_ has joined #openstack-infra | 19:58 | |
| *** e0ne has joined #openstack-infra | 20:00 | |
| *** e0ne has quit IRC | 20:00 | |
| *** sarob_ has quit IRC | 20:00 | |
| *** sarob has joined #openstack-infra | 20:01 | |
| *** teran has quit IRC | 20:02 | |
| ashp | now... now they are back | 20:02 |
| ashp | whatttttt | 20:02 |
| *** Ryan_Lane1 has joined #openstack-infra | 20:02 | |
| ashp | I clicked the tickbox on/off and it magically helped | 20:03 |
| *** sarob has quit IRC | 20:03 | |
| fungi | disabled and enabled the plugin? | 20:03 |
| *** sarob has joined #openstack-infra | 20:03 | |
| ashp | no | 20:03 |
| ashp | just the tickbox in the main jenkins conf | 20:03 |
| ashp | that says "use the gearman plugin" | 20:03 |
| fungi | zaro: ^ something to look forward to the next time we upgrade jenkins, possibly | 20:03 |
| clarkb | ya so I think what happened there is the plugin was disabled then enabled | 20:04 |
| clarkb | and enabling it forced it to reregister | 20:04 |
| *** Ryan_Lane has quit IRC | 20:04 | |
| ashp | Now I'm all the way back to my original problem of gerrit-git-prep not working! | 20:04 |
| ashp | A worthy 24 hours to get back to where I was :) | 20:04 |
| ashp | actually to be fair this is a slightly different set of failures as I updated the scripts | 20:04 |
| ashp | fatal: http://zuul.metacloud.in/metacloud/horizon/info/refs not found: did you run git update-server-info on the server? | 20:05 |
| ashp | before it just complained about missing refs | 20:05 |
| ashp | I guess that's basically the same problem :) | 20:05 |
| ashp | It's hard to tell if the zuul-merger process is just broken for me or it's just a boring permissions type thing | 20:06 |
| fungi | ashp: this is where you were seeing the refs in zuul's local working copies on the filesystem but apache wasn't returning them when queried? | 20:06 |
| ashp | Yeah, I think this is the same thing, just exposed slightly different | 20:06 |
| ashp | http://pastebin.com/pAGKhu4p | 20:06 |
| clarkb | jogo: you said https://review.openstack.org/#/c/128974/ is doing the correct thing for partial ncpu ya? | 20:06 |
| *** otherwiseguy has quit IRC | 20:06 | |
| openstackgerrit | Jan Kundrát proposed a change to stackforge/gertty: Fix changeset fetching https://review.openstack.org/124239 | 20:07 |
| fungi | probably double-checking the vhost config to see if it matches the git serving recommendations in zuul's documentation, and looking at your apache error logs are what i'd suggest next | 20:07 |
| ashp | I took a day detour because I found when I wanted to update the .sh it runs we were relying on a nested set of bash scripts hosted in gist.github.com links and I was horrified, so I had to stop and fix it first. | 20:07 |
| clarkb | fungi: I may end up self approving that change since SergeyLukjanov is back on the other side of planet and we appear to have no mordred or jeblair today | 20:07 |
| clarkb | sdad: is afk and jhesketh won't wake for a bit | 20:07 |
| fungi | clarkb: i can approve it | 20:07 |
| clarkb | fungi: cool thanks | 20:07 |
| ashp | fungi: when git clone/git X runs against apache there's no errors | 20:07 |
| fungi | was just about done going through the child change for it | 20:07 |
| fungi | ashp: oh, one other thing, cloning from the zuul merger is possibly not a great idea. we generally clone from somewhere else (a local copy cached on the filesystem image or a gerrit replica) and then fetch specific refs from the zuul merger | 20:10 |
| ashp | fungi: oh maybe that's what is killing me | 20:10 |
| fungi | ashp: teh zuul merger is really only meant to be serving specific refs you can fetch | 20:10 |
| ashp | Ahhh, hmm, let me experiment :) | 20:11 |
| clarkb | ok at least one job on 06 reported success and I don't see jobs running out of order all over the place | 20:12 |
| clarkb | gate-tempest-dsvm-ceilometer-mongodb that is showing up as not registered though | 20:12 |
| clarkb | I think unrelated. I wonder if ceilometer has jus tbeen ignoring that /me looks | 20:12 |
| *** otherwiseguy has joined #openstack-infra | 20:13 | |
| clarkb | hrm looks new | 20:13 |
| fungi | ashp: at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n1 you can see that we pass our gerrit and then another git server to gerrit-git-prep, and so the only interaction it has with the zuul merger is to fetch the ref mentioned in $ZUUL_REF | 20:13 |
| clarkb | but prior to my 06 and 07 work so likely unrelated | 20:14 |
| clarkb | https://review.openstack.org/#/c/130142/ is an early incidence | 20:14 |
| *** aysyd has quit IRC | 20:14 | |
| ashp | I'll try and get it to pull out of our private github stuff | 20:14 |
| ashp | seems most sensible | 20:14 |
| fungi | clarkb: yeah i think that's fallout from the jjb breakage | 20:14 |
| *** freemanbrandon has quit IRC | 20:14 | |
| clarkb | fungi: actually I think it may be more subtle | 20:15 |
| clarkb | its running against icehouse | 20:15 |
| clarkb | but we set node to devstack-trusty | 20:15 |
| fungi | clarkb: aha, then yes, just badly configured | 20:15 |
| clarkb | then zuul says if icehouse then devstack-precise | 20:15 |
| * clarkb will figure out if we should make that work on stable brnches | 20:15 | |
| fungi | yep. config issue | 20:15 |
| clarkb | or if we can stop running it on icehouse | 20:16 |
| fungi | i think it was only meant to run on trusty because mongodb on precise was unsatisfactory? | 20:16 |
| fungi | if so, then we should skip it for stable/icehouse | 20:16 |
| clarkb | yup | 20:16 |
| *** imcsk8 has quit IRC | 20:18 | |
| *** imcsk8 has joined #openstack-infra | 20:18 | |
| *** MarkAtwood has joined #openstack-infra | 20:19 | |
| *** Ryan_Lane1 has quit IRC | 20:19 | |
| *** signed8bit has joined #openstack-infra | 20:19 | |
| clarkb | fungi: zaro I am going to continue on to 05 and 04 | 20:19 |
| clarkb | putting them in shutdown mode now | 20:19 |
| fungi | clarkb: sounds great | 20:19 |
| *** ChuckC has joined #openstack-infra | 20:20 | |
| *** talluri has joined #openstack-infra | 20:21 | |
| *** _nadya_ has quit IRC | 20:22 | |
| *** e0ne has joined #openstack-infra | 20:23 | |
| *** todd_dsm has quit IRC | 20:23 | |
| ashp | for now I'll let it clone from gerrit if it will, and fix the github bit later, :/ | 20:23 |
| ashp | damn, same ref error once it clones a copy from gerrit, shame :/ | 20:24 |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 20:24 | |
| *** amcrn has joined #openstack-infra | 20:26 | |
| *** oramraz has quit IRC | 20:26 | |
| *** amcrn has quit IRC | 20:26 | |
| *** talluri has quit IRC | 20:26 | |
| *** mmedvede has quit IRC | 20:27 | |
| *** signed8bit has quit IRC | 20:27 | |
| *** sarob has quit IRC | 20:28 | |
| openstackgerrit | Jan Kundrát proposed a change to stackforge/gertty: Allow specifying the path to CA certificate bundle https://review.openstack.org/130355 | 20:28 |
| *** sarob has joined #openstack-infra | 20:28 | |
| fungi | ashp: okay, so it has a working local clone but bombs trying to fetch the zuul ref? | 20:29 |
| fungi | ashp: that's at least a little easier failure to reason through | 20:29 |
| fungi | ashp: what does your apache error log have to say about the get? | 20:30 |
| ashp | nothing, it's completely empty | 20:30 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Pass args to stress tox call https://review.openstack.org/127561 | 20:30 |
| ashp | I am tempted to run the merger as www-data and make all of git owned by apache just in case | 20:30 |
| *** sarob has quit IRC | 20:30 | |
| *** sarob has joined #openstack-infra | 20:31 | |
| fungi | ashp: ours is world readable | 20:32 |
| ashp | Having said that if it was permissions I'd expect apache to freak out, so I have no idea :( | 20:32 |
| ashp | yeah, mine is too, I'm just... grasping at straws | 20:32 |
| ashp | as www-data I can manually read all the refs, so it's not that :/ | 20:33 |
| *** amcrn has joined #openstack-infra | 20:33 | |
| fungi | ashp: and the vhost is configured like shown at http://ci.openstack.org/zuul/merger.html#serving-zuul-git-repos ? | 20:34 |
| fungi | ashp: for the record, here's the vhost config template we're using... http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/zuul/templates/zuul.vhost.erb | 20:35 |
| *** gokrokve has joined #openstack-infra | 20:35 | |
| ashp | mine is pretty much identical to that, with slightly different paths, I checked :/ | 20:35 |
| ashp | while I was looking at this my jenkins gave up running jobs again, jesus :/ | 20:35 |
| ashp | this is not my day today | 20:36 |
| *** oramraz has joined #openstack-infra | 20:36 | |
| ashp | ok, it's definitely apache related as I see no hits in access.log when those jobs run | 20:37 |
| fungi | any chance the network between your jenkins and zuul servers is flaky? | 20:37 |
| ashp | OH, oh god, I know what it is | 20:37 |
| ashp | I'm so dumb. | 20:37 |
| ashp | oh dear :( | 20:37 |
| ashp | so... while testing this awful pipeline stuff I've had to stick in /etc/hosts entries due to a lack of dns | 20:38 |
| fungi | gah | 20:38 |
| ashp | jenkins build slaves... are not jenkins masters, and therefore ... | 20:38 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Cleanup grenade machinery in prep for Juno https://review.openstack.org/128973 | 20:38 |
| ashp | ;_; | 20:38 |
| ashp | so obvious! | 20:38 |
| ashp | still, rebuilding the slaves was worth it as this stuff is easier to add now, before I had a nest of gist scripts I had no permission to edit | 20:38 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/project-config: Stop running ceilometer mongodb jobs on icehouse https://review.openstack.org/130357 | 20:39 |
| *** baoli has joined #openstack-infra | 20:39 | |
| *** signed8bit has joined #openstack-infra | 20:39 | |
| fungi | i still shudder to think about automation driven from github gists | 20:39 |
| ashp | i replaced it with an ansible playbook for now | 20:39 |
| ashp | I don't know what the hell they were thinking | 20:40 |
| ashp | i think my favorite was the base64 encoded ssh private key in an ENV var | 20:40 |
| clarkb | boom! | 20:40 |
| *** cody-somerville has joined #openstack-infra | 20:41 | |
| fungi | j00 hax0red | 20:42 |
| *** Ryan_Lane has joined #openstack-infra | 20:43 | |
| ashp | once i get this actually running and live | 20:43 |
| *** cpowell has left #openstack-infra | 20:43 | |
| *** wenlock_ has joined #openstack-infra | 20:43 | |
| ashp | i can spend a month converting things from your puppet modules to docker containers built via ansible | 20:43 |
| *** koolhead17 has quit IRC | 20:44 | |
| ashp | I hope you're all gonna be in paris so I can buy you beers to make up for all my dumb questions | 20:44 |
| fungi | i don't think they'd let any of us not be in paris. we have to answer for our heinous crimes | 20:44 |
| ashp | excellent! I used to work for puppetlabs which is why I'm not using the puppet modules :D | 20:44 |
| fungi | heh | 20:45 |
| crinkle | ashp: :'( | 20:45 |
| ashp | crinkle: It's not that they are bad, (someofitisprettybad) it's that I need a break | 20:46 |
| ashp | anything but! | 20:46 |
| ashp | we're going for containers for openstack internally so I'll be containerizing all my QA stuff as I go | 20:46 |
| crinkle | lols | 20:46 |
| fungi | seems like this channel has started to turn into a puppetlabs refugee camp | 20:46 |
| *** jistr has quit IRC | 20:46 | |
| ashp | (and releasing it publically unless cisco torpedo it) | 20:46 |
| mtreinish | wow, so now that the juno jobs were added each tempest change has 23 dsvm jobs (all but 1 are voting) running in check | 20:47 |
| AJaeger_ | clarkb, fungi, anteaya: I see in layout.yaml some -nv dsvm jobs. Those should be non-voting. | 20:47 |
| fungi | ashp: if you want to give back, we have some fairly mind-numbing puppet module splitting to churn through ;) | 20:47 |
| fungi | ashp: http://specs.openstack.org/openstack-infra/infra-specs/specs/puppet-modules.html | 20:47 |
| AJaeger_ | But I see also dsvm jobs that are non-voting and don't end with -nv. Is there a special rule? | 20:48 |
| fungi | the more the merrier | 20:48 |
| mtreinish | AJaeger_: if it's the ironic ones, I think that's just a stale name. They were recently made voting | 20:48 |
| *** oramraz has quit IRC | 20:48 | |
| mtreinish | adam_g: ^^^ right? | 20:48 |
| *** sarob has quit IRC | 20:49 | |
| *** todd_dsm has joined #openstack-infra | 20:49 | |
| *** sarob has joined #openstack-infra | 20:49 | |
| AJaeger_ | mtreinish: I see patches like https://review.openstack.org/#/c/129509/ and https://review.openstack.org/#/c/124433/5/zuul/layout.yaml | 20:49 |
| clarkb | ashp: please let us know if you solve the iscsi problems | 20:49 |
| clarkb | ashp: also I Think kolla devs said neutron was problematic too | 20:50 |
| AJaeger_ | both add new non-voting dsvm ones - and I don't know whether they violate some convention | 20:50 |
| ashp | Yeah, we are taking the easy route and avoiding nova/neutron to start with, and focusing on the easy bits | 20:50 |
| ashp | but with time we'll eventually try our hand at fixes | 20:50 |
| clarkb | ashp: cinder is the hard one aiui | 20:50 |
| ashp | considering i'm doing this solo by the time we get there it'll probably be long fixed upstream | 20:50 |
| fungi | well, this is kernel driver level issues which need to be addressed ultimately | 20:51 |
| clarkb | and I think it was mordred who said kernel said wontfix | 20:51 |
| fungi | oh too fun! | 20:51 |
| clarkb | so it may need a super fancy hackaround | 20:52 |
| ashp | i really do have to toggle that checkbox on and off every 10 minutes | 20:52 |
| clarkb | ashp: is that with the older plugin version? | 20:53 |
| fungi | ashp: is there maybe a firewall between your zuul and your jenkins master which keeps aggressively timing out the state on that connection? | 20:53 |
| AJaeger_ | clarkb: regarding your python 2.6 deprecation mail: Please have a look at my patches, I think they're ready to be announced | 20:53 |
| clarkb | AJaeger_: oh good I will pull them up shortly | 20:53 |
| *** sarob has quit IRC | 20:54 | |
| ashp | fungi: it's just in ec2, sharing a security group, so I think not | 20:54 |
| ashp | clarkb: I'll have to downgrade and experiment | 20:54 |
| fungi | AJaeger_: i don't see any conventions violated with the naming on 129509 | 20:54 |
| *** tomoe_ has quit IRC | 20:54 | |
| openstackgerrit | A change was merged to openstack-infra/project-config: Stop running ceilometer mongodb jobs on icehouse https://review.openstack.org/130357 | 20:55 |
| AJaeger_ | fungi, so what is the convention on -nv usage? | 20:55 |
| fungi | AJaeger_: the -nv branch designator was just a means to provide an alternate job name which used the same job template, so that it could be made non-voting while leaving another voting version of the same job (basically) | 20:56 |
| *** sarob has joined #openstack-infra | 20:56 | |
| clarkb | yup and that way the only difference between the jobs is the name | 20:56 |
| clarkb | otherwise they stay in complete sync with each other | 20:56 |
| AJaeger_ | ah, so if we want to have a job *only* as non-voting, we don't need the -nv. | 20:57 |
| fungi | AJaeger_: it's a work around for zuul's lack of ability to make the same job voting on one project and non-voting on another | 20:57 |
| AJaeger_ | fungi: got it, thanks! | 20:57 |
| openstackgerrit | Steve Martinelli proposed a change to openstack-infra/project-config: Add jobs to api-wg repo https://review.openstack.org/130363 | 20:57 |
| *** patrickeast has quit IRC | 20:58 | |
| *** patrickeast has joined #openstack-infra | 20:58 | |
| *** patrickeast has quit IRC | 20:59 | |
| *** signed8bit has quit IRC | 21:00 | |
| stevemar | clarkb, beat me to the email explaining the description! | 21:00 |
| *** boris-42 has quit IRC | 21:00 | |
| *** signed8bit has joined #openstack-infra | 21:00 | |
| *** sarob has quit IRC | 21:01 | |
| *** tbeckham has joined #openstack-infra | 21:01 | |
| *** viglesias has joined #openstack-infra | 21:02 | |
| *** lbiasi has joined #openstack-infra | 21:02 | |
| *** otherwiseguy has quit IRC | 21:02 | |
| *** bswartz1 has quit IRC | 21:02 | |
| AJaeger_ | stevemar: 130363 will not work as is, you use build_sphinx and thus should simply use api-wg-docs for publishing. | 21:02 |
| ashp | [Wed Oct 22 21:02:04 2014] [crit] [client 172.31.242.118] configuration error: couldn't perform authentication. AuthType not set!: /metacloud/horizon/info/refs <-- progress, it's actually hitting the right thing | 21:03 |
| *** MaxV has quit IRC | 21:03 | |
| AJaeger_ | the tox variant is used by the documentation team repositories and they publish from a different directory | 21:03 |
| stevemar | AJaeger, ah! | 21:03 |
| AJaeger_ | stevemar: just publish to docs.o.o/developer/api-wg with the suual api-wg-docs job... | 21:04 |
| fungi | ashp: apache 2.2 or 2.4? | 21:05 |
| stevemar | AJaeger, my mistake! you are right, i', not sure if thats the final place, i'm waiting for jaypipes and annegentle to weigh in on that :) | 21:05 |
| ashp | fungi: 2.2, just changing to allow from all | 21:05 |
| fungi | aha | 21:05 |
| stevemar | AJaeger, i was thinking... maybe specs would be better | 21:05 |
| *** kgiusti has quit IRC | 21:05 | |
| fungi | ashp: but yeah, sounds like a mod_authz issue | 21:06 |
| *** boris-42 has joined #openstack-infra | 21:06 | |
| AJaeger_ | stevemar: then use the template openstack-specs-jobs - which uses api-wg-publish-specs | 21:07 |
| openstackgerrit | Steve Martinelli proposed a change to openstack-infra/project-config: Add jobs to api-wg repo https://review.openstack.org/130363 | 21:08 |
| stevemar | AJaeger_, yep, but that's not my call to make :) | 21:08 |
| *** aysyd has joined #openstack-infra | 21:08 | |
| *** mmedvede has joined #openstack-infra | 21:09 | |
| fungi | stevemar: yeah, i still am not entirely sure why the api working group isn't simply generating cross-project specifications suitable for the openstack-specs repo instead of using a separate one | 21:09 |
| *** pradk has quit IRC | 21:09 | |
| ekarlso | is anyone using the warlock stuff still for model stuff from jsonschema ? | 21:10 |
| *** david-lyle_ has joined #openstack-infra | 21:10 | |
| zaro | clarkb: everything ok with jenkins GP so far? | 21:11 |
| stevemar | fungi, the more i think about, specs makes more sense | 21:11 |
| *** david-lyle has quit IRC | 21:11 | |
| clarkb | zaro: up | 21:11 |
| clarkb | *yup | 21:11 |
| stevemar | fungi, even if it's just specs.o.org/openstack/api-wg-specs | 21:11 |
| fungi | stevemar: basically iterate on the design within a review change in wip, then make the recommendation by setting it ready for review, and acceptance of the spec happens by the tc approving that change | 21:12 |
| fungi | (or some semblance of that, just extrapolating from what i grokked in the ml thread and tc meeting) | 21:13 |
| AJaeger_ | fungi: yeah, should work | 21:14 |
| *** tsg has quit IRC | 21:14 | |
| fungi | i suspect it fell out the way it did simply because the cross-project specs discussion and api working group discussion were happening separately in parallel | 21:15 |
| *** AJaeger_ has quit IRC | 21:15 | |
| *** weshay has joined #openstack-infra | 21:16 | |
| fungi | converged on similar needs but ended up with separate implementations | 21:16 |
| *** bradjones has joined #openstack-infra | 21:16 | |
| *** dustins has quit IRC | 21:18 | |
| *** pc_m has quit IRC | 21:19 | |
| *** miqui has quit IRC | 21:20 | |
| *** andreykurilin_ has joined #openstack-infra | 21:21 | |
| *** esker has quit IRC | 21:25 | |
| *** dprince has quit IRC | 21:27 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: add a JJB command reference https://review.openstack.org/128106 | 21:27 |
| *** packet has quit IRC | 21:30 | |
| *** david-lyle_ is now known as david-lyle | 21:31 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Start testing Juno properly with grenade. https://review.openstack.org/128974 | 21:33 |
| *** tsg has joined #openstack-infra | 21:33 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Havana branches are dead, remove havana from d-g https://review.openstack.org/128975 | 21:33 |
| clarkb | woot! | 21:33 |
| mtreinish | clarkb: ^^^ nice | 21:33 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Add kilo to features.yaml https://review.openstack.org/128976 | 21:33 |
| ashp | my first success since starting this job | 21:33 |
| ashp | feels good! | 21:33 |
| ashp | a job actually ran to completion | 21:34 |
| *** mjturek has quit IRC | 21:34 | |
| ashp | oh wait spoke too soon an old python_functions.py blew up | 21:34 |
| *** dkliban is now known as dkliban_biab | 21:35 | |
| * fungi puts on his fancy hat and robes and proclaims kilo officially open! | 21:35 | |
| fungi | it even has upgrade testing now | 21:35 |
| ashp | a wizard hat and robes on irc normally... ends differently | 21:35 |
| clarkb | jogo if you propose an unrevert it should fail now | 21:37 |
| *** MarkAtwood has quit IRC | 21:41 | |
| *** rlandy has quit IRC | 21:42 | |
| clarkb | fungi: I need to step out and do some pre trip shopping. when I get back I plan on doing 04 and 05 | 21:42 |
| zaro | yo! review.o.o seems really slow | 21:43 |
| fungi | clarkb: thanks | 21:43 |
| clarkb | zaro: can you quantify that a bit more? it seems to be normally slow to me | 21:44 |
| zaro | no, it's just occasionally slow to load and sometimes i'll get an error. | 21:44 |
| zaro | not consistent though. anybody else having same problem? | 21:45 |
| fungi | cacti graphs for it look typical | 21:45 |
| zaro | maybe it's my home network. | 21:45 |
| *** bcourt has quit IRC | 21:46 | |
| jogo | clarkb: yes https://review.openstack.org/#/c/128974/ looks like it is doing the right thing for partial ncpu, its setting the nova.conf setting correctly and not restarting ncpu | 21:48 |
| jogo | clarkb: unrevert of what? | 21:49 |
| jogo | fungi: good catch, I forgot gate-grenade-forward is non voting as well | 21:49 |
| jogo | so it must have been like this for a while | 21:49 |
| *** markmcclain has joined #openstack-infra | 21:50 | |
| *** MarkAtwood has joined #openstack-infra | 21:50 | |
| *** hdd has joined #openstack-infra | 21:50 | |
| mtreinish | jogo: the oslo-middleware patch I think | 21:51 |
| openstackgerrit | Mehdi Abaakouk proposed a change to openstack-infra/project-config: oslo.messaging: Add experimental functional tests https://review.openstack.org/130370 | 21:51 |
| openstackgerrit | A change was merged to openstack-infra/project-config: Define experimental check-heat-dsvm-functional-postgres https://review.openstack.org/129509 | 21:52 |
| *** mriedem has quit IRC | 21:52 | |
| jogo | mtreinish: ahh, sweet | 21:52 |
| clarkb | jogo the nova change that broke juno to kilo | 21:54 |
| *** tdasilva has quit IRC | 21:58 | |
| *** patrickeast has joined #openstack-infra | 22:00 | |
| *** freyes has quit IRC | 22:04 | |
| *** homeless_ has joined #openstack-infra | 22:05 | |
| * zaro thinks, how to quantify review.o.o slowness? | 22:06 | |
| clarkb | if you run firebug or chrom* developer tools and make requests you should be able to see where slowness is | 22:07 |
| *** homeless has quit IRC | 22:08 | |
| *** hdd has quit IRC | 22:09 | |
| *** rfolco has quit IRC | 22:09 | |
| *** signed8bit has quit IRC | 22:12 | |
| ianw | clarkb: i assume the centos7 images got rebuilt OK? i'm seeing the job passing | 22:14 |
| cyeoh | fungi: thanks! Everything seems to be fine.. | 22:15 |
| *** mrodden has joined #openstack-infra | 22:16 | |
| *** __TheDodd__ has quit IRC | 22:17 | |
| *** adam_g is now known as adam_g_gone | 22:19 | |
| openstackgerrit | Christopher Yeoh proposed a change to openstack-infra/project-config: Adds homepage for API WG project https://review.openstack.org/130377 | 22:21 |
| fungi | cyeoh: awesome. glad i broke nothing (of consequence at least!) | 22:21 |
| *** dmsimard is now known as dmsimard_away | 22:21 | |
| *** ivar-lazzaro has quit IRC | 22:22 | |
| cyeoh | fungi :-) | 22:22 |
| *** ivar-lazzaro has joined #openstack-infra | 22:23 | |
| *** tonytan4ever has quit IRC | 22:25 | |
| *** hdd has joined #openstack-infra | 22:26 | |
| *** bknudson has quit IRC | 22:27 | |
| *** hdd has quit IRC | 22:27 | |
| *** andreykurilin_ has quit IRC | 22:28 | |
| jeblair | clarkb, fungi: i think we're still seeing load spikes on git servers | 22:28 |
| jeblair | clarkb, fungi: looks like git05 is having one right now | 22:29 |
| *** hdd has joined #openstack-infra | 22:29 | |
| *** david-lyle_ has joined #openstack-infra | 22:30 | |
| *** david-lyle has quit IRC | 22:30 | |
| *** ChuckC___ has joined #openstack-infra | 22:32 | |
| eharney | i would like to enable another system service in check-tempest-dsvm-full to experiment with a Cinder gate bug... can someone point me to where i would do such a thing? | 22:32 |
| *** mfer has quit IRC | 22:35 | |
| fungi | jeblair: eek, checking | 22:35 |
| jeblair | fungi: seem to be on the downward side of that... i don't think i got anything useful from the server | 22:36 |
| jeblair | doesn't seem to be correlated with a gate reset or anything | 22:36 |
| *** ChuckC___ has quit IRC | 22:37 | |
| *** ChuckC___ has joined #openstack-infra | 22:38 | |
| *** e0ne has quit IRC | 22:38 | |
| fungi | maybe that one wasn't related to stale images | 22:38 |
| *** hdd has quit IRC | 22:38 | |
| jeblair | fungi: yeah, the theory was maybe the stale image was pushing up the overall load | 22:40 |
| fungi | well, we do have stale images in places | 22:40 |
| *** stevemar has quit IRC | 22:40 | |
| fungi | i'm looking to see if i can narrow it down to specific providers or specific image types | 22:41 |
| jeblair | fungi: did you merge the patch that reverts the rax dib thing to get that image updated? | 22:41 |
| fungi | it merged before today's image updates ran, yes | 22:41 |
| jeblair | so i have seen some git processes that look like nova clones that are using 400% cpu | 22:41 |
| *** nelsnelson has quit IRC | 22:42 | |
| *** mfink has joined #openstack-infra | 22:42 | |
| fungi | hpcloud bare-centos6 seems to be hurting for a long time | 22:43 |
| *** dims_ has joined #openstack-infra | 22:44 | |
| fungi | also py3k-precise seems to be globally broken | 22:44 |
| fungi | as far as image updates are concerned | 22:44 |
| *** dims_ has quit IRC | 22:45 | |
| *** dims_ has joined #openstack-infra | 22:46 | |
| fungi | though it's also possible we ended up with a ready but broken image which is missing a cached copy of nova or something? | 22:46 |
| fungi | we've certainly seen that wreak havoc in the past | 22:46 |
| *** dims has quit IRC | 22:47 | |
| jeblair | i think it's specific git operation(s) triggering it; i think trying to map back to what git commands were going on might be useful | 22:48 |
| jeblair | but that's a few steps and i don't have the data anymore | 22:49 |
| *** r-daneel has quit IRC | 22:50 | |
| jeblair | it might just be that nova clones are more intensive than the last time we benchmarked this | 22:50 |
| *** hdd has joined #openstack-infra | 22:50 | |
| *** dims_ has quit IRC | 22:50 | |
| *** markmcclain has quit IRC | 22:51 | |
| fungi | i can certainly work on adding a few more git servers to the pool tomorrow if we think it's warranted | 22:51 |
| *** armax has joined #openstack-infra | 22:53 | |
| *** palar has quit IRC | 22:54 | |
| *** bdpayne_ has quit IRC | 22:55 | |
| fungi | 2014-10-22 22:53:57,369 INFO nodepool.image.build.hpcloud-b1.bare-centos6: Error: Cannot retrieve metalink for repository: epel. Please verify its path and try again | 22:55 |
| fungi | this may be the thing ianw was looking at | 22:55 |
| fungi | but that seems to be why we're not getting bare-centos6 image updates completing in hpcloud | 22:56 |
| *** msabramo has quit IRC | 22:57 | |
| *** baoli has quit IRC | 22:59 | |
| openstackgerrit | Eric Harney proposed a change to openstack-infra/devstack-gate: Enable lvmetad before creating VG https://review.openstack.org/130386 | 22:59 |
| fungi | ahh, nope, i was confusing it with https://review.openstack.org/130077 | 23:00 |
| jeblair | hrm, tracing the git processes through the lb is pretty hard | 23:00 |
| *** timrc is now known as timrc-afk | 23:01 | |
| jeblair | fungi: but every time i look at a process using >100% it's doing something with nova | 23:02 |
| fungi | we should squash nova into a single commit and start over | 23:02 |
| *** xyang0 has quit IRC | 23:02 | |
| *** shashankhegde has quit IRC | 23:03 | |
| openstackgerrit | Cody A.W. Somerville proposed a change to openstack-infra/devstack-gate: Improved error message for when branch not whitelisted in features yaml https://review.openstack.org/130388 | 23:03 |
| *** msabramo has joined #openstack-infra | 23:03 | |
| *** mfink has quit IRC | 23:04 | |
| *** dimtruck is now known as zz_dimtruck | 23:05 | |
| *** dangers is now known as dangers_away | 23:07 | |
| *** timrc-afk is now known as timrc | 23:08 | |
| reed | how do I tell gertty to refresh the list of repositories hosted on gerrit? | 23:08 |
| jeblair | reed: restart | 23:08 |
| jeblair | reed: maybe ctrl-r ? | 23:08 |
| reed | uhm I did... maybe I just need to wait longer? | 23:09 |
| *** ZZelle_ has quit IRC | 23:09 | |
| jeblair | not sure if that's implemented for project listing | 23:09 |
| jeblair | reed: what's missing? | 23:09 |
| reed | jeblair, I don't see openstack/api-wg for example | 23:09 |
| *** gokrokve_ has joined #openstack-infra | 23:09 | |
| *** shashankhegde has joined #openstack-infra | 23:10 | |
| reed | I'm also missing openstack-infra/project-config | 23:10 |
| jeblair | reed: yeah, i think exit/restart should do it | 23:10 |
| reed | ok | 23:10 |
| jeblair | reed: for system-config, extra work is needed (see fungi's ml post about the rename completion) | 23:10 |
| *** homeless has joined #openstack-infra | 23:10 | |
| jeblair | reed: (and don't forget 'L' to get the full list of unsubscribed projects) | 23:11 |
| reed | I see project-config now but still don't see api-wg ... | 23:11 |
| reed | I am on the full list | 23:11 |
| *** gokrokve has quit IRC | 23:12 | |
| *** micah has left #openstack-infra | 23:12 | |
| fungi | reed: when i toggle to the full project list i see openstack/api-wg | 23:12 |
| reed | jeblair, while I'm at it, what's the meaning of the number on the top right 'Sync'? | 23:12 |
| *** homeless_ has quit IRC | 23:13 | |
| jeblair | reed: number of pending tasks in the queue | 23:13 |
| reed | in gerrtty queue? it's like a progress status? | 23:13 |
| jeblair | reed: yep, counts down to 0 (nothing to do) | 23:14 |
| jeblair | reed: "sync project list" is such a task (the first one run on startup) | 23:14 |
| reed | I see an 'Error' there... and 114... I'll wait to 0 and check again | 23:14 |
| reed | thanks jeblair, fungi | 23:14 |
| jeblair | reed: ah, you may want to check .gertty.log | 23:14 |
| jeblair | reed: it should log the error there | 23:14 |
| *** dane_leblanc has quit IRC | 23:15 | |
| jeblair | reed: if the project sync failed, you'll probably need to start again.... | 23:15 |
| reed | oh, yeah, it failed :) | 23:15 |
| reed | 2014-10-22 16:10:31,400 Exception running task <SyncProjectListTask> | 23:15 |
| jeblair | reed: it's also entirely possible the project-config rename has actually broken things and you may be _required_ to either run the steps in fungi's message or just delete the db and start over... | 23:15 |
| jeblair | reed: i have not tested that. i think i do have a gertty config from pre-rename so i can test it later | 23:16 |
| jeblair | fungi: how do we repack the git servers? | 23:17 |
| *** msabramo1 has joined #openstack-infra | 23:17 | |
| lifeless | mtreinish: ay yup | 23:18 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: Add additional documentation for passing objects to templates https://review.openstack.org/130391 | 23:18 |
| jeblair | fungi: is there supposed to be a crontab or something? | 23:18 |
| *** msabramo2 has joined #openstack-infra | 23:19 | |
| jeblair | ah, yes, cgit user | 23:19 |
| *** alexpilotti has quit IRC | 23:19 | |
| *** msabramo has quit IRC | 23:19 | |
| fungi | jeblair: yep | 23:20 |
| reed | jeblair, it worked after following fungi's instructions, thanks | 23:20 |
| fungi | reed: in my defense, those were jeblair's instructions really, i merely cut and pasted them | 23:20 |
| jeblair | i guess we'll need to do that everytime a project is renamed.... | 23:20 |
| fungi | (with attribution of course!) | 23:20 |
| reed | double kudos for jeblair then :) | 23:20 |
| jeblair | fungi: i'm trying to ascertain whether we're repacking correctly | 23:21 |
| *** msabramo1 has quit IRC | 23:22 | |
| fungi | jeblair: that was why i assumed you asked, given the present anomalies | 23:23 |
| *** todd_dsm has quit IRC | 23:23 | |
| *** mfink has joined #openstack-infra | 23:24 | |
| *** aysyd has quit IRC | 23:25 | |
| *** MarkAtwood has quit IRC | 23:25 | |
| *** hemna is now known as hemna__ | 23:27 | |
| *** gokrokve_ has quit IRC | 23:27 | |
| *** freemanbrandon has joined #openstack-infra | 23:27 | |
| jeblair | i think they are | 23:27 |
| fungi | aha, and the py3k-precise image updates seem to have been broken when we merged the change to start installing zuul everywhere | 23:28 |
| *** marun has joined #openstack-infra | 23:28 | |
| *** bswartz has joined #openstack-infra | 23:30 | |
| *** dkliban_biab is now known as dkliban | 23:31 | |
| fungi | very weird... http://paste.openstack.org/show/123345/ | 23:34 |
| fungi | something is causing python-daemon's version_info.py to use a format string syntax the interpreter doesn't recognize | 23:35 |
| lifeless | fungi: 500 error on that paste | 23:37 |
| *** tsg has quit IRC | 23:37 | |
| fungi | lifeless: visit http://paste.openstack.org/ and then load http://paste.openstack.org/show/123345/ again. that seems to work around the cookie issue | 23:38 |
| lifeless | wow thats fucked up | 23:38 |
| lifeless | fungi: what version of python are you using? | 23:39 |
| fungi | lifeless: that's what i'm trying to ascertain. we have both 2.7 and 3.3 installed on that platform | 23:39 |
| lifeless | 3.3 then | 23:40 |
| fungi | was that format dropped in 3.x? | 23:40 |
| *** hdd has quit IRC | 23:40 | |
| *** eharney has quit IRC | 23:42 | |
| fungi | strangely, default python on those machines is 2.7 not 3.3 | 23:42 |
| *** gokrokve has joined #openstack-infra | 23:43 | |
| fungi | aha, yep. if i scroll further up it uses a python3.3 virtualenv | 23:43 |
| *** dims has joined #openstack-infra | 23:44 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/project-config: Split out github as it's own module https://review.openstack.org/126448 | 23:49 |
| *** wenlock_ has quit IRC | 23:49 | |
| *** sarob has joined #openstack-infra | 23:50 | |
| *** wenlock has quit IRC | 23:54 | |
| *** sarob has quit IRC | 23:54 | |
| *** lbiasi has quit IRC | 23:56 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/project-config: Improve logging around which images are cached https://review.openstack.org/130402 | 23:56 |
| clarkb | wow stepped away for a short time and everything happens | 23:56 |
| fungi | yep | 23:56 |
| jogo | fungi: ^ moar logs | 23:57 |
| *** gyee has quit IRC | 23:57 | |
| fungi | jogo: lgtm | 23:58 |
| clarkb | fungi: jogo it prints out every single image in the image build log that it wgets | 23:58 |
| *** teran_ has quit IRC | 23:59 | |
| clarkb | fungi: jogo have we at least confirmed that the image is not wgot? | 23:59 |
| fungi | clarkb: a grep in the logs did not find it | 23:59 |
| clarkb | and it appeared we did confirm that the devstack script spits it out as an image to cache | 23:59 |
| fungi | seems to be retrieving the fedora image we need for juno/master but not the one icehouse needs, which is different | 23:59 |
| fungi | and yes, on devstack's icehouse branch it does get listed by the list script | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!