Friday, 2016-05-20

mordredjeblair: I can confirm log streaming00:00
mordredjeblair: both of those seemed unhappy00:01
jeblairmordred: i agree; i have switched nodepool back00:01
*** jamesmcarthur has joined #openstack-infra00:05
*** lakshmiS has joined #openstack-infra00:05
*** dimtruck is now known as zz_dimtruck00:06
jeblairoh i wonder if $HOME is not set00:07
openstackgerritJohn L. Villalovos proposed openstack-infra/irc-meetings: Make sorting consistent on smaller devices
*** flepied1 has quit IRC00:07
*** pilgrimstack has quit IRC00:07
*** Swami has quit IRC00:10
*** thorst has joined #openstack-infra00:11
*** mixos has quit IRC00:11
*** thorst_ has joined #openstack-infra00:12
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: set $HOME
jeblairlook at all the things we're learning00:13
*** sflanigan has joined #openstack-infra00:13
*** sflanigan has joined #openstack-infra00:13
*** salv-orlando has joined #openstack-infra00:13
clarkbare you using the shell module?00:13
clarkbI want to say that shpuld set $HOME00:13
clarkbbut command wont00:14
jeblairclarkb: no, the zuul_runner module (so that we get redirected timestamped console logs)00:14
clarkbthis is why the d-g stuff uses shell over command iirc00:14
*** Qiming has quit IRC00:16
*** thorst has quit IRC00:16
*** csomerville has joined #openstack-infra00:16
*** sridhar_ram_ has joined #openstack-infra00:17
*** salv-orlando has quit IRC00:17
clarkbjeblair: you may also need to set localr stuff00:18
notmorganjeblair, mordred:
clarkbas libc default is very often not utf8 friendly00:18
notmorganjeblair, mordred: that can be dropped into .ssh/config and it should do exactly what you asked for, if you want to pre-populate the ECDSA keys it would be possible to make a global entry for them somewhere on thesystme00:19
openstackgerritMatthew Thode proposed openstack-infra/project-config: Move prepare-node to openstack-repos
notmorganjeblair, mordred: and/or move to stricthostkeychecking yes (if keys are pre-populated to a file)00:19
jeblairclarkb: hrm.  maybe we should make this a login shell?  what do you think would be better -- we set the minimal parameters explicitly, or we punt and source things like a login shell before running anything?00:19
* notmorgan left HashKnownHosts for simplicity sake00:20
clarkbjeblair I like punting but then distros may argue over what values shpuld be00:20
*** r-mibu has quit IRC00:20
jeblairnotmorgan: yes, thanks for thinking of the hash thing -- that's really helpful on systems like these00:20
*** kzaitsev_mb has quit IRC00:20
*** EricGonc_ has quit IRC00:20
clarkbnotmorgan: ecdsa? there still potentially be dragons there00:20
*** sdake has quit IRC00:21
notmorganclarkb: hostkeys are default to ECDSA00:21
openstackgerritSachi King proposed openstack-infra/zuul: Support post jobs by supporting rev checkout
notmorganbut can be rsa00:21
clarkbmaybe centos6 was the last one to only dsa and rsa00:21
notmorganclarkb: (this is server keys vs. user keys)00:21
*** sdake has joined #openstack-infra00:21
clarkbnotmorgan: we definitely had hosts that didmt ecdsa but I may be thinking of centos600:21
clarkbyes I know00:21
notmorganclarkb: wouldn't matter though00:21
notmorganif the keys are pre-populated to a central known file we could stricthostkeycheck00:21
*** sdake has quit IRC00:21
notmorganto eys00:22
notmorganotherwise ask is really the only "safe" option to use00:22
notmorganor "no"00:22
clarkbnotmorgan: there is no good way with opemstack to strict check00:22
*** pilgrimstack has joined #openstack-infra00:22
jeblairnotmorgan: what's 'ask'?00:22
notmorganclarkb: well this would be for known hosts like .o.o00:23
clarkbbecause you cant get hostkey out of banf00:23
notmorganclarkb: vs throw-aways00:23
prometheanfirecan't git-review?00:23
*** jamesmcarthur has quit IRC00:23
notmorganjeblair: the default behaviro "HEY found this unknown key, do you want to add it?"00:23
notmorganjeblair: and error if the key changes from the known host00:23
jeblairis that the same as 'yes'?00:23
clarkbnotmorgan: ya I think its a good idea to have something just pointing out the first one is always going to have some risk involved00:23
notmorganjeblair: yes requires the key to be there, no asking00:23
jeblairoh ok00:24
notmorganjeblair: if the key isn't there it fails00:24
clarkbmordred has asked for a feature in nova to handle this better00:24
notmorganclarkb: and that config basically says anything you're not reaching for via a hostname of * will not store hostkeys and not check00:25
notmorganclarkb: we could include lots of ips as well for known-long-lived things00:25
jeblairnotmorgan: cool, so i think we can use 'ask' when we get to a point where we have a source for host keys for our semi-permanent servers.  but we don't now, so we still probably want 'no' for (otherwise, we will need to do a lot of manual intervention when bringing up a new launcher)00:25
mordredclarkb: yah. I need to update that spec00:25
notmorganclarkb: but that is extending beyond the very simplle case.00:25
*** Sukhdev has joined #openstack-infra00:25
notmorganjeblair: sure. easy enough.00:25
*** r-mibu has joined #openstack-infra00:25
prometheanfireoh, it did work00:25
prometheanfirewonder why I didn't see it00:25
jeblairnotmorgan: (we only have host keys for a few servers so far; we can probably add more to hiera moderately easily)00:26
notmorganjeblair: i'll create a special entry for ones we care about next. let me figure out where to put this in system-config?00:26
mnaser(not sure if im tuned in but, if only puppet exported resources actually worked properly)00:26
notmorganjeblair: the idea will be " will store host keys", non-o.o will not store at all, and we'll have one we can add hosts to if we want hostkeychecking00:27
ianwhmmm ...
*** salv-orlando has joined #openstack-infra00:27
jeblairnotmorgan: yep00:27
notmorganjeblair: cool. i'll brew up some puppet for this. long term we def. want to pre-populate host-keys for know semi-permanent/permanent things. and do strict checking00:28
jeblairianw: is that supposed to have cycles?00:28
ianwjeblair : i would say no :)00:28
mordredmnaser: :)00:28
*** pvaneck has quit IRC00:28
mnasero/ mordred00:28
jeblairianw: neat!00:28
mnaseri hear you took this thing called days off00:28
*** salv-orlando has quit IRC00:30
openstackgerritMatthew Thode proposed openstack-infra/project-config: Remove node-devstack from nodepool
prometheanfireianw: jeblair: I don't think cycles mater for dib, since actual ordering is done other ways00:31
ianwyeah, the requirements are I guess in different phases00:31
prometheanfireianw: jeblair: element-deps is just a collector, not really a dep-graph00:31
*** ddieterly has joined #openstack-infra00:31
*** jamielennox|away is now known as jamielennox00:31
prometheanfireianw: check my last two reviews, they remove node-devstack00:32
*** mixos has joined #openstack-infra00:32
*** sdake has joined #openstack-infra00:32
*** sdake has quit IRC00:32
*** sarob has quit IRC00:32
*** piet has quit IRC00:36
*** ffrank has quit IRC00:36
*** sarob has joined #openstack-infra00:39
*** bpokorny_ has joined #openstack-infra00:39
*** Apoorva has quit IRC00:40
*** tqtran has quit IRC00:41
*** andreykurilin has quit IRC00:42
*** baoli has joined #openstack-infra00:43
*** andreykurilin__ has quit IRC00:43
*** bpokorny has quit IRC00:44
*** sarob has quit IRC00:44
*** bpokorny_ has quit IRC00:44
*** amotoki has joined #openstack-infra00:46
*** amotoki has quit IRC00:50
*** jamesmcarthur has joined #openstack-infra00:51
*** mixos has quit IRC00:53
*** ddieterly has quit IRC01:01
*** baoli has quit IRC01:02
*** zhurong has joined #openstack-infra01:02
*** jamesmcarthur has quit IRC01:03
*** mixos has joined #openstack-infra01:06
*** amotoki has joined #openstack-infra01:07
*** thorst_ has quit IRC01:07
*** ddieterly has joined #openstack-infra01:08
*** thorst has joined #openstack-infra01:08
*** EricGonczer_ has joined #openstack-infra01:08
*** amotoki_ has joined #openstack-infra01:10
*** sarob has joined #openstack-infra01:10
*** amotoki has quit IRC01:11
*** Apoorva has joined #openstack-infra01:12
*** hichihara has joined #openstack-infra01:14
*** sarob has quit IRC01:15
*** sdake has joined #openstack-infra01:15
*** ddieterly is now known as ddieterly[away]01:15
openstackgerritMorgan Fainberg proposed openstack-infra/system-config: Provide customized /etc/ssh/ssh_config
*** thorst has quit IRC01:16
*** Qiming has joined #openstack-infra01:16
*** sarob has joined #openstack-infra01:16
*** baoli has joined #openstack-infra01:17
notmorganjeblair, mordred, clarkb: ^ there is the 1st pass, but i def need to 2x check a couple things before it's ready01:18
notmorganjeblair, mordred, clarkb: i did not find a good pre-made module to cover complex ssh_config options (very basic ones only). I'll circle up and poke at a few of the puppet-experts we have on that and make sure i'm not doing something absurdly silly.01:19
notmorganalso... what nodes should get customized ssh things?01:19
*** jianghuaw has quit IRC01:20
openstackgerritSachi King proposed openstack-infra/zuul: Support post jobs by supporting rev checkout
*** sarob has quit IRC01:22
*** baoli has quit IRC01:23
*** sarob has joined #openstack-infra01:23
jheskethjeblair: finished reviewing your zuul changes fyi01:27
*** jamesmcarthur has joined #openstack-infra01:28
*** baoli has joined #openstack-infra01:28
*** Guest60864 is now known as jgriffith01:29
ianwgreghaynes / SpamapS : you might like to look at when around.  i think it could fit great with the booting test we want to get in01:30
clarkband Nakato has a new ps for zuul cloner change01:30
clarkbwill.have to try rereviewing tonight01:30
*** sflanigan has quit IRC01:31
prometheanfireianw: have gentoo be that test :P that'll help get eyes on this lol01:31
*** flepied has joined #openstack-infra01:31
*** jamesmcarthur has quit IRC01:34
*** salv-orlando has joined #openstack-infra01:37
*** ddieterly[away] is now known as ddieterly01:40
*** ddieterly is now known as ddieterly[away]01:40
openstackgerritTim Burke proposed openstack-dev/hacking: Fix H703 test
*** salv-orlando has quit IRC01:41
*** Clint has left #openstack-infra01:42
*** Apoorva has quit IRC01:44
*** baoli has quit IRC01:46
*** thorst has joined #openstack-infra01:47
*** thorst has quit IRC01:47
*** thorst has joined #openstack-infra01:47
*** jamesmcarthur has joined #openstack-infra01:48
*** baoli has joined #openstack-infra01:53
*** jamesmcarthur has quit IRC01:53
*** eliqiao has joined #openstack-infra01:54
*** thorst has quit IRC01:56
ianwprometheanfire: i think represents the conclusion i've come to pulling this apart01:57
*** piet has joined #openstack-infra01:57
* ianw goes to get a sandwich and resists temptation to make dib output graphviz01:57
*** baoli has quit IRC01:58
prometheanfireianw: Sam-I-Am knows all about sandwiches01:59
*** jamesmcarthur has joined #openstack-infra02:03
*** gomarivera has joined #openstack-infra02:03
*** banix has joined #openstack-infra02:06
*** gomarivera has quit IRC02:07
*** xiangxinyong has quit IRC02:11
*** xiangxinyong has joined #openstack-infra02:11
*** lakshmiS has quit IRC02:12
*** Sukhdev has quit IRC02:16
*** piet has quit IRC02:21
*** _prad_ has quit IRC02:23
*** vhosakot has joined #openstack-infra02:23
*** sridhar_ram_ has quit IRC02:25
*** amotoki_ has quit IRC02:27
*** xiangxinyong has quit IRC02:27
*** jamesmcarthur has quit IRC02:28
*** x00350071 has joined #openstack-infra02:28
openstackgerritMasayuki Igawa proposed openstack-infra/subunit2sql: Add session param doc of get_recent_failed_runs()
prometheanfireianw: cache-bindep depended on openstack_repos before the repos had a dep on node-devstack02:29
*** sarob has quit IRC02:29
*** yonglihe has joined #openstack-infra02:30
*** amotoki has joined #openstack-infra02:30
prometheanfireianw: should I change openstack_repos to dep on the new prepare-node element at the same time I create the element?02:31
*** flepied has quit IRC02:35
openstackgerritMerged openstack-infra/subunit2sql: Add optional start_date parameter to get_recent*() methods
*** gildub has quit IRC02:38
*** ddieterly[away] has quit IRC02:41
*** vhosakot has quit IRC02:42
*** vhosakot has joined #openstack-infra02:43
openstackgerritMatthew Thode proposed openstack-infra/project-config: Move prepare-node to it's own element
openstackgerritMerged openstack-infra/subunit2sql: Add session param doc of get_recent_failed_runs()
openstackgerritMerged openstack-infra/subunit2sql: Add link to published release notes in readme
*** jpr has joined #openstack-infra02:46
*** jamesmcarthur has joined #openstack-infra02:47
*** jpr1 has joined #openstack-infra02:47
*** csomerville has quit IRC02:49
*** jpr has quit IRC02:51
*** tlian has quit IRC02:51
openstackgerritMatthew Thode proposed openstack-infra/project-config: Remove node-devstack from nodepool
*** yuanying has quit IRC02:51
prometheanfireianw: ok, think those two reviews clean it up like your graph shows02:52
*** piet_ has joined #openstack-infra02:53
*** jamesmcarthur has quit IRC02:53
*** shashank_hegde has joined #openstack-infra02:54
*** thorst has joined #openstack-infra02:54
*** dims_ has quit IRC02:55
*** woodster_ has quit IRC02:58
*** dims has joined #openstack-infra03:00
notmorganmnaser: mordred is bad at taking days off03:00
notmorganmnaser: he totally should be better at it03:00
prometheanfirenotmorgan: we are all bad at that03:00
notmorganprometheanfire: but mordred is *really* bad at it03:01
*** thorst has quit IRC03:01
*** yuanying has joined #openstack-infra03:01
*** jamesmcarthur has joined #openstack-infra03:02
openstackgerritEthan Lynn proposed openstack-infra/project-config: Add gate-senlin-dsvm-tempest-api job to senlin's experimental queue
*** salv-orlando has joined #openstack-infra03:07
*** piet_ has quit IRC03:11
*** piet_ has joined #openstack-infra03:11
*** salv-orlando has quit IRC03:12
*** fawadkhaliq has joined #openstack-infra03:12
*** lakshmiS has joined #openstack-infra03:13
openstackgerritMerged openstack-infra/subunit2sql: Remove cap on oslo.db version
openstackgerritMerged openstack-infra/subunit2sql: Move oslo.concurrency to test-requirements
*** salv-orlando has joined #openstack-infra03:16
*** EricGonczer_ has quit IRC03:18
*** salv-orlando has quit IRC03:18
*** lakshmiS has quit IRC03:19
*** psachin has joined #openstack-infra03:23
*** jpr1 has quit IRC03:29
*** greghaynes has quit IRC03:31
*** bkero has quit IRC03:31
*** ramishra has quit IRC03:35
*** ramishra has joined #openstack-infra03:36
*** fawadkhaliq has quit IRC03:38
*** ramishra has quit IRC03:38
*** ramishra has joined #openstack-infra03:39
*** amotoki_ has joined #openstack-infra03:39
*** ramishra has quit IRC03:39
*** jamesmcarthur has quit IRC03:41
*** amotoki has quit IRC03:41
*** banix has quit IRC03:43
*** ramishra has joined #openstack-infra03:44
*** yuanying has quit IRC03:45
*** yuanying has joined #openstack-infra03:45
*** yuanying has quit IRC03:47
*** yuanying has joined #openstack-infra03:48
*** vhosakot has quit IRC03:48
*** harlowja_at_home has joined #openstack-infra03:49
*** x00350071 is now known as xiangxinyong03:50
*** shashank_hegde has quit IRC03:53
*** piet_ has quit IRC03:53
*** amotoki_ has quit IRC03:53
*** shashank_hegde has joined #openstack-infra03:53
*** Sukhdev has joined #openstack-infra03:54
*** amotoki has joined #openstack-infra03:58
*** jamesmcarthur has joined #openstack-infra03:58
*** armax has quit IRC03:58
*** thorst has joined #openstack-infra03:59
*** sdake has quit IRC04:00
*** oanson has joined #openstack-infra04:01
*** shashank_hegde has quit IRC04:03
*** psachin has quit IRC04:04
*** piet_ has joined #openstack-infra04:06
*** thorst has quit IRC04:06
*** harlowja_at_home has quit IRC04:07
*** shashank_hegde has joined #openstack-infra04:10
*** asselin__ has joined #openstack-infra04:11
*** links has joined #openstack-infra04:13
*** asselin has quit IRC04:15
*** ccamacho1 has joined #openstack-infra04:15
*** ccamacho has quit IRC04:15
*** fawadkhaliq has joined #openstack-infra04:16
*** amotoki has quit IRC04:17
*** psachin has joined #openstack-infra04:17
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Make sorting consistent on smaller devices
*** bhunter71 has quit IRC04:20
*** flepied has joined #openstack-infra04:20
*** piet_ has quit IRC04:23
*** piet has joined #openstack-infra04:23
*** Sukhdev has quit IRC04:25
*** Sukhdev has joined #openstack-infra04:26
*** sdake has joined #openstack-infra04:28
*** gomarivera has joined #openstack-infra04:29
*** sdake_ has joined #openstack-infra04:30
*** yamahata has joined #openstack-infra04:33
*** gyee has quit IRC04:33
*** yamamoto has quit IRC04:34
*** sdake has quit IRC04:34
*** gomarivera has quit IRC04:34
*** amotoki has joined #openstack-infra04:34
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add cloud-initramfs-growroot for Precise
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add release to pkg-map
*** salv-orlando has joined #openstack-infra04:37
*** mrmartin has joined #openstack-infra04:37
*** javeriak has joined #openstack-infra04:38
*** jamesmcarthur has quit IRC04:38
*** jcoufal has joined #openstack-infra04:39
*** gildub has joined #openstack-infra04:41
*** shashank_hegde has quit IRC04:43
*** jamesmcarthur has joined #openstack-infra04:45
*** shashank_hegde has joined #openstack-infra04:46
*** oanson has quit IRC04:46
openstackgerritSachi King proposed openstack-infra/zuul: Support post jobs by supporting rev checkout
*** mixos has quit IRC04:51
*** jamesmcarthur has quit IRC04:53
*** salv-orlando has quit IRC04:53
*** mrmartin has quit IRC04:56
*** javeriak has quit IRC04:56
*** javeriak has joined #openstack-infra04:58
*** kushal has joined #openstack-infra04:59
*** piet has quit IRC05:01
*** rlandy has quit IRC05:01
*** jaosorior has joined #openstack-infra05:02
*** thorst has joined #openstack-infra05:03
openstackgerritMadhuri Kumari proposed openstack-infra/project-config: Add new project Higginsclient to openstack repo
*** Nakato has quit IRC05:07
*** thorst has quit IRC05:11
*** timl0101 has quit IRC05:12
*** javeriak_ has joined #openstack-infra05:13
*** fawadkhaliq has quit IRC05:14
*** javeriak has quit IRC05:15
*** yamamoto has joined #openstack-infra05:19
*** flepied has quit IRC05:21
openstackgerritMadhuri Kumari proposed openstack-infra/project-config: Change Magnum Description
*** Sukhdev has quit IRC05:33
openstackgerritMadhuri Kumari proposed openstack-infra/project-config: Change Magnum Description
*** salv-orlando has joined #openstack-infra05:40
*** sree has joined #openstack-infra05:41
*** rcernin has joined #openstack-infra05:48
*** Qiming has quit IRC06:00
*** greghaynes has joined #openstack-infra06:02
*** Qiming has joined #openstack-infra06:02
*** YorikSar_ has quit IRC06:03
*** YorikSar has joined #openstack-infra06:05
*** thorst has joined #openstack-infra06:09
*** amotoki has quit IRC06:13
*** salv-orl_ has joined #openstack-infra06:14
*** kushal has quit IRC06:14
*** aeng has quit IRC06:15
openstackgerritAndreas Florath proposed openstack/diskimage-builder: [WIP] Add: lvm element
*** thorst has quit IRC06:16
*** salv-orlando has quit IRC06:17
*** rossella_s has quit IRC06:24
*** fawadkhaliq has joined #openstack-infra06:24
*** rossella_s has joined #openstack-infra06:24
*** flepied has joined #openstack-infra06:25
*** david-lyle_ has joined #openstack-infra06:26
*** SlickNik has quit IRC06:26
*** bmjen has quit IRC06:26
*** timburke has quit IRC06:26
*** bmjen has joined #openstack-infra06:26
*** SlickNik has joined #openstack-infra06:26
*** szaher has quit IRC06:27
*** ptoohill has quit IRC06:28
*** david-lyle has quit IRC06:28
*** boris-42 has quit IRC06:28
*** SamYaple has quit IRC06:28
*** LaT0rtue has joined #openstack-infra06:29
*** boris-42 has joined #openstack-infra06:29
*** SamYaple has joined #openstack-infra06:29
*** szaher has joined #openstack-infra06:29
*** flepied has quit IRC06:30
*** flepied has joined #openstack-infra06:31
*** Nakato has joined #openstack-infra06:31
*** ptoohill has joined #openstack-infra06:32
*** timburke has joined #openstack-infra06:32
*** jiggalator has joined #openstack-infra06:32
*** shardy has joined #openstack-infra06:35
*** netsin has quit IRC06:35
*** mrmartin has joined #openstack-infra06:36
*** mrmartin has quit IRC06:38
*** jokke_ has quit IRC06:45
*** jokke_ has joined #openstack-infra06:45
*** sflanigan has joined #openstack-infra06:46
*** salv-orl_ has quit IRC06:47
*** arxcruz has joined #openstack-infra06:52
*** nadya has joined #openstack-infra06:58
*** jaosorior has quit IRC07:00
*** jaosorior has joined #openstack-infra07:01
*** nadya has quit IRC07:01
*** jtomasek has quit IRC07:01
*** mrmartin has joined #openstack-infra07:10
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: networking-generic-switch-dsvm: Enable q-l3
*** kushal has joined #openstack-infra07:13
*** thorst has joined #openstack-infra07:14
yolandagood morning07:14
*** zeih has joined #openstack-infra07:15
*** ihrachys has joined #openstack-infra07:19
*** thorst has quit IRC07:22
dmelladomorning yolanda ;)07:24
*** hichihara has quit IRC07:24
*** akaszuba has joined #openstack-infra07:25
*** mikelk has joined #openstack-infra07:25
*** esikachev has joined #openstack-infra07:26
*** mrmartin has quit IRC07:27
*** ihrachys has quit IRC07:28
*** Qiming has quit IRC07:29
*** jlanoux has joined #openstack-infra07:29
*** vincentll has joined #openstack-infra07:30
*** salv-orlando has joined #openstack-infra07:32
*** slaweq has joined #openstack-infra07:33
*** bauzas is now known as bauwser07:34
openstackgerritGhe Rivero proposed openstack-infra/shade: Add volume quotas support
*** savihou has joined #openstack-infra07:36
*** matrohon has joined #openstack-infra07:39
*** yamahata has quit IRC07:41
*** sarob has joined #openstack-infra07:42
*** sarob has quit IRC07:47
*** Qiming has joined #openstack-infra07:47
*** dtantsur|afk is now known as dtantsur07:47
*** javeriak_ has quit IRC07:48
*** electrofelix has joined #openstack-infra07:49
*** jamielennox is now known as jamielennox|away07:49
*** electrofelix has quit IRC07:50
*** javeriak has joined #openstack-infra07:56
*** Qiming has quit IRC07:56
*** markvoelker has joined #openstack-infra07:56
*** nadya has joined #openstack-infra07:58
*** ihrachys has joined #openstack-infra07:59
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** nadya has quit IRC08:02
*** ihrachys_ has joined #openstack-infra08:02
*** daemontool has joined #openstack-infra08:03
*** ihrachy__ has joined #openstack-infra08:04
*** ihrachys has quit IRC08:05
*** Qiming has joined #openstack-infra08:05
*** akaszuba has quit IRC08:05
*** markvoelker has quit IRC08:05
*** ihrachy__ has quit IRC08:05
*** zhurong_ has joined #openstack-infra08:07
*** ifarkas has joined #openstack-infra08:07
*** ihrachys_ has quit IRC08:07
*** gildub has quit IRC08:08
*** shashank_hegde has quit IRC08:10
*** zhurong has quit IRC08:10
*** DevBox has joined #openstack-infra08:10
*** chem`` has joined #openstack-infra08:12
*** chem` has quit IRC08:14
*** amotoki has joined #openstack-infra08:15
*** salv-orlando has quit IRC08:15
*** jistr has joined #openstack-infra08:15
*** mrmartin has joined #openstack-infra08:16
*** salv-orlando has joined #openstack-infra08:16
*** javeriak has quit IRC08:19
*** thorst has joined #openstack-infra08:19
*** asettle has joined #openstack-infra08:20
*** mrmartin has quit IRC08:20
*** alex_xu has joined #openstack-infra08:21
*** thorst has quit IRC08:26
*** markusry has joined #openstack-infra08:32
*** lucas-afk is now known as lucasagomes08:32
*** sflanigan has quit IRC08:40
*** nadya has joined #openstack-infra08:41
*** derekh has joined #openstack-infra08:44
*** nadya has quit IRC08:48
*** mhickey has joined #openstack-infra08:56
*** YorikSar has quit IRC08:58
*** YorikSar has joined #openstack-infra09:00
*** Na3iL has joined #openstack-infra09:00
*** daemontool has quit IRC09:05
*** daemontool has joined #openstack-infra09:05
*** jyuso1 has quit IRC09:06
*** e0ne has joined #openstack-infra09:06
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config: Add virtualbmc from source non-voting test
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Add ironic cleaning job
openstackgerritGhe Rivero proposed openstack-infra/shade: Add network quotas support
*** dmk0202 has joined #openstack-infra09:19
*** Ravikiran_K has joined #openstack-infra09:20
*** thorst has joined #openstack-infra09:24
*** sambetts|afk is now known as sambetts09:24
*** Na3iL has quit IRC09:31
*** thorst has quit IRC09:31
*** savihou has quit IRC09:33
*** Na3iL has joined #openstack-infra09:33
*** jtomasek_ has joined #openstack-infra09:34
*** zhurong_ has quit IRC09:36
*** matrohon has quit IRC09:36
*** mhickey has quit IRC09:37
*** BobBall_AWOL is now known as BobBall09:38
openstackgerritMerged openstack-infra/tripleo-ci: Update 60-setup-testenvs to match what we use
openstackgerritchenying proposed openstack-infra/project-config: Add smaug fullstack job
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: Disable automated cleaning feature in ironic-inspector jobs
openstackgerritShu Muto proposed openstack-infra/project-config: Add Zaqar-UI translation settings
*** ccamacho1 is now known as ccamacho09:51
*** devananda has quit IRC09:52
*** nadya has joined #openstack-infra09:56
*** nadya has quit IRC10:00
*** nadya has joined #openstack-infra10:02
*** camunoz_ has quit IRC10:02
*** devananda has joined #openstack-infra10:03
*** abregman|nb is now known as abregman|afk10:05
*** amotoki has quit IRC10:05
*** sshnaidm has quit IRC10:06
*** sarob has joined #openstack-infra10:09
*** amotoki has joined #openstack-infra10:13
*** sdague has joined #openstack-infra10:14
*** sarob has quit IRC10:14
*** ihrachys has joined #openstack-infra10:15
*** LaT0rtue has quit IRC10:19
*** Qiming has quit IRC10:19
*** ihrachys has quit IRC10:20
*** rhallisey has joined #openstack-infra10:21
*** gildub has joined #openstack-infra10:21
*** javeriak has joined #openstack-infra10:23
*** rossella_s has quit IRC10:24
*** rossella_s has joined #openstack-infra10:24
*** ihrachys has joined #openstack-infra10:25
*** claudiub has joined #openstack-infra10:27
*** thorst has joined #openstack-infra10:28
*** nadya has quit IRC10:29
*** ihrachys has quit IRC10:31
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Allow the mirror server to be publicly accessible
*** amotoki has quit IRC10:35
*** thorst has quit IRC10:36
*** csaba has quit IRC10:43
*** alaski is now known as lascii10:43
*** _amrith_ is now known as amrith10:48
*** LaT0rtue has joined #openstack-infra10:52
*** yamamoto has quit IRC10:55
*** kzaitsev_mb has joined #openstack-infra10:56
*** javeriak has quit IRC10:59
*** jaosorior has quit IRC11:04
*** nadya has joined #openstack-infra11:11
*** zzzeek has quit IRC11:12
*** kushal has quit IRC11:12
*** Qiming has joined #openstack-infra11:13
*** gildub has quit IRC11:16
*** nadya has quit IRC11:16
*** gildub has joined #openstack-infra11:16
*** Na3iL has quit IRC11:24
*** nadya has joined #openstack-infra11:24
*** Ravikiran_K has quit IRC11:26
*** dguitarbite has quit IRC11:28
*** sagausi has joined #openstack-infra11:29
*** ccamacho is now known as ccamacho|lunch11:33
*** daemontool has quit IRC11:34
*** ldnunes has joined #openstack-infra11:35
*** jpr has joined #openstack-infra11:38
robcresswellAny chance of some eyes on this? It's giving us a ton of false failures right now. Temporary fix.11:39
robcresswell"us" being Horizon.11:39
*** daemontool has joined #openstack-infra11:41
*** baoli has joined #openstack-infra11:43
*** baoli_ has joined #openstack-infra11:44
Zarawe see similar problems with patches for the storyboard webclient; I don't feel confident that I understand the possible side-effects of upping the timeout, so I won't +1, but I'm lurking.11:45
*** weshay has joined #openstack-infra11:46
*** sree has quit IRC11:46
*** baoli has quit IRC11:47
*** xenthree3 has joined #openstack-infra11:49
*** xenthree3 has left #openstack-infra11:49
*** ihrachys has joined #openstack-infra11:49
*** EricGonczer_ has joined #openstack-infra11:52
*** banix has joined #openstack-infra11:52
*** banix has quit IRC11:54
*** javeriak has joined #openstack-infra11:54
*** raildo-afk is now known as raildo11:55
*** rfolco has joined #openstack-infra11:56
*** sagausi has quit IRC11:57
*** thorst has joined #openstack-infra12:00
*** markusry has quit IRC12:01
*** julim has joined #openstack-infra12:02
*** eliqiao has quit IRC12:02
*** lucasagomes is now known as lucas-hungry12:02
*** deadnull_ has joined #openstack-infra12:03
*** flwang has quit IRC12:03
*** deadnull_ has quit IRC12:03
*** vincentll has quit IRC12:04
*** deadnull_ has joined #openstack-infra12:04
*** eliqiao has joined #openstack-infra12:05
*** jtomasek_ has quit IRC12:05
*** deadnull_ has quit IRC12:05
*** claudiub has quit IRC12:05
*** jtomasek_ has joined #openstack-infra12:05
*** deadnull_ has joined #openstack-infra12:05
*** aysyd has joined #openstack-infra12:06
*** claudiub has joined #openstack-infra12:06
kamszShrews: is it possible to update quotas through shade? v've seen PR in ansible that leverages shade, but i'm having a hard time finding method in the documentation that was used12:07
*** EricGonczer_ has quit IRC12:08
*** vincentll has joined #openstack-infra12:10
Shrewskamsz: not yet. GheRivero has some patches up to add support. I just need to get around to reviewing them12:11
*** ddieterly has joined #openstack-infra12:11
*** flwang has joined #openstack-infra12:11
*** EricGonczer_ has joined #openstack-infra12:13
*** daemontool_ has joined #openstack-infra12:14
*** ddieterly is now known as ddieterly[away]12:14
*** dprince has joined #openstack-infra12:14
kamszShrews: alright, thanks12:15
*** banix has joined #openstack-infra12:15
*** jtomasek_ has quit IRC12:16
*** ayoung has quit IRC12:16
*** jtomasek has joined #openstack-infra12:16
*** daemontool has quit IRC12:17
*** piet has joined #openstack-infra12:18
*** flepied has quit IRC12:19
*** amotoki has joined #openstack-infra12:22
*** psilvad has joined #openstack-infra12:24
*** trown|outtypewww is now known as trown12:25
*** haypo has joined #openstack-infra12:26
haypohi. my change "Build also wheel packages for Python 3" was merged, but I see that gate-trove-python34-db still has to compile a lot of wheel packages:
*** sylwesterB has left #openstack-infra12:27
*** banix has quit IRC12:27
haypodoes someone know if my change jenkins/scripts/ is already effective or not?12:27
GheRiverothere is a ansible module in progress for the compute quotas, and for the network and volume, I hope to have something today12:27
haypodoes someone know what triggers the wheel mirror job? in i only see Mako-1.0.4-py2-none-any.whl with the timestamp 2016-04-0312:31
*** deadnull_ has quit IRC12:31
*** amotoki has quit IRC12:32
*** salv-orl_ has joined #openstack-infra12:33
clarkbhaypo I think it runs periodically like once per day12:34
clarkband probably has to succeed before the afs volume will publish changes12:36
*** pilgrimstack1 has joined #openstack-infra12:36
*** sarob has joined #openstack-infra12:36
haypoclarkb: are logs of this job public? i would like to check if it was executed and if it succeeded :)12:37
*** flepied has joined #openstack-infra12:37
haypoclarkb: this job would reduce the duration of most python3 job by 5-10 minutes!12:37
*** EricGonczer_ has quit IRC12:37
*** salv-orlando has quit IRC12:37
haypo(at least, for trove, i expect that the job will be 7 minutes shorter)12:38
clarkbyes should be under
*** pilgrimstack has quit IRC12:38
odyssey4meyolanda If you have a moment, could you please take a look at & ?12:39
*** markusry has joined #openstack-infra12:40
*** binbincong has joined #openstack-infra12:40
*** piet has quit IRC12:40
*** sarob has quit IRC12:41
robcresswellZara: Ah fair enough. Yeah, I left my thoughts as a comment on the patch. I'm not terribly happy about it, but need to get the gate moving for the contributors while the integration test guys figure out a more stable solution.12:41
*** bhunter71 has joined #openstack-infra12:42
*** gordc has joined #openstack-infra12:43
*** piet has joined #openstack-infra12:43
*** ddieterly[away] is now known as ddieterly12:43
haypoclarkb: oh, "pip3: command not found" :) it explains why nothing was built :)12:44
*** rlandy has joined #openstack-infra12:45
Zararobcresswell: yeah, I want to see how it goes since it affects us too, so I'll keep watching! :) I just don't know enough about the gate in general to know if lengthening test-times will cause anyone else problems, so I'd need to learn more about that first. our tests have been failing all over the place today, grr.12:46
*** _prad_ has joined #openstack-infra12:46
*** marcusvrn_ has joined #openstack-infra12:47
*** Na3iL has joined #openstack-infra12:49
*** julim has quit IRC12:49
*** savihou has joined #openstack-infra12:54
*** piet has quit IRC12:55
*** matt-borland has joined #openstack-infra12:55
*** salv-orl_ has quit IRC12:55
*** psachin has quit IRC12:56
*** julim has joined #openstack-infra12:57
*** salv-orlando has joined #openstack-infra12:58
*** tlian has joined #openstack-infra12:58
*** annegentle has joined #openstack-infra12:59
*** ddieterly has quit IRC12:59
*** lucas-hungry is now known as lucasagomes13:00
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add release announcement jobs for major OSA repos
*** Hal1 has quit IRC13:07
*** Hal1 has joined #openstack-infra13:07
openstackgerritVictor Stinner proposed openstack-infra/project-config: Fix for Python 3
*** javeriak has quit IRC13:08
openstackgerritMartin André proposed openstack-infra/project-config: Add tripleo-validations project to tripleo team
*** daemontool_ has quit IRC13:09
*** xyang1 has joined #openstack-infra13:10
*** gildub has quit IRC13:11
odyssey4methanks yolanda13:11
yolandaodyssey4me, np13:12
openstackgerritVitaly Kramskikh proposed openstack-infra/project-config: Switch npm-lint to non-voting for fuel-ui
vkramskikhkrotscheck: ^ I think we should disable voting for fuel-ui linting job until all the issues (I know at least 3) with NPM are resolved - we're almost blocked this week13:13
fungihaypo: i think you need `python3 -m pip ...` instead of `pip3 ...`13:14
haypofungi: hey, see 15:07 < openstackgerrit> Victor Stinner proposed openstack-infra/project-config: Fix for Python 3
*** _ari_ has joined #openstack-infra13:15
fungihaypo: ahh, yep, looks like that's exactly what you did13:15
haypofungi: the problem is that i'm unable to fix if python3 and/or pip for python3 are installed13:16
haypofungi: yep13:16
haypofungi: some weeks ago, i read that "pip" command should not be used, it can lead to issues, especially with virtualenv13:16
haypofungi: "python -m pip" (python2 -m pip, python2.7 -m pip, etc.) is preferred13:16
fungihaypo: there is no "pip for python 3"13:17
fungihaypo: pip3 is just a distro fabrication (a wrapper to invoke pip under a non-default copy of the interpreter)13:17
*** vincentll has quit IRC13:18
haypofungi: pip should be installed once for python 2, and once for python 3, no?13:18
*** vincentll has joined #openstack-infra13:18
haypofungi: i'm talking about /usr/lib/python3.4/site-packages/pip/13:18
fungihaypo: right, we don't use distro packaged pip13:18
fungihaypo: we have pip installed via the bootstrap script for it. ends up in /usr/local13:19
haypofungi: if you only install pip for python 2, python3 -m pip will fail, no?13:19
fungihaypo: we don't install pip "for" any specific interpreter13:19
*** salv-orl_ has joined #openstack-infra13:19
haypofungi: i'm asking if pip is also installed for python 3 (i don't care of /usr/bin/pip*)13:19
*** mdrabe has joined #openstack-infra13:20
fungipip is pure python and works under both python and python3 interpreters just fine13:20
haypofungi: ah. do you mean that "python2 -m pip" command can be used to install a package for python 3 ?13:20
fungipip is _installed_ for any python interpreter you care to call it under13:20
fungihaypo: don't use "python2" (python 2.x is called "python" according to the upstream python community, and python 3.x is called "python3")13:22
*** salv-orlando has quit IRC13:22
haypo"python" is sometimes "python3", like in virtual environements :)13:22
pabelangerI am cleaning up jenkins0113:22
haypofungi: for my change, i'm ok to write "python" if you prefer13:23
pabelangerthere is a large number of ready nodes attached to the server13:23
pabelangerover 120 nodes13:23
fungisome distros made the regrettable decision to symlink /usr/bin/python to python3 in their installations, and yes it can be a link to python3 in a virtualenv but that's a rarified environmen13:23
pabelangerI've stopped jenkins01 and in the process of deleting the nodes from nodepool13:23
openstackgerritMerged openstack-infra/irc-meetings: Cleanup old Trainers and Evoque meetings
openstackgerritMerged openstack-infra/irc-meetings: Make networking-midonet meeting every two weeks
*** javeriak has joined #openstack-infra13:24
openstackgerritMerged openstack-infra/irc-meetings: Add Cinder-Nova-API-interaction meeting series
openstackgerritMerged openstack-infra/irc-meetings: Add the new Requirements team meeting
openstackgerritMerged openstack-infra/irc-meetings: Remove the compass meeting
openstackgerritMerged openstack-infra/irc-meetings: Remove cue meeting
*** esikachev has quit IRC13:25
openstackgerritMerged openstack-infra/irc-meetings: Remove hierarchical-multitenancy-meeting
*** Na3iL has quit IRC13:26
pabelangerjenkins01.o.o started again13:26
openstackgerritMerged openstack-infra/irc-meetings: Remove mercador meeting
odyssey4mepabelanger hrm, I've clearly not quite done right - can you advise how better to do it? I didn't want to have this be enforced on all roles13:28
*** mhickey has joined #openstack-infra13:28
*** cloudtrainme has joined #openstack-infra13:29
*** amitgandhinz has joined #openstack-infra13:29
openstackgerritMerged openstack-infra/irc-meetings: poppy: Add meeting_id, chair name, & better description
openstackgerritMerged openstack-infra/irc-meetings: Add chair IRC nicks to Ironic meetings
openstackgerritMerged openstack-infra/subunit2sql: Enable returning run id in get_test_runs_by_status_for_run_ids()
pabelangerodyssey4me: you are missing the change for JJB. So zuul cannot find the job in jenkins.13:30
odyssey4mepabelanger it needs a change in jenkins/ clearly, but I wasn't sure where13:31
pabelangerodyssey4me: you'll need to add the template to ansible-role-jobs.yaml13:31
*** akshai has joined #openstack-infra13:31
pabelangerodyssey4me: should be able to use ansible-role-jobs template13:31
*** dansmith is now known as superdan13:31
*** anteaya has joined #openstack-infra13:31
anteayaI was on #openstick-infra for a minute13:32
anteayait is a lot quieter in there13:32
*** chuck__ is now known as zul13:33
*** ddieterly has joined #openstack-infra13:33
*** ayoung has joined #openstack-infra13:33
*** thorst has quit IRC13:34
odyssey4mepabelanger would it be better to just add '{name}-announce-release' to ansible-role-jobs for the release queue - wouldn't that then apply the release announcements to all roles, and thus put you in a fix for the ansible-role-* repositories?13:34
*** piet has joined #openstack-infra13:34
*** rodrigods has quit IRC13:34
odyssey4meI suppose for it to do anything you need to actually use the release process?13:34
*** rodrigods has joined #openstack-infra13:34
*** piet has quit IRC13:34
pabelangerodyssey4me: add where?13:35
odyssey4mepabelanger if it's added to jenkins/jobs/ansible-role-jobs.yaml for the ansible-role-jobs group13:36
pabelangerodyssey4me: if you add it to JJB, you are just saying create these jobs, which for the most part is fine.  Add it to zuul, says you want to run it (the job) for these projects.  So, your layout.yaml is correct (you have limited it to specific projects). You just need to generate all the jenkins jobs for all ansible roles.13:37
pabelangerodyssey4me: Yup, add '{name}-announce-release' and you should be fine13:37
pabelangerto ansible-role-jobs group in JJB13:38
*** salv-orl_ has quit IRC13:38
anteayazuul's eth0 outbound is spiking and queue lengths are growing even though the top of the gate is 7 minutes away from being finished:
pabelanger#status log I've stopped jenkins01.o.o, it doesn't appear to be working properly.  Nodes attach to jenkins but are not launched by nodepool.  I believe zl01 might be the issue13:39
*** pabelanger has quit IRC13:39
*** pabelanger has joined #openstack-infra13:39
pabelanger#status log I've stopped jenkins01.o.o, it doesn't appear to be working properly.  Nodes attach to jenkins but are not launched by nodepool.  I believe zl01 might be the issue13:40
openstackstatuspabelanger: finished logging13:40
*** thorst has joined #openstack-infra13:40
pabelangerjeblair: when you have a moment^ if you have some thoughts about jenkins0113:40
anteayazl01 doesn't seem to have a presence in cacti13:41
odyssey4mepabelanger oh hang on, I need corresponding jobs in jenkins/jobs/projects.yaml right?13:41
pabelangerodyssey4me: Right, but adding it into ansible-role-jobs.yaml is the easier way. Otherwise, you need to manually add the announce-release to specific projects13:42
*** ddieterly is now known as ddieterly[away]13:42
pabelangeranteaya: Ya, jeblair is still testing the system. And likely overlooked by mistake13:43
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add release announcement jobs for major OSA repos
odyssey4mepabelanger ^ like that then?13:44
*** daemontool has joined #openstack-infra13:44
*** thorst has quit IRC13:44
anteayapabelanger: okay13:44
pabelangerodyssey4me: if you go that route, you need to add it to the other projects you've tagged in zuul too13:44
pabelangerodyssey4me: moving it to ansible-role-jobs.yaml, it will be a single line patch.13:45
anteayazuul queued events have dropped13:45
*** thorst has joined #openstack-infra13:46
openstackgerritVictor Stinner proposed openstack-infra/project-config: Fix for Python 3
*** links has quit IRC13:46
*** thorst has quit IRC13:47
*** thorst has joined #openstack-infra13:47
odyssey4mepabelanger ah, so just add to ansible-role-jobs, then remove the 'release-announce-jobs' template entries per-repo because it's included with the 'ansible-role-jobs' template, and also there's no need for the 'release-announce-jobs' macro at all?13:48
pabelangerodyssey4me: I'll comment on the review13:48
odyssey4mepabelanger thanks13:49
*** dtardivel has joined #openstack-infra13:49
*** kgiusti has joined #openstack-infra13:49
pabelangerodyssey4me: comment left13:51
odyssey4mepabelanger replied - that particular repo is a special case because it doesn't use ansible-role-jobs for test execution13:52
*** zhurong has joined #openstack-infra13:52
krotscheckvkramskikh: Well, the hope was that would land.13:53
krotscheckvkramskikh: because it's not just you.13:54
pabelangerodyssey4me: replied13:54
krotscheckBut no cores have looked at it.13:54
krotscheckHas anyone figured out what is up with our mirrors returning partial files?13:54
openstackgerritKirill Bespalov proposed openstack-infra/project-config: add reno jobs for oslo.messaging
odyssey4mepabelanger Is it not already in ansible-role-jobs?
krotscheckinfra-core: I'd like to disable (and unblock) the javascript jobs by temporarily switching back over to the public npm mirrors.
*** esikachev has joined #openstack-infra13:55
pabelangerodyssey4me: Oh, ya. Some how I missed that13:55
pabelangerodyssey4me: I blame not having coffee13:55
pabelangerodyssey4me: jenkins should be happy now13:55
* odyssey4me hands coffee to pabelanger 13:56
*** sarob has joined #openstack-infra13:56
*** sarob has quit IRC13:56
*** kushal has joined #openstack-infra13:56
*** sarob has joined #openstack-infra13:57
*** thorst has quit IRC13:57
*** ddieterly[away] is now known as ddieterly13:58
vkramskikhkrotscheck: but without mirrors there will be ECONNREFUSED error and that "module not found" error in npm 3.5.2 from Xenial, so the job will still be not functional13:58
*** thorst has joined #openstack-infra13:58
krotscheckvkramskikh: You didn't have problems last week, right?13:58
krotscheckvkramskikh: Xenial went in 3 weeks ago.13:58
*** _sarob has joined #openstack-infra13:58
*** ccamacho|lunch is now known as ccamacho14:00
vkramskikhI had that ECONNREFUSED error. ok, I'll put workflow -1 on my patch and see if your patch resolves the issues14:00
krotscheckvkramskikh: We can do a quick check, just a minute14:00
*** sarob has quit IRC14:01
krotscheckvkramskikh: Updated 31868714:01
vkramskikhthanks, let's see14:03
*** rlandy has quit IRC14:05
*** daemontool has quit IRC14:06
*** jiggalator has quit IRC14:08
*** sigmavirus24_awa is now known as sigmavirus2414:09
*** _sarob has quit IRC14:11
jeblairpabelanger: i reassigned all nodes that nodepool sent to zuul to jenkins01, because nodepool can't handle a target being removed from its configuration14:12
jeblairpabelanger: they should get cleaned up eventually14:12
pabelangerjeblair: okay, so safe to bring jenkins01 back online now?14:13
*** hemanthm|afk is now known as hemanthm14:14
jeblairpabelanger: i think so -- if the only thing you saw was nodes in 'nodepool list' that didn't show up on jenkins itself, and those nodes had 'zl01' as the manager, then yeah, should be ok14:14
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote OSA infrastructure CentOS jobs to non-voting
pabelangerjeblair: okay, starting. Will monitor14:16
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote OSA infrastructure CentOS jobs to non-voting
vkramskikhkrotscheck: it passed, i'll try a few more times14:17
krotscheckThat works14:18
Shrewsjeblair: any tips on trying to see the actual playbooks with your change in 318397? i mean, it looks fine, but would like to see the actual output somehow14:19
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote OSA infrastructure CentOS jobs to non-voting
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote OSA infrastructure CentOS jobs to non-voting
jeblairShrews: so far, i've only done so by running it for real, and launching a job, after commenting out the bit that deletes the JobDir on completion.  i think we shoul probably a) add an option to avoid deleting jobdirs :) and b) maybe make some kind of utility that will take a jjb config and construct the jobdir for it.  that may take some minor refactoring.14:21
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add release announcement jobs for major OSA repos
jeblair(i think that will be a useful utility to have in v3 as well14:22
odyssey4mesorry about the spam, I edited the wrong patch :/14:22
*** yamamoto has joined #openstack-infra14:22
*** yamamot__ has joined #openstack-infra14:23
*** akshai has quit IRC14:23
Shrewsjeblair: ack14:23
pabelangerjeblair: if you want to take a quick peak at jenkins01, we've attached over 40 nodes now, but none are being used14:23
*** regXboi has joined #openstack-infra14:24
*** sree has joined #openstack-infra14:24
*** pilgrimstack1 has quit IRC14:24
jeblairpabelanger: did you start/stop jenkins itself?14:25
pabelangerjeblair: I did, I put it into shutdown and clean up the nodes this morning14:25
pabelangerjeblair: and just started jenkins01 now14:25
jeblairpabelanger: ah, it is running jobs, but nodepool isn't getting zmq events from it14:25
jeblairpabelanger: we should try restarting nodepool14:26
jeblairpabelanger: you want to do that?14:26
pabelangerjeblair: sure, I can14:26
jeblair(this is why we're never touching zmq again)14:26
jeblairthere it goes14:27
pabelangerokay, stop / start on nodepool14:27
jeblairpabelanger: i see "Received: onCompleted" from jenkins01 in the nodepool log now14:27
fungipabelanger: yeah, if you see a jenkins master with lots of in-use or disabled nodes which nodepool thinks are still in a ready state, that's generally a sign that something (network hiccup?) caused the zmq connection to go dead14:27
*** yamamoto has quit IRC14:27
pabelangerfungi: okay, good to know14:29
*** Na3iL has joined #openstack-infra14:29
pabelangerjeblair: fungi: will nodepool bleed of the current ready nodes on jenkins01 after some time?14:30
jeblairpabelanger: they'll get used and nodepool will get the complete event and delete them; ones that are completed already will take longer to atrophy; they may be worth a manual cleanup in an hour or so (when they will be easier to spot)14:31
*** akshai has joined #openstack-infra14:31
jeblairpabelanger: in the mean time, nodepool will under-provision since it thinks it has more nodes ready; if we are in dire straights, it might be good to manually clean up now.14:31
pabelangerjeblair: Okay, I have a shell script I can run quickly to delete the ready nodes14:32
pabelangerwill take a minute14:32
*** tonytan4ever has joined #openstack-infra14:32
jeblairpabelanger: well, just realize that some of the nodes that nodepool thinks are ready are running jobs right now14:32
jeblairyou have to either wait an hour or correlate it with what jenkins says14:32
pabelangerjeblair: okay, I'll wait for a bit14:33
mtreinishjeblair, nibalizer, pabelanger: if you get a sec can you take a look at:
fungipabelanger: yeah, i usually wait a couple hours and then start deleting any nodes which nodepool says have been in a "ready" state since around the time of the restart or longer14:34
mtreinishit adds the setup bits need for openstack-health to use elastic-recheck (per )14:34
fungipabelanger: depending on how node-starved we are, i sometimes give it 3 hours since there are some ironic/tripleo jobs which run in the 2.5-hour neighborhood14:34
*** banix has joined #openstack-infra14:36
scottdaHi Infra Team. Cinder is wanting to enable multi-backend for Tempest single-node Devstack tests, a one-line change. Where is the best place to do this, in nodepool?14:40
openstackgerritSean M. Collins proposed openstack-infra/project-config: net-generic-switch: Do not create Neutron debug probes
*** trown is now known as trown|afk14:42
*** zeih has quit IRC14:43
*** zeih has joined #openstack-infra14:44
*** daemontool has joined #openstack-infra14:44
*** javeriak has quit IRC14:44
*** hongbin has joined #openstack-infra14:45
*** ramishra has quit IRC14:45
*** ramishra has joined #openstack-infra14:47
*** mtanino has joined #openstack-infra14:48
*** zeih has quit IRC14:49
pabelangerjeblair: Do we change any of the default MTU setting on afsd?  I am trying to debug some AFS failures in tripleo-cloud-test-rh1 which I think are related to the NAT in tripleo14:49
zigofungi: jeblair: pabelanger: After 2 days, it still looks like some jenkins have the old version of the pkgos-build-openstack-pkg-tools script, as the check job works, but the gate one doesn't. Please see:
zigoBoth ScotK and Igot Belikov are saying it's probably a Jenkins that has an outdated version of the job.14:50
jeblairpabelanger: you're not running afsd on a pnat are you?14:50
zigoSince we don't have access to Jenkins anymore, I can't check...14:50
*** inc0 has joined #openstack-infra14:51
*** kien-ha has joined #openstack-infra14:53
pabelangerjeblair: the AFS client appears to be behind a RHEL box doing the NAT translation. That's how much I currently know14:53
jeblairpabelanger: does it have a public  ip?14:54
pabelangerjeblair: yes, you'll have credentials14:55
*** kushal has quit IRC14:55
jeblairpabelanger: so it's straight nat, not pnat?14:55
pabelangerjeblair: I believe to, let me confirm14:56
pabelangerjeblair: Yes, a straight nat14:57
*** nwkarsten has joined #openstack-infra14:58
jeblairpabelanger: what's the problem you're seeing?14:58
*** arxcruz has quit IRC14:59
*** EricGonczer_ has joined #openstack-infra15:00
pabelangerjeblair: afs is operating slow. A basic ls on /afs/openstack simply hangs.  Here is a tcpdump of an attempt:
fungizigo: that example got its job definition from jenkins02. i'll check its configuration state15:01
zigofungi: cheers!15:01
fungibetting we're timing out trying to apply a large set of job updates on our slower jenkins masters for some days now15:01
*** EricGonc_ has joined #openstack-infra15:01
fungi-rw-r--r-- 1 jenkins nogroup 4046 May 14 01:11 /var/lib/jenkins/jobs/pkgdeb-build-openstack-pkg-tools/config.xml15:02
pabelangerjeblair: okay, here is some more info about NAT:
fungiso it's been 6 days since that job was last modified there15:02
pabelangerjeblair: some ports are forwarded15:02
*** tonytan4ever has quit IRC15:03
*** zhurong has quit IRC15:03
fungii'll start a jjb update on jenkins02 in a root screen session and check the other masters for similar signs of outdatedness15:03
jeblairpabelanger: i'm sorry.  we only work with servers that are on the internet.  that server is not on the internet.  i would like to reiterate that i think it is past time for us to work on moving the tripleo cloud to a third-party ci model.15:04
*** xarses has quit IRC15:04
fungioh, nice, broken jjb cache on jenkins0215:04
fungithat might explain it15:05
*** EricGonczer_ has quit IRC15:05
pabelangerjeblair: okay, I am unsure how to proceed with moving tripleo to third party15:05
openstackgerritMerged openstack-infra/tripleo-ci: Revert "Test with latest master heat"
*** cloudtrainme has quit IRC15:06
*** javeriak has joined #openstack-infra15:06
fungipabelanger: tripleo sysadmins use puppet-openstackci to deploy zuul/jenkins/nodepool somewhere in their network and point it at a tenant in the cloud they're providing?15:07
*** cody-somerville has joined #openstack-infra15:07
pabelangerfungi: jeblair: Right, but they don't want to leave. That is the issue15:08
jeblairpabelanger: i think we have two choices -- we can either work with the tripleo folks on a timeline and help them move operation of the zuul/nodepool component internally (like what fungi said).  or we can just say "we're going to turn it off in 4 months".  i'd prefer the former, but i don't have time to drive it.  i'm happy to help where i can though.15:08
fungipabelanger: and hopefully they don't turn it into a rathole to get those puppet modules working on rhel simply so that they can avoid installing non-rh distros inside their environment, or else it will drag on forever15:08
pabelangerright, so I think it will have to come to that. Impose some sort of migration date15:09
*** eventingmonkey has joined #openstack-infra15:10
*** asettle has quit IRC15:12
*** woodster_ has joined #openstack-infra15:12
zigofungi: Thanks a lot for your check...15:12
openstackgerritNathan Reller proposed openstack-infra/project-config: Add KMIP Experiment Check to Barbican
fungizigo: this will take a while to clean up, but i'll let you know once it's on track everywhere15:12
zigofungi: Please ping me when I should re-attempt to do a "recheck".15:12
fungiwill do!15:13
openstackgerritMerged openstack-dev/hacking: Fix H703 test
*** yamahata has joined #openstack-infra15:14
fungiall the other jenkins masters seem to be reasonably up to date. i'm going to put jenkins02 into prepare for shutdown so that it doesn't run any new jobs until the jjb updates complete. that should get it over with sooner and also avoid using stale job configs in the meantime15:15
*** akshai has quit IRC15:15
zigoI just wonder why I got *always* the same result. Bad luck and always got to jenkins02 ? :P15:17
fungiwell, it may not be the _only_ problem15:17
fungibut with this out of the way we can narrow the issue down more easily15:17
*** markvoelker has joined #openstack-infra15:18
*** markvoelker has quit IRC15:18
*** markvoelker has joined #openstack-infra15:18
fungizigo: okay, jenkins02 is in prepare for shutdown mode now, so it shouldn't pick up any new jobs until we bring it back into full service. if you recheck that it will hopefully pass and if not then we can move on to investigating whatever the next issue is with it15:19
*** xarses has joined #openstack-infra15:20
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: include PATH
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: add -q to rsync ssh command
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: create parent directories for SCP
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: several fixes
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: handle Ant paths in publishers
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: set $HOME
*** akshai has joined #openstack-infra15:21
openstackgerritNathan Reller proposed openstack-infra/project-config: Add KMIP Experiment Check to Barbican
*** shashank_hegde has joined #openstack-infra15:25
*** vincentll has quit IRC15:25
*** cloudtrainme has joined #openstack-infra15:26
openstackgerritJames E. Blair proposed openstack-infra/zuul: Estimate job runtimes internally
*** roxanagh_ has joined #openstack-infra15:26
zigofungi: Ok, doing the recheck. Thanks again.15:26
*** ri0 has joined #openstack-infra15:28
*** Sukhdev has joined #openstack-infra15:28
*** kushal has joined #openstack-infra15:28
*** Sukhdev has quit IRC15:29
*** Sukhdev has joined #openstack-infra15:30
*** daemontool_ has joined #openstack-infra15:31
*** lakshmiS has joined #openstack-infra15:31
*** sree has quit IRC15:33
*** Qiming has quit IRC15:33
*** mixos has joined #openstack-infra15:33
*** daemontool has quit IRC15:33
*** DevBox has quit IRC15:35
*** armax has joined #openstack-infra15:35
*** jsavak has joined #openstack-infra15:36
*** roxanagh_ has quit IRC15:37
*** mixos has quit IRC15:37
*** bkero has joined #openstack-infra15:38
*** mikelk has quit IRC15:38
*** jamesmcarthur has joined #openstack-infra15:39
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Retire openstack-ansible-py_from_git repository
*** jamesmcarthur has quit IRC15:41
*** kushal has quit IRC15:45
zarofungi: was able to figure out how to force running online re-index15:46
*** ri0 has quit IRC15:46
*** jamesmcarthur has joined #openstack-infra15:47
*** Swami has joined #openstack-infra15:48
fungizaro: oh? what bit of magic were we missing last time we tried?15:48
zarofungi: do you happen to know the number of cores on review.o.o?15:49
fungizaro: i'm willing to give it a shot in the upcoming rename maintenance, but will still plan for an offline reindex in case online still doesn't do what we need15:49
jeblair--for-realzies ? :)15:49
*** Nakato has quit IRC15:49
zarofungi: there's a groovy script that luca wrote to do it,
fungizaro: `grep -c ^processor /proc/cpuinfo` says "16"15:50
zarofungi: it works but the only problem is that it doesn't respect gerrit access rules15:51
*** rcernin has quit IRC15:51
*** asettle has joined #openstack-infra15:52
krotscheckAny other infra-cores able to look at ? Most of the JS jobs are borked because mirrors.15:52
vkramskikh 5 successful runs so far, so +1 for merge of ^15:53
zarofungi: i'm not sure if it's possible to enforce access rules from a script but i'm pretty sure not.15:53
krotscheckjeblair: Was that log from yesterday useful?15:53
*** markvoelker has quit IRC15:54
*** jamesmcarthur has quit IRC15:54
jeblairkrotscheck: sorry i missed that.15:55
openstackgerritMerged openstack-infra/tripleo-ci: Add timeout to get_host_info task
krotscheckjeblair: No worries. I'm guessing the segfault problems you've run into are taking up your time?15:55
jeblairkrotscheck: nope, i'm working on zuul15:55
krotscheckjeblair: Ah. hrm. Is anyone working on figuring out why we got partial downloads?15:56
*** inc0 has quit IRC15:56
jeblairkrotscheck: i do not know the answer to that15:56
*** asettle has quit IRC15:56
fungikrotscheck: i'm thinking at this point the only additional info we're going to get out of apache on the random segfaults is to run with coredumps enabled and load those into a debugger to get backtraces15:56
krotscheckfungi: Hrm. This sounds like something better done on a non-infra vm.15:57
fungikrotscheck: i don't know whether we can reasonably recreate it without building a custom load generator otherwise15:57
krotscheckfungi: I can generate load, but I may not be able to generate infra-levels of load.15:58
fungiwe're not doing anything particularly fancy with our apache configuration and have a fairly minimal set of standard modules in play15:58
*** jamesmcarthur has joined #openstack-infra15:58
krotscheckfungi: I'd be willing ot help, but I have zero experience reading coredumps. Or, well, anything below the web tier, really.15:59
*** sdake_ has quit IRC15:59
*** Apoorva has joined #openstack-infra15:59
fungiyeah, i've got the apache docs in hand and am going to give it a whirl here shortly. just want to make sure it has my undivided attention in case it causes any unexpected service disruption16:00
*** cody-somerville has quit IRC16:00
*** sdake has joined #openstack-infra16:01
*** salv-orlando has joined #openstack-infra16:02
openstackgerritMário Santos proposed openstack-infra/shade: Add support for changing metadata of compute instances
*** slaweq has quit IRC16:03
*** cody-somerville has joined #openstack-infra16:03
*** cody-somerville has quit IRC16:03
*** cody-somerville has joined #openstack-infra16:03
openstackgerritMerged openstack-infra/project-config: Temporarily start using the public registry again
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Retire openstack-ansible-py_from_git repository
*** vhosakot has joined #openstack-infra16:03
krotscheckfungi: Alright, need me to get some load generation thingies handy?16:03
fungikrotscheck: it seems to happen with sufficient regularity on mirror.ord.rax.o.o; i'm going to see if it's happening elsewhere before i plan16:05
zigofungi: It's looking green now.16:05
zigofungi: How can I check that my post job really published something in tarballs.o.o ?16:05
fungizigo: did the change merge?16:06
zigofungi: Yup.16:06
*** nadya has quit IRC16:06
*** salv-orlando has quit IRC16:06
*** e0ne has quit IRC16:06
*** gyee has joined #openstack-infra16:07
*** jlanoux has quit IRC16:07
fungizigo: so first is to look at the new branch tip16:07
*** daemontool has joined #openstack-infra16:07
fungizigo: says the change merged without needing a merge commit, so we'll take an abbreviation of its git sha16:08
fungizigo: in this case the first 7 characters of the sha are (amusingly) "220dead"16:08
zigoLOL ! :)16:08
fungizigo: if it ended up needing a merge commit, you'd want to use teh sha of the merge commit instead16:08
*** daemontool_ has quit IRC16:09
fungizigo: next go to or parse the json from the zuul status api looking for that abbreviated sha16:09
*** Sukhdev has quit IRC16:09
*** Na3iL has quit IRC16:10
fungizigo: if we're running a backlog in the post pipeline, it may be queued up there so you'd have to make sure to wait until its jobs run16:10
zigoI don't see it in the queue.16:11
*** bpokorny has joined #openstack-infra16:11
zigoWouldn't the post job show in gerrit as well?16:11
*** Sukhdev has joined #openstack-infra16:11
fungiyeah, which is a little odd. i guess it's possible its jobs completed already16:11
fungino, post pipeline results don't get added in gerrit comments but we can go find the logs16:11
*** zz_dimtruck is now known as dimtruck16:12
fungizigo: look under
*** Sukhdev_ has joined #openstack-infra16:12
fungithat's followed by the first two characters of the git sha and then the full git sha16:12
*** sdake has quit IRC16:12
zigoLooks like it uploaded the files ! :)16:12
zigo2016-05-20 16:07:40.246 | [SCP] uploading file: '/srv/static/tarballs/{project}/uploads/220dead31374edabced0d890bfaf2f900da4b66c/openstack-pkg-tools_44+10~bpo8+1.dsc'16:12
zigo2016-05-20 16:07:40.362 | [SCP] uploading file: '/srv/static/tarballs/{project}/uploads/220dead31374edabced0d890bfaf2f900da4b66c/openstack-pkg-tools_44+10~bpo8+1.tar.xz'16:12
zigo2016-05-20 16:07:40.441 | [SCP] uploading file: '/srv/static/tarballs/{project}/uploads/220dead31374edabced0d890bfaf2f900da4b66c/openstack-pkg-tools_44+10~bpo8+1_all.deb'16:12
zigo2016-05-20 16:07:40.520 | [SCP] uploading file: '/srv/static/tarballs/{project}/uploads/220dead31374edabced0d890bfaf2f900da4b66c/openstack-pkg-tools_44+10~bpo8+1_amd64.changes'16:12
fungii think since the debian-jessie nodes are only used for this project so far, you got a node assignment right away and the job completed16:13
fungithat's why it wasn't in the post pipeline backlog16:13
*** javeriak has quit IRC16:13
zigoOh, nice ! :)16:13
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Add grenade jobs to check pipeline for Ironic
zigoI got a VIP priority ! :)16:13
fungiso anyway, next take a look at...16:14
pabelangermust be a typo some place16:14
fungiwas about to point that out16:14
fungiso we have an improper macro expansion happening there16:14
*** Na3iL has joined #openstack-infra16:14
*** sridhar_ram has joined #openstack-infra16:14
fungiwe wanted it to replace {project} with the short project name16:14
zigoAlso, I didn't expect to have "uploads" there, just its content.16:14
zigoI guess that's not really a problem though.16:15
*** Sukhdev has quit IRC16:15
fungiright, the uploads thing is a shortcoming of hierarchy preservation in the scp publisher plugin, there are some options/workarounds to address that later16:15
fungibut for now let's get the macro expansion problem solved16:16
*** Sukhdev_ has quit IRC16:16
zigoSo, what about the {project} macro expansion?16:16
zigoHow to fix?16:16
fungiyeah, i'm looking to see how we address that in the normal tarball publisher16:17
*** dmk0202 has quit IRC16:17
*** jamesmcarthur has quit IRC16:17
*** jamesmcarthur has joined #openstack-infra16:18
*** roxanagh_ has joined #openstack-infra16:18
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Don't try to get stack details if the stack doesn't exist
fungizigo: so one option is to do like the javascript jobs do...
fungibasically pass '{name}' (which expands from the short project name from the jenkins/jobs/projects.yaml)16:19
*** ddieterly is now known as ddieterly[away]16:19
*** claudiub|2 has joined #openstack-infra16:20
zigofungi: So, use {name} instead of {project} ?16:21
*** ddieterly[away] is now known as ddieterly16:23
fungizigo: i don't think {name} will expand to the parent project from within the macro (it might). i was suggesting have your job at pass project: '{name}' as a parameter under the - deb-pkgs: publisher16:23
*** claudiub has quit IRC16:24
*** vsaienko has joined #openstack-infra16:24
fungizigo: though it's also possible that only works from within a job-template rather than a job. we should be able to tell from the results of teh xml comparison job running on the fix you propose16:25
vsaienkohello infra team, please help to merge
*** salv-orlando has joined #openstack-infra16:25
*** salv-orlando has quit IRC16:26
*** salv-orlando has joined #openstack-infra16:26
vsaienkoproject-config core, please help to merge the patch adds job to test ironic cleaning16:26
*** daemontool has quit IRC16:27
*** daemontool_ has joined #openstack-infra16:27
*** shashank_hegde has quit IRC16:28
*** johnny__ has joined #openstack-infra16:29
zarofungi: ok. i figured out how to enforce access rules on the gerrit reindex script, if you want to give it a try on next rename i'll prepare the changes.16:29
*** jamesmcarthur has quit IRC16:29
*** jpr has quit IRC16:29
*** Hal1 has quit IRC16:30
fungizaro: yes, please!16:31
openstackgerritThomas Goirand proposed openstack-infra/project-config: Fix {project} expansion
*** nwkarsten has quit IRC16:33
zigofungi: Is that what you were talkinga bout/16:34
*** lakshmiS has quit IRC16:34
fungizigo: almost, but needs more spaces. see inline comment16:36
*** annegentle has quit IRC16:37
*** tonytan4ever has joined #openstack-infra16:37
*** HeOS has joined #openstack-infra16:37
openstackgerritThomas Goirand proposed openstack-infra/project-config: Fix {project} expansion
fungizigo: yep, that's what i was suggesting16:38
fungizigo: now once it runs the xml comparison job, we can take a look at the actual jenkins job configuration created by this16:39
zigoI'm not sure I understood why the 2 more spaces were needed, but I fully trust! :)16:39
*** jamesmcarthur has joined #openstack-infra16:40
*** cloudtrainme has quit IRC16:40
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add test of post cloner with cached repo
*** camunoz has quit IRC16:41
*** esikachev has quit IRC16:42
*** camunoz has joined #openstack-infra16:42
*** rajinir has quit IRC16:42
*** trown|afk is now known as trown16:43
fungizigo: easier to demonstrate than to explain in this case
zigooh, indeed ! :)16:44
*** jsavak has quit IRC16:44
*** flepied1 has joined #openstack-infra16:44
fungizigo: the flat dict in the first case vs the nested dict in the second16:44
zigoGot it.16:45
*** roxanagh_ has quit IRC16:45
fungithis is one reason why some people despise yaml16:45
*** roxanagh_ has joined #openstack-infra16:45
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Batch calls to list_floating_ips
*** flepied has quit IRC16:45
fungithis particular quirk doesn't bother me, but it does bother a lot of people16:45
*** tonytan4ever has quit IRC16:46
*** ddieterly is now known as ddieterly[away]16:46
zigoWell, one got to know about it, and once one does, that's ok-ish, I guess...16:47
openstackgerritMerged openstack-infra/shade: Be more precise in our detection of provider networks
clarkbzaro: wow gerrit's gerrit is switching to use notedb16:48
*** abregman|afk has quit IRC16:48
*** jamesmcarthur has quit IRC16:50
notmorganclarkb: ?16:50
notmorganclarkb: oh wow16:50
clarkbnotmorgan: getting rid of the sql db and replacing it with content in git16:50
*** cloudtrainme has joined #openstack-infra16:51
notmorganclarkb: interesting choice.16:51
*** asettle has joined #openstack-infra16:51
fungiin git notes, specifically (hence the name, "notedb")16:51
notmorganclarkb: i.. hmm. very interestin16:51
Shrewsi fear anything with "notes" in the name16:51
*** asettle has quit IRC16:51
notmorganShrews: lol ++16:51
*** yamamot__ has quit IRC16:51
*** kien-ha has quit IRC16:51
fungigerrit's gerrit is switching to use lotus notes!16:51
*** tonytan4ever has joined #openstack-infra16:51
vsaienkofungi, I'm sorry for bothering you. Could you please help to merge it is a blocker for Ironic team on grenade stuff16:52
notmorganShrews: lets use notedb for nodepool! (JK don't throw anything at me :P )16:52
jeblairclarkb, jhesketh: i left a couple of suggestions on nakato's post change that i would like you to look over:
fungivsaienko: is the goal there just to shorten the job runtime?16:53
openstackgerritBrent Eagles proposed openstack-infra/tripleo-ci: Exit with non-zero status on create and update errors
fungivsaienko: jroll: any significant risk of introducing an upgrade regression which would have been caught by a tempest test but isn't exposed by grenade's smoke tests?16:54
*** banix has quit IRC16:55
vsaienkofungi, yes it will speed up the job. There is an example
*** lucasagomes is now known as lucas-afk16:55
*** kus has joined #openstack-infra16:55
vsaienkofungi, we launching smoke tests which do actually the same thing (boot/delete instance) in There is no need to launch extra tests from devstack-gate.sh16:56
*** jamesmcarthur has joined #openstack-infra16:56
*** jpr has joined #openstack-infra16:59
*** pblaho has quit IRC16:59
openstackgerritJames E. Blair proposed openstack-infra/puppet-zuul: Install build-essential instead of just gcc
openstackgerritJames E. Blair proposed openstack-infra/puppet-zuul: Remove quotes from zuul.conf launcher sites
openstackgerritJames E. Blair proposed openstack-infra/puppet-zuul: Add /usr/local/bin to zuul-launcher path
openstackgerritJames E. Blair proposed openstack-infra/puppet-zuul: Add launcher logging config file to zuul.conf
*** jsavak has joined #openstack-infra17:00
*** alexey_weyl has joined #openstack-infra17:01
*** eventingmonkey has quit IRC17:01
alexey_weylI have pushed a change to gerrit a few days ago, and I would like please if someone could approve it17:02
*** derekh has quit IRC17:03
clarkbjeblair: I agree with the fetch thing, I expect that that has some potentially nasty and later unfun to debug corner cases17:03
clarkbjeblair: an explicit fetch should be relatively quick if the update has already pulled it17:03
jeblairclarkb: ok, that's reassuring, thanks :)17:04
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Retire openstack-ansible-py_from_git repository
clarkbcommented on the change with that and suggested we go ahead and address it now before someone hsa a bad weekend debugging it in 2 years17:05
*** sambetts is now known as sambetts|afk17:05
openstackgerritVladyslav Drok proposed openstack-infra/project-config: Don't use DEVSTACK_GATE_TEMPEST for Ironic grenade
openstackgerritBrian Stajkowski proposed openstack-infra/project-config: New Project Quark Plugin
*** jamesmcarthur has quit IRC17:08
openstackgerritMerged openstack-infra/shade: Add initial setup for magnum in shade
clarkbvsaienko: why can't just be part of the existing job?17:09
*** jistr has quit IRC17:09
*** alexey_weyl has quit IRC17:09
*** jamesmcarthur has joined #openstack-infra17:10
*** alexey_weyl has joined #openstack-infra17:10
*** jsavak has quit IRC17:10
*** tpsilva has joined #openstack-infra17:11
*** shashank_hegde has joined #openstack-infra17:13
*** jtomasek has quit IRC17:14
*** ccamacho has quit IRC17:15
zaroclarkb: yup, exciting times, i'm wondering what the migration might look like.17:15
*** ccamacho has joined #openstack-infra17:16
clarkbzaro: ya, judging by the thread people are worried about perofmrnace too17:17
clarkbbut neat to see it happening17:17
*** inc0 has joined #openstack-infra17:18
*** tonytan4ever has quit IRC17:18
notmorganjeblair: FYI, about to push some restructuring changes for nodepool up. Adding an abstraction for nodedb (so we can move to nodezk but not need to drop nodedb immediately at the same time), among a few other things.17:18
notmorganjeblair: will let you know when i get to that point.17:19
jeblairnotmorgan: cool, thanks17:19
*** alexey_weyl has quit IRC17:19 proposed openstack-infra/project-config: New project request for SwiftOnHPSS
*** shashank_hegde has quit IRC17:20
pleia2fungi: looks like no one took an action item to announce the upcoming renames 2 weeks from now, shall I in case folks need to get some others queued up?17:20
pleia2I can send a follow-up reminder the week of17:20
openstackgerritKhai Do proposed openstack-infra/project-config: Add a build for Gerrit groovy-provider plugin
*** dtantsur is now known as dtantsur|afk17:22
*** david-lyle_ is now known as david-lyle17:22
pleia2and to everyone: sorry for being away-ish since summit (especially w/ reviews), but happy to say my book is pretty much done now, just some dangling feedback out there, and I've even been able to make some patches to the openstack puppet modules even if I have been less infra-y :)17:22
pleia2so not wasted time away from day to day incidents, I should be able to help infra-cloud more now17:23
jeblairpleia2: welcome back!17:23
pleia2we fixed several bugs in mitaka/stable17:23
pleia2re: puppet17:23
*** inc0_ has joined #openstack-infra17:24
pleia2jeblair: thanks :)17:24
clarkb should save a minute or two per devstack-gate run17:24
*** ddieterly[away] is now known as ddieterly17:24
*** inc0 has quit IRC17:25
openstackgerritKhai Do proposed openstack-infra/project-config: Add a build for Gerrit groovy-provider plugin
mtreinishpleia2: any thoughts on:
zaropleia2: ^ that may help the rename, force online reindex17:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add Ops repo to OpenStack-Ansible
openstackgerritVladyslav Drok proposed openstack-infra/project-config: Don't use DEVSTACK_GATE_TEMPEST_ALL_PLUGINS for Ironic grenade
*** Na3iL has quit IRC17:27
pleia2mtreinish: you don't waste any time :)17:27
*** roxanagh_ has quit IRC17:27
clarkbfungi: jeblair can you double check my bash on ? if that comment looks right to you I will go ahead and push up a new patchset17:27
pleia2zaro: ooh, neat17:28
*** jsavak has joined #openstack-infra17:28
mtreinishpleia2: heh17:28
jeblairclarkb: agreed17:29
*** vhosakot has quit IRC17:29
clarkbpabelanger: the WIP on probably hides it from most reviewers fwiw17:30
clarkbpabelanger: I wil lreview it nowish17:30
pleia2mtreinish: so we're not adding a second instance of running e-r on static.o.o, just the source+config so health can use it?17:30
mtreinishpleia2: yeah it just needs the source and the python interface available on health.o.o so the api server can leverage the queries in the repo17:31
mtreinishpleia2: the patch using it is here:
openstackgerritClark Boylan proposed openstack-infra/devstack-gate: Make reproduce function quiet
mtreinish(I just noticed I put the wrong changeid in the commit msg)17:32
clarkbjeblair: ^ ok new patchset there that should make it quieter17:32
*** roxanagh_ has joined #openstack-infra17:32
*** nwkarsten has joined #openstack-infra17:34
*** nwkarsten has quit IRC17:34
openstackgerritJohn L. Villalovos proposed openstack-infra/irc-meetings: Add a meeting_id to openstack-community-office-hours
*** jamesmcarthur has quit IRC17:34
*** jamesmcarthur has joined #openstack-infra17:35
pleia2mtreinish: does it actually manipulate it at all? wondering if we can use the existing e-r instance on status17:35
*** salv-orlando has quit IRC17:36
*** daemontool has joined #openstack-infra17:37
clarkbpabelanger: mtreinish I think needs an element-deps with package-installs in it17:37
pleia2oh, I guess this would all be on health.o.o, just the dashboard is on status17:37
mtreinishpleia2: it does, the o-h api server is actually querying es. It's just using e-r as a lib (and a source of queries) The other instances that run e-r are either a daemon or a cron job creating a json blob17:37
clarkbpabelanger: mtreinish it may work for us because we already include package-installs into our dep list but if someone else doesn't this won't work17:37
mtreinishpleia2: yeah17:38
clarkbpabelanger: you volunteered to babysit, would you like that change to be updated or have a followup change to include the deps17:38
pleia2mtreinish: thanks, +2A17:38
*** jerryz has joined #openstack-infra17:39
mtreinishpleia2: I thought about publishing e-r on pypi for this, but then there is the data files in python packaging annoyance so I decided not to bother17:39
mtreinishpleia2: cool thanks17:39
*** jamesmcarthur has quit IRC17:39
*** daemontool_ has quit IRC17:39
mtreinishclarkb: hmm, do we expect anyone else to use the stackviz stuff?17:39
pleia2mtreinish: heh, yeah, that's fair17:40
clarkbmtreinish: possibly? the elements should function on their own17:40
*** shashank_hegde has joined #openstack-infra17:40
mtreinishclarkb: sure, that's fair17:41
clarkbarmax: regXboi re I am not sure if that should be abandoned or merged. I think I saw recent mail that the debug command would be removed completely?17:41
*** esikachev has joined #openstack-infra17:41
armaxclarkb: we started a long process of deprecation17:42
regXboiarmax: I assume I can abandon this then?17:42
armaxclarkb: for the neutron-debug command17:42
*** gomarivera has joined #openstack-infra17:42
*** sdague has quit IRC17:43
armaxclarkb, regXboi it will eventually need to merge17:43
armaxonce neutron-debug is replaced for good, but I don’t see the urgency atm17:43
clarkbarmax: regXboi ok if it is a long way out I think we should abandon that change to get it out of the way. When we are ready to remove neutron-debug or replcae it or whatever we can push a new change or restore and update17:44
*** akshai has quit IRC17:44
regXboiand so abandoned17:46
*** e0ne has joined #openstack-infra17:46
*** akshai has joined #openstack-infra17:47
clarkbvsaienko: I will have a comment on shortly as well17:48
fungipleia2: yeah, rename announcement would be good. sorry forgot to action anyone since we were crunched on time at the end of the meeting slot17:49
pleia2fungi: ok, will do :)17:49
*** ddieterly is now known as ddieterly[away]17:51
*** kzaitsev_mb has quit IRC17:51
fungiclarkb: yep, 296171 lgtm17:51
*** yamamoto has joined #openstack-infra17:52
armaxclarkb: thanks17:52
clarkbtherve: are you around to have a quick chat about ?17:54
*** markvoelker has joined #openstack-infra17:54
*** sdague has joined #openstack-infra17:55
openstackgerritMorgan Fainberg proposed openstack-infra/nodepool: Move nodedb to nodestore.nodedb
openstackgerritMorgan Fainberg proposed openstack-infra/nodepool: Move constants to nodestore module
openstackgerritMorgan Fainberg proposed openstack-infra/nodepool: Create pivot point for nodestate storage
*** lakshmiS has joined #openstack-infra17:57
*** mtanino has quit IRC17:57
*** chem`` has quit IRC17:57
openstackgerritMerged openstack-infra/puppet-openstack_health: Add elastic recheck to api server
notmorganjeblair, Shrews: ^ there is the starting place for the nodepool changes. working thorugh some of the ZK specific things and changes around that for better parsing.17:58
notmorganbut that structure will allow us to transition from sql->zk more easily17:58
*** markvoelker has quit IRC17:59
*** vhosakot has joined #openstack-infra17:59
*** yamamoto has quit IRC18:00
*** jpr has quit IRC18:01
*** cody-somerville has quit IRC18:01
*** nwkarsten has joined #openstack-infra18:01
*** jcoufal has quit IRC18:01
openstackgerritMário Santos proposed openstack-infra/shade: Add support for changing metadata of compute instances
clarkbnotmorgan: jeblair I am confused, are we completely replacing sql with zk or just the bits that require shared state (eg image builds)?18:02
notmorganclarkb: in nodepool the structure of everything could easily lend itself to being 100% zk18:03
*** sridhar_ram has quit IRC18:03
clarkbright thats the question, what is your intention18:03
notmorganclarkb: my intention is 100% zk18:03
clarkbbecause aiui we were keeping sql but skimming your changes you are replacing sql18:03
notmorganclarkb: unless there is a good reason for keeping SQL with this type of structure.18:05
clarkbnotmorgan: I am not sure if there is, but iirc the spec was limited to image building18:05
notmorganclarkb: i could do it as a split, but it felt... awkward?18:05
clarkbso just making sure I understand18:05
notmorganclarkb: i spent a good amount of time looking at the structure of the data before writing this though.18:05
notmorganspec might need an update too.18:05
openstackgerritJohn L. Villalovos proposed openstack-infra/irc-meetings: Setup a tox pep8 check and update test-requirements.txt
clarkband yes generally the data is resrouce tracking, it isn't rickly relational where SQL would really shine18:06
notmorganclarkb: exactly.18:06
notmorganclarkb: which is why the split between sql and zk felt so awkward18:07
*** nadya has joined #openstack-infra18:07
*** chem`` has joined #openstack-infra18:07
*** jcoufal has joined #openstack-infra18:07
notmorganit is almost purely hierarchical as well. <image>/<node based on image>|snapshot|etc18:07
*** oanson has joined #openstack-infra18:07
notmorganwith some levels for proviers too.18:07
notmorganor it could be attrs on the node.18:08
clarkbnotmorgan: I think the one place where this could be a potential problem is if/when we start treating other resources like floating IPs and volumes and networks and routers etc as things nodepool should know about18:08
notmorganbut it is def only somewhat relational18:08
clarkbbecause then you would be tracking relations between things18:08
clarkbwe have long avoided that because the theory is openstack provides that management for us but reality is it often does not work18:08
notmorganclarkb: hmm.18:09
*** piet has joined #openstack-infra18:09
clarkbnotmorgan: one concrete example of something that would possibly benefit from sql is if we managed a pool of preallocated fips directly18:10
notmorganclarkb: i'll have to think on that a bit - it may justify a split.18:10
openstackgerritKhai Do proposed openstack-infra/system-config: Install the Gerrit delete-project plugin
clarkbnotmorgan: rather than boot instance, create and attach fip, delete fip, delete instance18:10
notmorganclarkb: it also may be a narrow enough dataset that relationality and normalized data isn't a huge win18:11
clarkbnotmorgan: have full quota of fips preallocated (or lazily allocate them), then assign and unassign as we boot and delete nodes18:11
*** ifarkas has quit IRC18:11
notmorganclarkb: it depends, if w'ere not using FKs and updating the rows directly - it's about the same overhead as locking a node and setting an attr18:11
clarkbnotmorgan: yes that is possible too, we are talking a relatively small number of resources unless computing fundamentally change stomorrow18:11
notmorganclarkb: if we lean on RDBMS technology more, sure.18:11
*** nadya has quit IRC18:11
openstackgerritJohn L. Villalovos proposed openstack-infra/irc-meetings: Add helper to find full slots and pep8 check
notmorganclarkb: i almost think a /<provider>/fips/18:12
notmorganmakes sense in this case18:12
notmorgani'll mull on it a bit18:12
clarkbnotmorgan: you have to track what instance owns what ip at any given time18:12
clarkbnotmorgan: beucase neutron and nova can't actually do that reliably today18:12
notmorganclarkb: right. but that may in fact be ok. it depends on how we want to do that with sql18:12
notmorganit's a difference is how we do that.18:13
*** sarob has joined #openstack-infra18:13
notmorganif the number is low, non-normal form might be as elegant/easier.18:13
notmorgani'll ponder some before driving to much further here.18:13
*** nadya has joined #openstack-infra18:13
*** mtanino has joined #openstack-infra18:14
clarkbnotmorgan: we have also discussed possibly using volume backed instances but apparently cinder and nova do a good job of tracking that cooperation18:14
clarkbthat may be a another similar case but probably not due to cloud holding our hand properly18:14
openstackgerritamrith proposed openstack-infra/project-config: Adding new repository trove-image-builder
*** mikelk has joined #openstack-infra18:15
clarkbnotmorgan: and probably the most complicated set of relations would be if we went nodepool managed networks, subnets, routers, dhcp servers (and maybe security groups)18:16
*** esikachev has quit IRC18:17
notmorganclarkb: i think it comes down to the question of how likely that is18:17
clarkbnotmorgan: it is something we struggled with when hpcloud was a thing18:18
notmorganclarkb: if we're going to track FIPs and maybe volumes -- i'm thinking non-normalized is fine18:18
clarkbnotmorgan: we used a workaround to scale up by having a different logical provider per network then manually preprovisioned everything18:18
notmorganif we really thing we should do networks etc and highly complex stuff18:18
notmorgani'm inclined to say we should keep some level of relationship18:18
clarkbso I am not sure we know how we would like to best solve that18:18
clarkbI think we do know what we had was subpar18:18
*** nadya has quit IRC18:18
notmorgans/relationship/relational engine/18:19
openstackgerritamrith proposed openstack-infra/project-config: Adding new repository trove-image-builder
notmorganbut we def don't want to just do "oh we'll update these rows with references" because thats no different than what we're doing now.18:19
notmorganerm, s/now/without relational db18:19
fungieven then, i doubt scaling that (non-normal form lookups and updating secondary indices) in zk would result in anywhere near the performance issues with scaling call volume on the openstack api side18:20
notmorganclarkb: it comes down to what the ops look like.18:20
notmorganfungi: ++18:20
notmorganand the desired direction18:20
fungiso while it might be inefficient, i have a feeling it would be imperceptably less efficient held up against the other stuff nodepool has to wait on18:20
clarkbnotmorgan: my initial impression is that the ops will be scale up/down related18:20
clarkbkeep an accurate count of instances per network/subnet/router, then make and delete as necessary to maintain a ratio that achieves desired perfomrance18:21
notmorganfungi: and i do like the hierarchical nature of the resource tracking.18:21
clarkbcan be done without sql for sure18:21
notmorganfungi: vs. the SQL model.18:21
notmorgani also like that the data is mostly discoverable by just saying "get_children"18:22
notmorganrather than needing to know the schema18:22
clarkbbut also keep basic attributes for each, know what external net is attache dto each router (required for fips), know what subnets belong to which networks in order to make sure we don't make overlapping subnets and so on18:22
notmorgannot that i would want to do lots of get_children for genreral use-casee18:23
*** eventingmonkey has joined #openstack-infra18:23
*** rossella_s has quit IRC18:23
notmorganclarkb: so again. is this something we're really looking to do? thats going to be what dictates my choices here more than anything18:24
notmorganclarkb: and i don't really have a horse in this race.18:24
*** rossella_s has joined #openstack-infra18:24
clarkbnotmorgan: it is something we need to solve if we can't get nova/neutron to reliably keep track of fips and if we want to support clouds like the old hpcloud18:25
clarkbnotmorgan: the hack we had for hpcloud works, it is just less than ideal. So we can limp along with that and worry about it if/when another cloud like that appears18:25
jeblairso, the approved spec is for zookeeper for image builds18:26
clarkbjeblair: yes that is what brought this up :)18:26
notmorganand if i'm also pushing an update to the spec.18:26
jeblairthere is a not-yet-approved spec to extend zk to everything in nodepool:
notmorgani can narrow the scope of the pivot point easily18:26
fungiright, all of the performance discussion aside, that spec started small in scope for good reason18:26
jeblairit is the logical next step for this, and we should take it if the image build part works out18:26
notmorganfungi: and i'm also looking at this needing to be a fairly large change in either case.18:27
notmorganfungi: unfortunately18:27
*** julim has quit IRC18:27
notmorgana lot of the work is internal code shuffling anyway.18:27
jeblair(honestly, at this point, i think that we *need* that for zuulv3, so i really hope it works out)18:27
notmorganbut narrowing down the pivot for building only is doable18:27
notmorganbut we need the pivot so we can work on this without doing a big bang change imo18:28
* notmorgan doesn't like the big-bang changes.18:28
clarkbnotmorgan: no big bang chang ewould be required I odn't think18:28
notmorgansupporting both for a while will be required.18:28
jeblairi personally don't think that abstracting this is the right approach18:28
clarkbnotmorgan: the image builds stuff is relatively limited in scope compared ot the rest of the state in nodepool18:28
jeblairbecause as was mentioned earlier, they are different ways of thinking about the problem18:29
*** jcoufal has quit IRC18:29
jeblairso i think we should start by thinking about the problem of image builds in the zookeeper way18:29
jeblair(and then later i think we should think about the problem of node builds in the zookeeper way)18:29
openstackgerritNathan Reller proposed openstack-infra/project-config: Add KMIP Experiment Check to Barbican
jeblairbut having said that, perhaps the best way to implement it is an abstraction18:29
notmorganjeblair: i'll start again with a different scope in mind.18:30
jeblairso we should look at that and review it18:30
*** dtardivel has quit IRC18:30
*** vhosakot has quit IRC18:30
*** dimtruck is now known as zz_dimtruck18:30
jeblairnotmorgan: i haven't looked at it at all, so i wouldn't ask you to change anything at this point :)18:30
notmorganjeblair: fwiw, the split between building and "state" is pretty awkward.18:30
*** Kaiyan has joined #openstack-infra18:30
notmorganjeblair: nah, it's fine this convo w/ clarkb has highlighted we can't do what i was proposing18:30
notmorganjeblair: for now.18:31
notmorganjeblair: so let me restart this with a different scope in mind.18:31
jeblairnotmorgan: if you find something that does't jive with the spec, can you update the spec?18:32
jeblairi'd really like to work from that if possible18:32
notmorganjeblair: yeah. it's not anything with the spec itself.18:32
notmorganjeblair: just was looking at the whole data model. the way nodepool is built it's pretty tightly integrated atm.18:32
jeblair(i mean, it's intentionally not detailed as to how to accomplish the thing, but it should describe the end-state well enough that we shouldn't need to question the fundamentals after the fact)18:32
clarkbnotmorgan: ya, I think the current data model is lacking when it comes to specific problems we have had though18:33
notmorganjeblair: so i was trying to avoid pull apart the internal bits and split that up.18:33
notmorganjeblair: to start, but look slike that is the first step.18:33
clarkbnotmorgan: in particular floating IP (re)use and network management for clouds that require us to use >1 network18:33
jeblairclarkb: that's definitely out of scope for the zk builders18:33
notmorganjeblair: i'll start reworking the datamodel so we can have the logical split we need. its just a change in the order of operations18:33
jeblairclarkb: but even so, i think it's out of scope in the future too.  that is a shade problem18:34
clarkbjeblair: right but would likely factor into the replace sql18:34
*** bpokorny has quit IRC18:34
*** zz_dimtruck is now known as dimtruck18:34
clarkbjeblair: shade doesn't have persistent state and both of these problems require persistent state to be managed18:34
notmorganjeblair: instead of rework to make ZK a possibility, it'll be rework data model, then rework to make zk hook where we need.18:34
jeblairclarkb: can you look over ?18:34
clarkbjeblair: I was kind of hoping we could make progress on the first zk stuff before trying to throw darts at that18:35
jeblairnotmorgan: it is definitely okay for us to have a flag-day transition.  we don't need to support both things.  all the data is ephemeral :)18:35
clarkbbut I can review it sooner18:35
openstackgerritDavid Lyle proposed openstack-infra/devstack-gate: Test patch
jeblairclarkb: i'm not asking anyone to implement that now.  but i think it might help if we had an idea of where we wanted to go next.18:35
*** davidgiluk has joined #openstack-infra18:35
*** jsavak has quit IRC18:36
jeblairclarkb: i will say that in that vein, i'm not ready to say that we need to track floating ips, but, if we feel that tracking fips is something we might need to do in the future, it is worth making sure we can slot that into the proposed data model so that we don't back ourselves into a corner.18:36
*** davidgiluk has left #openstack-infra18:36
clarkbjeblair: to clarify we would only need to track fips if we want to reuse them18:36
notmorganjeblair: no no, the pivot was to make this an easy "now use X" but not break sql until we change the config18:36
clarkbjeblair: rather than doing a full create, attach, detach, delete cycle for each fip18:37
*** jsavak has joined #openstack-infra18:37
notmorganand if we needed to roll back we could then. thats all18:37
clarkbjeblair: if we tracked them then we could just attach, detach18:37
*** jpr has joined #openstack-infra18:37
fungihrm, interesting correlation with the apache segfaults on our mirror servers. i consistently see them (roughly proportional to node quota/utilization) on all three mirrors in rax... and _nowhere_ else18:37
fungileading me to wonder if this is something to do with xen domu timekeeping18:38
fungiseems to come and go in bursts maybe? hard to tell as the frequency even on the ord mirror is still pretty sparse18:39
jeblairclarkb: yeah, i think the whole port creation thing is supposed to make a lot of that better. i'd like for us to chase that to it's logical conclusion before we change course, but that is a conversation best left for later.  i _do_ agree that we should allow ourselves the flexibility to do that if we need to, and i _think_ that if you look at the way i have described the zk stuff (for both builders and nodes), we should be able to ...18:39
jeblair... handle it (after all, zookeeper is good at describing collections of things).18:39
*** _sarob has joined #openstack-infra18:39
*** jed56 has quit IRC18:39
*** _prad_ has quit IRC18:39
fungibut there are segfaults happening within a few seconds or minutes of each other and then hours pass before another incident18:40
clarkbfungi: is it worth trying to get a core dump?18:40
jeblairfungi: maybe they happen under memory/process/thread pressure?18:40
*** pradk has quit IRC18:41
clarkbjeblair: on the network management side I think that is something we do want and I don't know that it can go in shade. Basically point nodepool at cloud and once it has 100 instances (or whatever that scale factor is) on a router make a new router/network/subnet plumb it all and start distributing instances there18:41
fungijeblair: maybe. though there's got to be something specific to rackspace's environment or maybe their images, since memory/process/thread pressure on dfw is way lower than, say, ovh18:41
clarkbjeblair: because if we do that then we can avoid the nastyness we had in the hpcloud configs which was 5 logical providers one per router/subnet/network18:41
jeblairnotmorgan, clarkb: i think we need to keep in mind that we are replacing both the data store (sql) and the event system (gearman) in the image builders at the same time.  that's why my intuition tells me that a slightly different way of thinking of things will be required, but like i said, i haven't written the code so i'm open.  :)18:42
*** sarob has quit IRC18:42
mtreinishfungi, jeblair, clarkb: how long should it take for the pypi mirrors to sync?18:42
*** abregman has joined #openstack-infra18:43
clarkbmtreinish: its on a 5 minute loop18:43
mtreinishthe openstack-health python jobs aren't seeing the latest subunit2sql release, it's been a few hours18:43
jeblairclarkb: yeah, i agree there.  i think that is something we can support with zk, so i don't think we're backing ourselves into a corner.  but i also would prefer not to solve it now since it isn't pressing.18:43
*** vhosakot has joined #openstack-infra18:43
clarkbjeblair: agreed on not solving it now, we currently have no clouds that require thinking about that again18:44
jeblairclarkb: aren't we lucky? :)18:44
notmorganclarkb: yay? ;)18:44
notmorganok. thats fine. i'll work on splitting up the data model to start.18:44
notmorgani just need to spend more time thinking about that split than i have (I was going to do that after moving to ZK, previously)18:45
clarkbmtreinish: I know the wheel mirror updates are/were broken due to missing pip318:45
* notmorgan goes to stare at the db schema more.18:45
clarkbmtreinish: but that is separate from pypi mirro18:45
clarkbfungi: ^ did the pip3 thing get addressed?18:45
fungimtreinish: i don't see bandersnatch logging any update for subunit2sql18:45
fungiclarkb: i've +2'd haypo's proposed fix18:45
clarkbooh tehre is a patch to review /me does this18:46
funginot sure if it merged yet?18:46
*** sdake has joined #openstack-infra18:46
fungiwe got into the weeds a little over "python2" vs "python" (probably just me being overly pedantic)18:46
clarkbdoesn't look like it, just double checking if python3 is installed then will approve18:46
fungiyeah, i see a python3 executable on mirror-update.o.o18:47
fungichecked that already, should have mentioned it when reviewing18:47
*** ihrachys has quit IRC18:47
mtreinishfungi: hmm, well it's definitely on pypi18:47
jeblairmtreinish, fungi, dstufft: i think bandersnatch is stuck18:47
clarkbfungi: wheels are built ona different host18:47
jeblairmtreinish, fungi, dstufft: I think18:48
* clarkb figures out the something or other18:48
fungijeblair: mtreinish: yeah, i see ERROR lines18:48
fungiabout stale serials18:48
*** tonytan4ever has joined #openstack-infra18:48
*** yamahata has quit IRC18:48
jeblairi think that is bandersnatch stuck trying to sync that package, and it can't move past it18:48
jeblair(somehow, that warrants a successful exit code)18:48
funginot stuck as in hung at least, just keeps trying to resume from there and doesn't ever get it18:49
*** piet has quit IRC18:49
fungilooks like it began at 01:15 utc today18:50
*** dmk0202 has joined #openstack-infra18:50
clarkbconfirmed that python3 is on and wheel-mirror-centos-7-amd64.slave.openstack.org18:50
fungithough prior to that we were getting "ERROR: Stale serial for package awscli" around 21:05 utc yesterday18:51
*** ddieterly[away] has quit IRC18:51
*** rajinir has joined #openstack-infra18:51
clarkbjeblair: fungi: we haven't updated to the new version of bandersnatch have we? I want to say it requires a full resync18:51
jeblairclarkb: i have not done anything on that front18:51
fungibandersnatch (1.9.dev0)18:52
fungiso no, we haven't yet18:52
clarkbis it possible the mirror protocol has changed on the pypi side too requiring an update?18:52
fungii think we were needing to apply the rebased hash pathing patch to 1.10, but then that got included in 1.1118:52
jeblairclarkb: i don't know the answer to that, but i know i have seen this before and have been unable to fix without pypi admin help.18:53
*** ams__ has joined #openstack-infra18:54
fungicould i guess be what this meant in the release 1.11 notes: "Fix an edge case: IO errors while marking off packages as “done” could result in crashing workers that would result in bandersnatch getting stuck."18:54
*** tonytan_brb has joined #openstack-infra18:56
openstackgerritMerged openstack-infra/devstack-gate: Start using CINDER_VOLUME_CLEAR to replace CINDER_SECURE_DELETE
jeblairfungi: this is why we put the curl command in our bandersnatch script:
jeblairi tried that manually though and it did not fix the problem18:57
fungijeblair: dstufft: looks like this traceback preceded it:
zigofungi: Looks like it didn't work, right?
dstufftit's trying to get /simple/FlexGet/18:58
dstufftinstead of /simple/flexget/18:58
zigofungi: It's doing {name} instead of {project} without expansion, right?18:58
dstufftthe first is a redirect18:58
dstufft(and redirects don't have X-PyPI-Serial_18:58
dstufftthat hasn't changed recently though...18:59
jeblairdstufft: would the recent changes to bandersnatch avoid this?  (we aren't running that yet)18:59
fungizigo: yep, that's what i thought might happen. i'll propose another patchset to that change with what i think is the solution, because that'll be easier than trying to describe it18:59
dstufftjeblair: yes, but only because they don't require /simple/ at all :]18:59
dstufftthey only hit the JSON endpoint18:59
dstufftand the files19:00
fungijeblair: dstufft: right, we're still on 2.9 because we haven't gotten around to upgrading and firing off a full resync yet19:00
zigofungi: I don't even think we really need the project name, since all of that are just temporary folders for the repository pickup job.19:00
fungiwe ended up putting apache reqrite workarounds on our mirrors in the meantime to deal with the . to - conversion19:00
*** tonytan4ever has quit IRC19:00
zigofungi: We could as well just do deb-pkg/uploads/<SHA256>/19:00
zigofungi: Your thoughts?19:00
dstufftfungi: I guess 1.9 has jsut been broken for awhile and nobody noticed for... well I d'nt know why19:01
zigoIIRC, it's just monty that suggested to use {project}19:01
dstufftI don't see how this ever worked for projects like FlexGet19:01
fungizigo: i guess that's an option. will this job end up running on additional repos, or only ever for that one repo? if the latter, we could probably even simplify further by moving that publisher macro directly into the job definition19:01
haypooops, gate-project-config-layout just failed on my approved change "fatal: dumb http transport does not support --depth" on "Cloning into 'zuul'..."19:02
haypowhat is this error?19:02
*** ddieterly has joined #openstack-infra19:02
zigofungi: deb-openstack-pkg-tools is a special repo, because it builds itself, but other repos will only be different because they don't hold the scripts themselves and will just apt-get install openstack-pkg-tools.19:02
zigofungi: So, I'm tempted to reply to you by "hum... half half..." :P19:03
fungizigo: so their builds will use a separate job from this one?19:03
zigoBut one that will be very simillar.19:03
jeblairfungi, dstufft: i did cdn purges on flexget, and it worked.  (i don't know if that's why, or if dstufft did something else behind the scenes)19:03
zigoThe difference is apt-get install of the build scripts instead of a cp + chmod, but that's it.19:03
jeblairfungi, dstufft: if that did fix it, then i guess we should update our auto-purge thingy to also purge the normalized versions of those urls19:04
zarofungi: thought i would ask you about the online reindex thing before doing more work since there's a few ways to make that work.19:04
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: New project: js-openstack-lib
fungihaypo: that looks like it may be a misleading error message following a silent network failure and fallback19:04
zarofungi: basically there are 3 options to be able to use that functionality.19:05
jeblairmtreinish, fungi, dstufft: and if i grok bandersnatch correctly, we are now synced with where we should have been 18 hours ago, and the next run, in 10 minutes, should actually sync us up to the present.19:05
haypofungi: ah maybe. should i just recheck?19:05
jeblairmtreinish, fungi, dstufft: you can check for that19:05
jeblairdstufft: is there a handy function to normalize a name?19:06
zarofungi: it's available now as the groovy script but in order for that to work we would need to install the groovy-provider plugin + the groovy indexing script.  i wasn't sure whether you would be comfortable with that.19:06
fungihaypo: i think so. if we keep seeing it, then maybe we've changed something with zuul-cloner or run-layout.sh19:06
haypofungi: ok19:06
mtreinishjeblair: ok, cool19:06
fungihaypo: unfortunately there's not much detail to go on there (the things it's cloning from should support "smart" hht transport/backend)19:07
zarofungi: the groovy-provider plugin is  the thing that enables groovy in gerrit.19:08
*** gordc has quit IRC19:08
fungijeblair: i _think_ the relevant pep includes an example python function. checking19:08
zarofungi: an alternative is to change gerrit core to allow force reindex19:09
fungijeblair: indeed...
dstufftjeblair: PEP 503 has a copy/paste function, or ``pip install packaging; python -c 'import packaging.utils; print(packaging.utils.cannonclize_name("FlexGet"))'``19:09
dstufftI think that's the name19:10
dstufftsomething like that19:10
dstufftprobably spelled better than I can type off hand19:10
zarofungi: I think 2nd option is preferrable but may take a little more time.  I'm pretty sure we can backport if it gets accepted on upstream.19:10
jeblairdstufft, fungi: thx, i'll add that in after lunch19:11
fungizaro: can you elaborate on "change gerrit core to allow force reindex" more?19:11
fungizaro: is that the review you pushed?19:11
fungizaro: nevermind, i see you pushed a review for the groovy script19:12
*** shashank_hegde has quit IRC19:12
fungior rather, for the groovy-provider plugin as a prerequisite19:12
zarofungi: i mean propose change to gerrit core to add a force flag to gerrit index command.19:13
fungizaro: oh, got it. so online reindexing needs some forcing, and currently the command line tool to initiate a reindex lacks support in gerrit to force it to happen, so it can either be extended directly in an upcoming patch to gerrit _or_ worked around by using a groovy script through a plugin19:14
zaroyes, it's a pre-req for the groovy script.  so we would need to add both the groovy-provider + reindex script which would mean update our puppet scripts for that19:14
zarofungi: yup.19:14
fungizaro: so is the current lack of workingness for the command-line reindex because it/gerrit has a bug?19:15
*** ericksonsantos has quit IRC19:15
fungii.e., does it need to be fixed in gerrit eventually anyway, and using groovy to do it through a plugin is just working around the bug for now?19:15
zarofungi: no, not a bug.  it just needs the option to force reindex19:15
fungizaro: so how is it supposed to work currently without having an option to force reindex?19:16
zaroso we think it's a bug, but i'm not not sure if gerrit guys think the same.19:16
fungii'm mostly just curious what the actual purpose of that reindex command line is, if it doesn't have the ability to do online reindexing. is it only intended for offline use?19:17
zaroi can propose a change to find out.19:17
*** pvaneck has joined #openstack-infra19:17
*** markusry has quit IRC19:18
zaropurpose is for upgrading gerrit.  when you upgrade you have the option to offline or online index19:18
fungiand how is the online index supposed to work in that context?19:18
*** pradk has joined #openstack-infra19:19
*** nadya has joined #openstack-infra19:19
zarorun gerrit init to upgrade gerrit site which upgrades the lucene index version.19:19
fungitrying to figure out if this is an intended feature which doesn't work at all (which by any definition is a bug, i would think?) or whether we're using it in unexpected ways19:20
*** julim has joined #openstack-infra19:20
zaronow you can offline reindex, or online reindex.  since the verion changed the online reindex will start19:20
fungididn't we try to online reindex when we upgraded gerrit last time?19:20
zarono, i don't believe so19:20
fungior am i misremembering?19:20
zaroi think you only tried it for renaming projects last time19:20
*** ddieterly is now known as ddieterly[away]19:20
fungiso the difference is that we want to use it to reindex _outside_ the context of a gerrit upgrade, and it's only meant to work _within_ the context of a gerrit upgrade19:21
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: New project: js-openstack-lib
therveclarkb, I am now!19:21
zarofungi: yeah, basically correct19:21
fungizaro: thanks, much clearer now. i can see why they might not consider it a bug in that case. off-label use of a tool intended for a slightly different purpose19:22
zarofungi: there is a 3rd option if it doesn't get accepted in gerrit core19:22
zarofungi: we can add it to this plugin
*** ddieterly[away] is now known as ddieterly19:23
fungiand now i get what you mean by "force" (normally it would automatically reindex on upgrade, but we want to force it to reindex even though it's not an upgrade)19:23
zaro*it/force reindex command19:23
clarkbtherve: I just sat down ti lunch is 10 minutes from now good?19:23
*** nadya has quit IRC19:24
therveclarkb, Yep!19:24
zarothis would allow us to just install one plugin and not have to worry about groovy or scripts.  unless groovy is something we want/need19:24
zarowhich i don't think we really want.19:24
fungizaro: is the admin-console plugin usable through rest api methods too?19:24
funginot that it needs to be for our case, just curious what the options would be to leverage it19:25
zaroyeah, i think these are all ssh commands19:25
zarothey seem to extend the SshCommand class anyway19:25
*** yamahata has joined #openstack-infra19:26
fungizaro: that seems like a reasonable alternative19:26
fungizaro: unless we start running into multiple situations where we need to work around missing/incomplete features like this, i don't see there being a lot of motivation to write groovy scripts and so probably not a strong use case for adding the groovy plugin19:27
fungizaro: if we didn't already have performance issues with gerrit i might not be as concerned, but every plugin is more surface area for added inefficiencies/leaks/bugs19:28
zarofungi: so just to recap.  right now only the groovy script is available.  i will attempt to add force reindex feature to core gerrit.  if that is rejected then i'll add it to the admin-console plugin.19:28
*** ayoung has quit IRC19:29
zarofungi: so maybe we should just skip the groovy option?19:29
zarowhich means online reindex will probably not be an option for next rename.19:30
fungizaro: that's where i'm leaning at least. we still have some time before we'd be able to use it either way, and we already know to just plan for lengthy outages regardless since even if we have the solution because there's no guarantee it would work and then we have to be able to do an offline reindex anyway19:30
fungiother infra-root admins who also work on project renames and gerrit administration in general might feel differently though. i'm only one voice19:31
*** mikelk has quit IRC19:34
clarkbtherve: ok19:34
zarofungi: ok, unless you guys screem for groovy option i won't be doing any work to make the groovy script option available.19:36
*** oanson has quit IRC19:36
clarkbtherve: I am struggling with a couple things, first how does thsi remove duplication? and we appear to lose tee's stdout with this method so it isn't preserving the old behavior of writing to stdout and a file19:36
clarkbtherve: specifically if we had double timestamps before that meant that tsfilter was getting already timestamped lines right?19:38
therveclarkb, That's correct19:38
clarkbtherve: I think the proposed code would apply a second timestamp to those lines just like the existing code19:38
*** cody-somerville has joined #openstack-infra19:38
clarkbsince it is still running it through tsfilter19:38
therveI have to say I did that a long time ago19:38
therveLet me test it again19:38
*** bpokorny has joined #openstack-infra19:39
therveclarkb, OK so from my test, what it does is send the timestamp to the file, but not to stdout19:40
thervestdout is then captured by the wrapper script, which calls tsfilter on it and apply it19:41
therveAt least that's my understanding19:41
*** tonytan4ever has joined #openstack-infra19:41
clarkbtherve: I actually think we are inverted here. The tsfilter that we are modifying there is the outermost tsfilter, so all other timestamps are applied before it runs19:41
clarkbtherve: this has two problmes I think, the firstis that the log file will continue to be double timestamped and the second is if there is no inner timestamp attachment then we get no timestamps. This tsfilter was added because of the second issue19:42
therveclarkb, I thought hooks were running inside something else?19:42
*** tonytan_brb has quit IRC19:42
*** nwkarsten has quit IRC19:42
clarkbtherve: well the do run inside the jenkins job which does console logs stuff including timestamps19:43
clarkbso maybe that is the real issue19:43
clarkbwe want the console log and file on disk to match up19:43
clarkbbut they separate attach logs19:43
therveAFAIU tsfilter is called in 2 places19:43
*** reed_ has joined #openstack-infra19:43
thervein run_command19:43
therveAnd in call_hook_if_defined19:44
clarkbrun_command isn't in the hook stack I odn't think19:44
clarkbya its just setup and cleanup for run_command19:45
therveHum okay19:45
therveSo the outside timestamp come from something else?19:46
clarkbtherve: I think in the case of d-g hooks one of two things is happening either a) the hook itself is calling tsfilter in the hook and attaching timestamps there or b) we are bubbling those logs all the way up to the jenkins console log and jenkins is attaching the outer timestamps19:46
therveIf I look at
therveYou're right that we have some tsfilter calls outside of hooks that don't add the double timestamps19:48
therveeg tsfilter setup_host19:48
clarkbya there is a single timestamp in which means that jenkins is adding the outer timestamp19:49
therveSo is the tee the issue? I don't see how19:49
clarkbtherve: probably the thing to do in this case is to just stop letting stdout get out19:49
clarkbtherve: ya I think its bceause we tee and let stdout get to jenkins19:49
*** ericksonsantos has joined #openstack-infra19:50
* therve nods19:50
clarkbif people are ok with not using the console log file and instead using the specific hook log files we can just dev null that or stop using tee entirely and just redirect straight to file19:50
openstackgerritAndreas Florath proposed openstack/diskimage-builder: [WIP] Add: lvm element
therveclarkb, But doesn't removing the timestamp from stdout work then?19:51
clarkbtherve: yes but that has the issue of writing the same bits twice and if I get to be selfish thats bad because it uses more disk :)19:51
clarkbtherve: I think we should choose where we want to write them, either to the separate lo gfile or to the console log and then fix it to work in that single case19:51
clarkbtherve: that should simplify it quite a bit19:52
clarkbto write to the console log we would remove the tsfilter there and stop writing to the special file19:52
therveclarkb, I think I'd prefer using the console log indeed19:53
clarkbeither way works for me, my biggest goal is write bits once and make that single place work correctly19:53
therveAt least in our case the test run in the hook, so having to go to the hook file is a bit worse19:53
therveexcept if people are actually relying on the hook output file?19:55
therveI don't personally19:55
*** mhickey has quit IRC19:55
clarkbI am not sure that anyone is. We don't index them in elasticsearch even19:55
clarkbIt should be relatively safe given that ^ isn't using it19:55
therveFair enough19:56
therveI'll update my patch to see if I understood correctly19:56
clarkbgreat, thank you for talking through that. I definitely understand the situation much better now19:57
*** asettle has joined #openstack-infra19:57
*** jsavak has quit IRC19:58
openstackgerritThomas Herve proposed openstack-infra/devstack-gate: Remove double timestamp from console logs
*** jsavak has joined #openstack-infra19:59
*** asettle has quit IRC19:59
*** asettle has joined #openstack-infra19:59
*** asettle has quit IRC19:59
clarkbtherve: yup that is about what I expected. If none of the d-g check jobs use a pre or post test hook we can use depends-on that patch in a project that does to double check it too20:00
openstackgerritMerged openstack-infra/project-config: Fix for Python 3
*** fawadkhaliq has quit IRC20:01
therveclarkb, Ah depends-on will work on devstack-gate? I was wondering20:03
therveLet me do a quick heat patch20:03
*** salv-orlando has joined #openstack-infra20:03
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci: Fix workaround to not install packages during dlrn build
*** Apoorva has quit IRC20:04
*** ihrachys has joined #openstack-infra20:04
therveclarkb, gate-swift-dsvm-functional-nv seems to use a hook so we're good20:04
*** jsavak has quit IRC20:04
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Add error logging around FIP delete
*** salv-orlando has quit IRC20:08
Shrewsjeblair: clarkb: mordred: ^^^20:08
*** e0ne has quit IRC20:08
Shrewswould be nice to see nodepool running with that20:08
*** _ari_ has quit IRC20:09
*** IlyaG has joined #openstack-infra20:10
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Hard-code deb-openstack-pkg-tools into upload job
fungizigo: ^20:10
fungithat simplifies things a bit for now20:11
fungimordred: ^ (though i know you're not _really_ around)20:11
*** banix has joined #openstack-infra20:11
fungiwhile i was at it, i corrected a separate bug with the archive publisher not using the same path for the artifacts as the scp publisher20:12
fungiif reviewers feel strongly, we can update the commit message to mention that20:12
*** e0ne has joined #openstack-infra20:12
fungibut given the job isn't working completely yet anyway, it seems pedantic to enumerate all the ways it was broken20:13
*** shashank_hegde has joined #openstack-infra20:13
*** dimtruck is now known as zz_dimtruck20:14
fungihaypo: 319236 merged, so i suspect that really was a transient network failure and the error message was coming from a fallback behavior, masking the actual network fault20:18
openstackgerritMerged openstack-infra/project-config: Don't use DEVSTACK_GATE_TEMPEST_ALL_PLUGINS for Ironic grenade
KaiyanHi this is Kaiyan from Monasca team, I'm trying to add monasca tempest tests as non-voting check job for all the monasca repos. But the new patch set failed "gate-project-config-layout". Can any one help me with it please? Thank you!
KaiyanIt failed locally too when I ran to. compare-xml-new: commands failed20:18
*** sdake has quit IRC20:22
*** sdake has joined #openstack-infra20:22
fungiKaiyan: i left review comments explaining how to fix that20:22
*** esikachev has joined #openstack-infra20:23
haypofungi: ok. i will check the job next week20:23
*** nadya has joined #openstack-infra20:24
jeblairShrews: left a comment and a question on 31943520:25
*** sdake has quit IRC20:25
*** ayoung has joined #openstack-infra20:26
*** fawadkhaliq has joined #openstack-infra20:27
Shrewsjeblair: hrm, i think that's actually a copypasta, but weird we treat the two cases differently right now. i'll fix to keep current behavior20:27
jeblairShrews: okay.  i think that's safe considering the stated goal of just logging.  i haven't really dug in to see if the behavior change would matter though.20:28
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Add error logging around FIP delete
kgiustiHi all - Q: I'm working on a feature branch (oslo.messaging/feature/amqp-dispatch-router) and I'd like to merge master into the feature branch.  But I don't have push merge permissions.20:31
kgiustiIs this something a core can do for me?  or do I need to get the permissions explicitly set?20:31
*** e0ne has quit IRC20:32
clarkbkgiusti: usually its given to the core group on the project20:32
clarkbif the currebt acl doesnt allow it the acl should be updated20:32
kgiustikgiusti: interesting, I think the oslo.messaging acl only allows pushMerge for oslo-release, not oslo-messaging-core20:33
*** _sarob has quit IRC20:33
kgiusticlarkb: sorry - I was talking to myself again20:33
kgiusticlarkb: I'll check with the ptl20:34
*** yamahata has quit IRC20:34
*** yamahata has joined #openstack-infra20:35
*** fawadkhaliq has quit IRC20:35
*** baoli_ has quit IRC20:36
*** baoli has joined #openstack-infra20:37
*** ddieterly is now known as ddieterly[away]20:38
*** EricGonc_ has quit IRC20:40
*** jsavak has joined #openstack-infra20:40
*** pramchan has joined #openstack-infra20:40
*** nwkarsten has joined #openstack-infra20:41
*** nwkarsten has quit IRC20:41
*** nwkarsten has joined #openstack-infra20:42
*** tonytan4ever has quit IRC20:42
*** cloudtra_ has joined #openstack-infra20:42
openstackgerritJames E. Blair proposed openstack-infra/puppet-bandersnatch: Purge normalized URLs
jeblairmtreinish, fungi, dstufft: ^20:43
*** cloudtrainme has quit IRC20:43
*** baoli has quit IRC20:44
*** sdake has joined #openstack-infra20:45
JayFhmm. If someone's address in gerrit is throwing bounces (well, autoresponder saying they aren't employed at $company anymore); do you all typically take action to prevent email to that domain?20:46
fungiJayF: nope20:46
JayFI can give the address/message in pm, just didn't want to say who isn't working where anymore in open chat20:46
JayFso basically I'm going to get bounces until this old spec merges. dray.20:46
jeblair(gerrit really should have bounce processing)20:47
*** Apoorva has joined #openstack-infra20:47
clarkbtherve: looks like that function may have a unittest that is unhappy. Not sure if the test just needs new expectations or if its properly not working20:47
Kaiyan@fungi Thank you! I changed {} to {{}} and seems like its working locally!20:47
fungiJayF: if you as the sysadmin of teh mailserver there would like to "formally" complain that we're spamming your server, i could certainly see nullifying the address for you20:47
JayFjeblair: I've worked on software in that domain before. Pretty rough.20:47
jeblairJayF: verp ftw20:48
JayFjeblair: verb?20:48
JayFjeblair: verp?20:48
Kaiyanso for job-template, do we need the quote'' outside the name?20:48
Kaiyanin my case 'gate-{name}-tempest-dsvm'20:48
jeblairJayF:   it makes it much easier -- bypasses all that nasty parsing and stuff :)20:49
JayFjeblair: ah. I haven't worked much in the bounce-parsing business for almost 8 years :)20:49
JayFjeblair: I worked for a $not-a-spammer-we-swear company and we had to parse bounces + abuse reports20:49
jeblairJayF: if you send mail out with VERP return paths, and you later receive a message on that address, you're done -- it's a bounce for that address.  you don't have to worry about things like "well, i need to parse an exchange bounce message this way to figure out who it was for"20:50
jeblairJayF: our mailman install uses it20:50
jeblairlooks like ""20:51
JayFyeah, we did similar stuff with return path for bounce and abuse parsing20:52
JayFall we'd do for actual parsing is to check the "bounce||abuse" to see if it matched feedback loop / spam report emails from various I/ESPs20:53
*** raildo is now known as raildo-afk20:53
nwkarstenclarkb: i'm getting started on the debugging we talked about yesterday. how do you turn on shade logging?20:54
*** pramchan has quit IRC20:55
clarkbnwkarsten: Shrews should be able to answer that (I don't know off the top of my head)20:56
*** cody-somerville has quit IRC20:56
*** haypo has left #openstack-infra20:56
*** rprakash has joined #openstack-infra20:57
*** IlyaG has quit IRC20:57
*** rprakash has quit IRC20:57
*** dprince has quit IRC20:58
jeblairnwkarsten: are you using the shade python api, or debugging an existing program with its own logging config?20:58
jeblairnwkarsten: if you're using the shade python api, you can call: shade.simple_logging(debug=True)20:58
openstackgerritMerged openstack-infra/zuul: Set default_flow_style=False for readability
*** tonytan4ever has joined #openstack-infra20:59
jeblairnwkarsten: if you're working with some existing program that uses the python logging api, you can configure it something like how we do for nodepool:
nwkarstenjeblair: the shade python api. thanks20:59
jeblairnwkarsten: (but set level=DEBUG instead of WARNING)20:59
jeblairnwkarsten: cool, so that function is an easy start point.  shade uses the python logging api internally, so if you want more control, or later want to better incorporate it into your app, i recommend using that.  (i don't know what you're actually doing, so i'm covering all bases)21:00
*** IlyaG has joined #openstack-infra21:01
*** claudiub has joined #openstack-infra21:01
nwkarstenjeblair: i was planning on injecting it into the nodepool code to figure out why i can't connect to my cloud21:02
jeblairnwkarsten: ah, then actually just changing the logging config that nodepool uses is probably easiest21:03
*** ddieterly[away] is now known as ddieterly21:03
*** gomarivera has quit IRC21:03
*** nadya has quit IRC21:03
jeblairi'm going to divert some jobs to zuulv2.521:04
*** claudiub|2 has quit IRC21:04
fungisounds good21:04
*** inc0_ has quit IRC21:04
fungiinfra jobs again, presumably?21:04
jeblairfungi: no, random.  nodepool isn't really set up for much else :(21:05
*** ddieterly is now known as ddieterly[away]21:05
openstackgerritKen Giusti proposed openstack-infra/project-config: Add pushMerge for oslo.messaging core for feature branches
*** inc0_ has joined #openstack-infra21:05
*** kgiusti has left #openstack-infra21:05
fungioh, right-o21:05
*** cody-somerville has joined #openstack-infra21:05
*** cody-somerville has joined #openstack-infra21:05
fungii guess you could maybe divert by node label21:05
fungi(infra is the only consumer of ubuntu-precise!)21:06
*** akshai has quit IRC21:06
fungibut friday afternoons are yolo territory anyway as far as i'm concerned21:07
clarkbjeblair: I hav reviewed the zk for nodepool and zuul interaction spec21:07
jeblairclarkb: thanks!21:08
fungialso worth noting, our fip quotas in bluebox and osic are completely full of leaked addresses21:08
fungii think they have been all day21:08
openstackgerritBrian Stajkowski proposed openstack-infra/project-config: New Project Quark Plugin
*** tonytan4ever has quit IRC21:08
fungiso we've really not been using those ~150 nodes worth of quota21:08
jeblairfungi: :(  and we reverted the fip cleanup, right?21:08
fungiyeah, pending the shade-side fix21:08
clarkbbecause it was crashy21:09
fungiused a previously unexercised and untested method in shade21:09
fungiwhich, as you'd expect, therefore did not actually work21:09
fungipabelanger worked on writing a fix and also some tests to exercise that code from now on21:10
clarkbhrm I thought that the claim was test wasn't needed in nodepool because it was tested in sahde :(21:10
*** lucas-afk has quit IRC21:11
*** thorst has quit IRC21:11
*** pradk has quit IRC21:11
fungimaybe nobody realized the testing for it in shade didn't exercise the same code paths/arguments nodepool used21:11
*** thorst has joined #openstack-infra21:11
Kaiyan@fungi, {{}} fixed the compare-xml-new problem. But when I run tox locally it is still failing with other errors. Here is the summary I got with running to:   compare-xml-new: commands succeeded   congratulations :) find: -printf: unknown primary or operator ERROR: InvocationError: '/Users/kaiyansheng/python_repos/project-config/tools/'21:11
Kaiyanlinters: commands succeeded ERROR:   gerrit: commands failed   grafyaml: commands succeeded ERROR:   projects: commands failed   jjb: commands succeeded   jenkins-project: commands succeeded ERROR:   zuul: commands failed21:12
KaiyanBut if I only run ./tools/, it passed...21:12
pabelangerfungi: clarkb: jeblair: The tests in shade are currently wrong, they test (and validate) the broken code path. See:
Kaiyanwhy it complained about a InvocationError when I ran tox then?21:13
*** trown is now known as trown|outtypewww21:13
fungiKaiyan: hard to say without seeing the full log. you could push your modified change up for a new patchset and we can just look at what the ci reports21:14
*** alexey_weyl has joined #openstack-infra21:14
*** jsavak has quit IRC21:14
KaiyanSounds good i can do that! Thanks!21:14
fungiKaiyan: looks like it's possible you're running that on a platform which doesn't support the same options to the find command, for example21:14
*** ldnunes has quit IRC21:15
fungiKaiyan: what platform did you try it on? we run that particular test on ubuntu 12.04 lts, i'm pretty sure21:15
fungimainly because the service which consumes that configuration is currently running on an ubuntu 12.04 lts server21:15
*** alexey_weyl has quit IRC21:15
clarkbpabelanger: a case of mocks are badness21:16
*** thorst has quit IRC21:16
fungiKaiyan: the "find: -printf: unknown primary or operator" bit is what makes me wonder21:16
pabelangerclarkb: indeed21:16
pabelangerclarkb: also, I can monitor stackviz build if needed21:16
*** cody-somerville has quit IRC21:17
clarkbpabelanger: ok you have the WIP on the change and said you would so was going to let you approve when ready21:17
*** lucasagomes has joined #openstack-infra21:18
pabelangerclarkb: WIP removed, I can approve this weekend to test the build21:18
*** jerryz has quit IRC21:19
*** rcernin has joined #openstack-infra21:20
openstackgerritKaiyan Sheng proposed openstack-infra/project-config: [Monasca] Add check-requirements check job for monasca projects
*** markvoelker has joined #openstack-infra21:21
*** daemontool has quit IRC21:22
*** kzaitsev_mb has joined #openstack-infra21:24
mordredpabelanger, fungi: did we change the OSIC rate back to a higher rate?21:24
mordredor are we leaking still at the lower rate?21:24
*** markvoelker has quit IRC21:24
fungimordred: i do not know whether the change to raise the delay for osic merged21:24
fungii'll check21:24
openstackgerritKaiyan Sheng proposed openstack-infra/project-config: [Monasca]Add Tempest Jobs as non-voting for monasca repos
*** markvoelker has joined #openstack-infra21:24
openstackgerritDavid Lyle proposed openstack-infra/devstack-gate: Test patch
fungidoesn't look like it's gotten approved yet. hunting it down now21:25
pabelangermordred: fungi: I manually tested that yesterday, we still leaked FIPs at 0.521:25
*** flepied1 has quit IRC21:25
fungioh, poo21:25
pabelangermordred: fungi: it wasn't as fast, but we still leaked21:25
*** markvoelker_ has joined #openstack-infra21:25
*** flepied has joined #openstack-infra21:26
fungimordred: if you're not around, you might want to not review the 318369 fix to shade's floating ip handling21:26
fungiShrews has a comment raising specific questions about your intent with it21:27
openstackgerritBrian Stajkowski proposed openstack-infra/project-config: New Project Quark Plugin
*** IlyaG has quit IRC21:27
*** ddieterly[away] is now known as ddieterly21:28
openstackgerritDavid Lyle proposed openstack-infra/devstack-gate: Test patch
*** markvoelker_ has quit IRC21:28
*** markvoelker_ has joined #openstack-infra21:28
openstackgerritDavid Lyle proposed openstack-infra/project-config: Update horizon dsvm tempest job
*** markvoelker has quit IRC21:30
*** matt-borland has quit IRC21:32
*** thorst has joined #openstack-infra21:34
asselin__Any puppet/Gem folks know what the correct format for this is supposed to be? There was an error parsing `Gemfile`: Illformed requirement ["'3.8.1'"]. Bundler cannot continue.  There was an error parsing `Gemfile`: Illformed requirement ["'~> 3.8.1'"]. Bundler cannot continue.21:35
asselin__nibalizer, ^^ ??21:35
clarkbasselin__: maybe no space?21:35
*** lakshmiS has quit IRC21:35
asselin__will try it21:35
clarkbno apparently the space is normal21:36
asselin__here shows a space:
clarkbasselin__: I have found examples of both so assuming that either is fine21:36
*** aysyd has quit IRC21:37
clarkbasselin__: ~> means >= 3.8.1 and < 4.021:37
clarkbso the ~> should be valid too21:37
*** xyang1 has quit IRC21:37
*** thorst has quit IRC21:38
asselin__this is using bundler-1.12.421:38
*** IlyaG has joined #openstack-infra21:40
openstackgerritMatthew Treinish proposed openstack-infra/devstack-gate: Print installed tempest-plugins after tempest run
clarkbmtreinish: ^ reminded me, any progress using a common venv for tempest?21:41
mtreinishclarkb: it's all done21:41
mtreinishclarkb: david-lyle is testing using it for the horizon plugin right now:
clarkbwoot, did that speed up jobs by a couple minutes?21:42
mtreinishthe real patch is:
mtreinishclarkb: you know I never checked, let me look at openstack-health21:42
clarkbI have a d-g change up that should give us back a minute or two as well21:43
openstackgerritKhai Do proposed openstack-infra/jenkins-job-builder: Prevent corruption of cache by using atomic update
clarkb(related to making swapfiles)21:43
*** cody-somerville has joined #openstack-infra21:45
openstackgerritPrakash Ramchandran proposed openstack-infra/project-config: add new project meghdwar
*** kzaitsev_mb has quit IRC21:45
mtreinishclarkb: hmm no noticeable change in devstack:
mtreinishthe patch landed on may 9th21:46
* mtreinish just noticed we don't graph run execute time anywhere on o-h21:47
mtreinishI should fix that21:47
mtreinish(for the whole job)21:47
clarkbgraphite has the data21:47
mtreinishclarkb: in this case devstack was always making to venvs, and d-g an occasional 3rd21:48
*** savihou has quit IRC21:48
clarkband now we are down to one right?21:48
*** kzaitsev_mb has joined #openstack-infra21:48
mtreinishexcept for the all-plugin job, then it's 2 (that 1 has to be seperate because it uses system site-packages)21:48
*** ffrank has joined #openstack-infra21:49
*** lascii is now known as alaski21:50
*** cody-somerville has quit IRC21:50
mtreinishclarkb: unrelated can you check if elastic-recheck got cloned and installed in the openstack-health venv on health.o.o?21:51
mtreinishI don't see any evidence the recent puppet changes took effect on puppetboard21:51
*** kzaitsev_mb has quit IRC21:53
clarkbmtreinish: there is a /opt/elastic-recheck21:53
mtreinishok cool21:53
mtreinishthen I guess it ran and just didn't say anything on puppetboard21:54
*** eventingmonkey has quit IRC21:54
mtreinishI guess I'll have to wait ~30min for the cache to go stale to see if it picks up anything on e-r on the next refresh21:55
*** Swami_ has joined #openstack-infra21:56
clarkbmtreinish: we don't report failures properly to puppetboard anymore21:56
*** jsavak has joined #openstack-infra21:57
*** dmk0202 has quit IRC21:57
clarkbbut I see no errors in syslog21:57
openstackgerritMerged openstack-infra/devstack-gate: Make reproduce function quiet
*** cody-somerville has joined #openstack-infra21:58
mtreinishclarkb: yeah I think it worked fine it just didn't show the exec or the new VCS repo on puppetboard21:58
mtreinishclarkb: it's definitely applied the latest patch: has the bug field now21:59
mtreinishit's just empty right now21:59
Kaiyan@fungi I ran it on macbook hmmm21:59
Kaiyan@fungi now it passed the check job!21:59
*** Swami has quit IRC21:59
*** sdake has quit IRC21:59
fungiKaiyan: that explains it. -printf is probably not covered by posix, likely a gnu-ism22:00
*** Swami_ has quit IRC22:00
fungi(or perhaps osx's bsd find is non-posix compliant?)22:00
fungiregardless, i'm unsurprised that script doesn't work as intended on non-gnu-linux22:00
Kaiyanohhh gotcha! Thank you!!!22:01
Kaiyanyeah that make sense!22:01
*** kus has quit IRC22:01
*** sigmavirus24 is now known as sigmavirus24_awa22:01
clarkbI almost missed so putting it here now for others that may have missed it22:02
mtreinishfungi: so probably a gnuism22:03
*** burgerk_ has quit IRC22:04
*** sarob has joined #openstack-infra22:04
*** _sarob has joined #openstack-infra22:05
*** ddieterly has quit IRC22:06
mtreinishfungi: although I don't see find on the commands tested:
mtreinishI do like how they check for a fortran 77 compiler though22:08
fungimtreinish: a sub-possibility of my "-printf is probably not covered by posix" guess. find on the whole is perhaps just not covered22:09
*** sarob has quit IRC22:09
*** sdake has joined #openstack-infra22:09
*** inc0_ has quit IRC22:09
fungiyeah, posix is an interesting tour of things people thought at one time would be important to standardize but which often had little relevance in hindsight22:10
pleia2aw "Your proposal for Jenkins World 2016 has not been accepted" (nibalizer submitted a refreshed version of how we use Jenkins)22:10
*** cloudtra_ has quit IRC22:11
pleia2but it's probably for the best, given zv3 and all22:11
timrc"How to systematically review Jenkins from your CI"22:11
timrclol @ review22:11
timrcI think my brain and fingers are working together to tell me it's time for a beer.22:12
fungijust now?22:12
*** vhosakot has quit IRC22:12
*** vhosakot_ has joined #openstack-infra22:12
fungii guess texas is an "after 5" crowd22:12
*** regXboi has quit IRC22:13
*** sridhar_ram has joined #openstack-infra22:16
jeblairi just wrote a 3 line function because it needs 18 lines of comments.  thanks rsync!22:16
*** esikachev has quit IRC22:16
*** amitgandhinz has quit IRC22:18
*** HeOS has quit IRC22:20
openstackgerritKhai Do proposed openstack-infra/system-config: Install the Gerrit delete-project plugin
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: include PATH
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: add -q to rsync ssh command
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: create parent directories for SCP
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: handle Ant paths in publishers
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: set $HOME
*** flepied has quit IRC22:21
openstackgerritKhai Do proposed openstack-infra/system-config: Install the Gerrit delete-project plugin
*** stevemar is now known as notstevemar22:24
*** IlyaG has quit IRC22:24
*** jsavak has quit IRC22:24
*** zz_dimtruck is now known as dimtruck22:26
*** tpsilva has quit IRC22:27
timrcfungi: It could be sooner than 5... desire > effort is usually the tipping point ;-)22:28
*** bhunter71 has quit IRC22:30
*** |-paul-| has joined #openstack-infra22:30
*** bauwser is now known as bauzas22:35
*** camunoz_ has joined #openstack-infra22:36
nibalizerpleia2: haha22:37
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: add -q to rsync ssh command
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: create parent directories for SCP
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: set $HOME
clarkbmtreinish: small bug in
mtreinishclarkb: oops, good catch22:40
mtreinishI'll respin it right now22:40
jeblairokay, one more zuulv2.5 test with those ^22:40
*** Kaiyan has quit IRC22:42
pleia2nibalizer: mind reader22:42
openstackgerritMatthew Treinish proposed openstack-infra/devstack-gate: Print installed tempest-plugins after tempest run
mtreinishclarkb: ^^^22:42
*** ddieterly has joined #openstack-infra22:43
*** ddieterly is now known as ddieterly[away]22:43
*** ddieterly[away] is now known as ddieterly22:43
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: include PATH
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: add -q to rsync ssh command
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: create parent directories for SCP
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: handle Ant paths in publishers
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: set $HOME
*** jamielennox|away is now known as jamielennox22:48
nibalizerpleia2: o/22:49
pleia2nibalizer: one of my close friends was just hired to maintain the infra for gluster <322:50
*** Swami has joined #openstack-infra22:50
*** chem`` has quit IRC22:50
*** chem`` has joined #openstack-infra22:51
jeblairOH!  we could have zuulv2.5 *not* upload logs when zuul aborts a running job!22:52
clarkbooh fancy features22:53
openstackgerritJohn L. Villalovos proposed openstack-infra/irc-meetings: Give example #startmeeting command
notmorganjeblair: oooh22:53
jeblair(zuul aborts a lot of jobs, that should save us some space)22:53
asselin__clarkb, just to follow-up. The version syntax is correct ~>, just that jenkins environment injection doesn't work with that. Exporting it directly works in the jenkins job solves the issue.22:55
fungiheads up that might be good to include some infra peeps with a vested interest in glance consistency/interoperability (note it's also during our precise-to-trusty upgrades week)22:56
*** krotscheck is now known as krotscheck_dcm22:57
*** xarses has quit IRC22:57
notmorganfungi: out of curiosity, we're planning to keey trusty around for a while? I noticed xenial ships with py35 not py34 (though the differences are relatively minor)22:59
*** salv-orlando has joined #openstack-infra22:59
notmorganfungi: (i ran into this when doing some py34 fixes for keystone)22:59
funginotmorgan: yeah, the idea is to test >=newton on xenial (so probably py35 jobs will get added)23:00
*** bpokorny has quit IRC23:00
notmorganfungi: ah okie.23:00
notmorgancool. that was my thought but just confirming23:00
funginotmorgan: <newton sticks to trusty, and we can continue to test py34 there23:00
* notmorgan is babysitting the final py34 fixes for keystone though the gate.23:00
*** csomerville has joined #openstack-infra23:01
notmorganhad to compile-from-source py34 so i didn't need to run a VM on my laptop23:01
notmorgani mena... sure that *sounds* insane when i say it outloud23:01
fungicompiling software from source sounds insane?23:02
notmorganno compiling software from source to avoid running a VM or docker container...23:02
notmorganthat sounds kindof insane23:02
*** hongbin has quit IRC23:02
*** sdague has quit IRC23:03
* fungi has done that plenty23:03
notmorgani haven't figured out where i misplaced _curses though in my py34 build :P23:03
*** |-paul-| has quit IRC23:03
*** ams__ has quit IRC23:03
notmorganso i can't use bpython23:03
jeblairthis is starting to look really close...23:04
*** cody-somerville has quit IRC23:05
*** csomerville has quit IRC23:05
clarkbnotmorgan: fwiw everyone I have talked to says py34 just works under py3523:05
clarkbthe other direction not so much23:05
notmorganclarkb: that is my experience23:05
notmorganclarkb: which is great.23:06
clarkbsince py35 introduces type annotations and other stuff23:06
notmorganyeah i kindof dig that23:06
notmorganbut unfortuantely we wont get to use that for a nice long time :P23:07
openstackgerritJames E. Blair proposed openstack-infra/zuul: Append to console log, rather than truncating
notmorganclarkb: fwiw, keystone should be python3 compat today :)23:07
clarkbnotmorgan: are you gatibg on it?23:07
jeblairthat will address the surprisingly short console logs :)23:07
notmorganclarkb: yes. will need to re-check 2 patches23:08
notmorganclarkb: and we'll have no more blacklisted tests23:08
notmorganclarkb: the biggest barrier was pyldap vs python-ldap23:08
notmorganand ldappool.23:08
clarkbnotmorgan: I also spoke to pypy people yesterday and they seem keen to do get pypy testing going again23:08
jeblairclarkb: i think this is the major zuulv2.5 mystery remaining:
clarkbnotmorgan: they also said keystone + eventlet + pypi is quicker than with wsgi server23:08
notmorganwell eventlet will not happen in keystone23:09
notmorganclarkb: it's been dropped from our code base 100%23:09
jeblairi'm guessing wsgi is pypy incompatible23:10
clarkbya I told tgen that they shouldve said somethinguch sooner23:10
notmorganjeblair: as is uwsgi, but gunicorn should be compat23:10
clarkbjeblair: it runs but I think it has trouble with jit warmup23:10
clarkbamong other things23:10
notmorganjeblair: the issue we have with gunicorn is oslo_config stomps all over gunicorn's config23:10
*** asselin has joined #openstack-infra23:11
notmorganbut i bet gunicorn + pypy + keystone will be good once we solve oslo_config issues23:11
* notmorgan prefers uwsgi.23:11
clarkbjeblair: file perms maybe?23:11
clarkbat least with apache mod wsgi I am not sure processes live long enough to achieve max jit goodness23:12
clarkbnot aure if gunicorn hamdles that better23:12
jeblairclarkb: i guess there are 2 mysteries -- 1) why no logs? and 2) why exit?23:12
notmorganclarkb they can w/ mod_wsgi23:12
notmorganclarkb: itr's just config driven23:12
notmorganbut... meh23:12
jeblairclarkb: i was able to hold that node, so that should help23:12
jeblairclarkb: it's at
notmorganclarkb, jeblair: what is the rule about dropping things from g-r23:13
notmorgancode search doesn't seem to indicate anyone in newton depends on python-ldap23:13
jeblairclarkb: i suspect the symlink at logs/ may be part of the cause of (1)23:13
clarkbjeblair: no logs because d-g isnt there23:13
* notmorgan is checking before propsing it.23:13
*** marcusvrn_ has quit IRC23:13
jeblairclarkb: there is a logs directory23:13
jeblair2016-05-20 22:55:06.449 | rsync: mkdir "/opt/stack/new/devstack" failed: Permission denied (13)23:14
*** asselin__ has quit IRC23:14
*** gyee has quit IRC23:15
jeblairnope, that's too late, sorry; actual cause is earlier i think23:15
jeblair2016-05-20 22:55:06.336 | /home/jenkins/workspace/gate-tempest-dsvm-neutron-full/devstack-gate/ line 354: blkid: command not found23:15
jeblairclarkb: i think we assume /sbin is in the path :)23:16
fungipath yeah23:16
jeblairso we could add it, or perhaps this is another thing that should nudge us to just running this stuff in a login shell :)23:16
fungiblkid is in util-linux after all23:16
jeblairyeah, which says it's in sbin23:17
clarkb++ to using login shell23:17
*** xarses has joined #openstack-infra23:17
jeblairokay, i'll do that and drop our PATH and HOME stuff; then figure out what the deal is with rsync and symlinks23:17
fungimay need to take extra care with child processes inheriting file descriptors in that case?23:18
fungii know running devstack-gate/tempest under an interactive shell has often left me unable to disconnect after without hunting down child processes to kill because they had refcount on stdin/stdout/stderr23:19
*** ihrachys has quit IRC23:19
fungithat's at least one up-side to the jenkins slave agent model23:20
jeblairi will probably "/bin/bash -l -c /tmp/"23:21
fungishould help, yeah23:21
fungii think it was mostly stuff forking and disassociating from child processes but not closing inherited file descriptors23:21
fungiso they weren't proper children any longer23:22
* fungi starts to worry he's giving questionable parenting advice now23:22
*** nwkarsten has quit IRC23:23
*** roxanagh_ has quit IRC23:25
*** vhosakot_ has quit IRC23:25
* fungi believes it was dr. spock who said, "fork once, close your inherited file descriptors, reset your cwd to /, unset the foreground process group, then fork again!"23:27
*** nwkarsten has joined #openstack-infra23:27
*** Swami_ has joined #openstack-infra23:27
*** charz_ has quit IRC23:27
*** timburke has quit IRC23:27
*** roxanagh_ has joined #openstack-infra23:30
*** Swami has quit IRC23:31
*** charz has joined #openstack-infra23:32
*** nwkarsten has quit IRC23:32
*** salv-orlando has quit IRC23:33
*** jpr has quit IRC23:35
notmorganfungi: lol23:36
*** timburke has joined #openstack-infra23:38
*** jerryz has joined #openstack-infra23:40
mnaserdoes nodepool build raw images at the moment?23:45
fungimnaser: it builds raw/vhd and qcow223:48
fungimnaser: we prefer to upload qcow2 to providers who support it since that has native compression applied23:48
*** bpokorny has joined #openstack-infra23:48
*** Swami__ has joined #openstack-infra23:48
mnaserfungi: hmm, this has positives and negatives, if you upload qcow2 it'll be quite slow to deploy (it has to download the image, convert to raw and upload it back to ceph)23:49
mnaservs if it's raw, it'll just make a new cow volume23:49
*** markvoelker_ has quit IRC23:49
openstackgerritDavid Lyle proposed openstack-infra/devstack-gate: Test patch
fungimnaser: the flip side is we frequently fail to upload raw versions of our images to providers because the significantly larger size makes them more prone to network disruption, timeouts, et cetera23:50
fungiand in theory the network inside the provider is faster and more stable than the internet at large23:50
mnaserfungi: i see where you're coming from, that's perfectly valid23:51
mnaserfungi: wonder if this could be worked around by having glance tasks that import qcow and do the conversion on the glance host23:52
*** Swami_ has quit IRC23:52
mnasersorta like how some other provider do conversions to vhd23:52
*** pvaneck has quit IRC23:52
openstackgerritBrian Stajkowski proposed openstack-infra/project-config: New Project Quark Plugin
fungimnaser: i thought more recent glance releases had implemented that, but i may have been misinformed23:52
mnaseri think this was introduced in kilo (tasks api + conversion)23:54
*** kus has joined #openstack-infra23:59
*** kus has quit IRC23:59
*** roxanagh_ has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!