Friday, 2017-12-01

*** chhavi has joined #openstack-powervm00:07
*** chhavi has quit IRC00:11
*** thorst has joined #openstack-powervm00:11
*** thorst has quit IRC00:15
*** thorst has joined #openstack-powervm00:57
*** thorst has quit IRC01:02
*** esberglu has quit IRC01:13
*** thorst has joined #openstack-powervm01:30
*** thorst has quit IRC01:35
*** esberglu has joined #openstack-powervm01:51
*** esberglu has quit IRC01:55
*** thorst has joined #openstack-powervm02:01
*** thorst has quit IRC02:07
*** AlexeyAbashkin has joined #openstack-powervm02:12
*** thorst has joined #openstack-powervm02:15
*** thorst has quit IRC02:16
*** AlexeyAbashkin has quit IRC02:16
*** zerick_ has joined #openstack-powervm02:19
*** zerick has quit IRC02:22
*** thorst has joined #openstack-powervm02:44
*** thorst has quit IRC02:44
*** thorst has joined #openstack-powervm03:25
*** thorst has quit IRC03:29
*** thorst has joined #openstack-powervm04:04
*** thorst has quit IRC04:08
*** thorst has joined #openstack-powervm04:44
*** thorst has quit IRC04:49
*** thorst has joined #openstack-powervm05:20
*** esberglu has joined #openstack-powervm05:23
*** thorst has quit IRC05:25
*** esberglu has quit IRC05:27
*** chhavi has joined #openstack-powervm05:35
*** thorst has joined #openstack-powervm05:54
*** thorst has quit IRC06:00
*** esberglu has joined #openstack-powervm06:03
*** esberglu has quit IRC06:08
*** thorst has joined #openstack-powervm06:29
*** thorst has quit IRC06:38
*** thorst has joined #openstack-powervm07:09
*** thorst has quit IRC07:13
*** thorst has joined #openstack-powervm07:47
*** thorst has quit IRC07:52
*** AlexeyAbashkin has joined #openstack-powervm08:15
*** thorst has joined #openstack-powervm08:25
*** thorst has quit IRC08:29
*** esberglu has joined #openstack-powervm08:48
*** thorst has joined #openstack-powervm08:57
*** thorst has quit IRC09:01
*** thorst has joined #openstack-powervm09:29
*** thorst has quit IRC09:34
*** thorst has joined #openstack-powervm10:07
*** thorst has quit IRC10:11
*** thorst has joined #openstack-powervm10:44
*** thorst has quit IRC10:48
*** chas has joined #openstack-powervm10:59
*** chas has quit IRC10:59
*** thorst has joined #openstack-powervm11:17
*** thorst has quit IRC11:22
*** esberglu has quit IRC11:49
*** esberglu has joined #openstack-powervm11:49
*** thorst has joined #openstack-powervm11:57
*** thorst has quit IRC12:01
*** thorst has joined #openstack-powervm12:11
*** catmando has joined #openstack-powervm12:48
catmandohey all12:48
catmandoi have a question regarding cloud-init12:48
catmandospecifically, i am reading!/wiki/Wc1c29d23e0fd_4346_b509_f1c00a2099f0/page/Troubleshooting%20when%20a%20VM%20using%20cloud-init%20does%20not%20ping%20or%20boot12:49
catmandonot that my cloud-init instance does not boot, but because I am trying to understand how config drive works on powervc12:50
catmandothe section that contains "On a VM you can mount the CD drive..." confuses me: what CD drive?12:50
efriedcatmando CD stands for "Compact Disc".  It is an ancient technology whereby you would place a small, flat, plastic donut into a tray or slot in a physical computer, and an actual laser would read data from it.12:52
catmando@efried :|12:53
catmandowhat I meant was, why is that in the documentation?12:53
efriedTechnically the drivers on the guest that are reading the data off of the config drive are the same ones from decades ago that read a physical CD drive.  The config drive is a virtualized CD.12:54
efriedcatmando But you're right that we probably shouldn't be calling it "CD" in the documentation, ever.12:54
efriedthorst mdrabe Do you know who owns that documentation?12:54
efriedesberglu Are you actually there?  I did some work on log coloring yesterday and I'm eager to try it out.12:55
catmandowait, so the config drive is automatically set up on the guest when cloud-init runs?12:55
efriedcatmando Other way around.  The config drive is set up automatically (assuming you have the appropriate conf options set, which you do if you're using PowerVC) and cloud-init is run based off of the data on the config drive.12:56
catmandothe logs contain many lines of the type: "[  257.767867] cloud-init[2234]: 2017-12-01 12:40:59,341 -[WARNING]: Calling '"12:56
catmandofrom my (so far basic) understanding of cloud-init, that is the config drive service it's trying to find12:56
efriedIs that a log on the guest or the host?12:56
catmandoguest, cloud-init.log12:57
efriedI'm not an expert there, I'm afraid.  mdrabe may know more.12:57
catmandoand i'm trying to understand what, if anything i'm missing. the guests are coming up fine, but that timeout is enourmous and it adds 2-3 minutes to initial boot12:57
catmando. /s/enourmous/enormous12:58
efriedcatmando Is that a real IP?12:59
efriedIs there anything behind it?13:00
catmandothat ip is not real, since there's nothing in the docs telling me what service i would need to run there or how to set it up13:02
catmandoefried ^13:02
efriedcatmando We're going to need to hook you up with someone who actually understands cloud-init, cause I don't.  mdrabe may know more, or know who to ask.13:05
catmandoefried, sure, i'll hang about, see if anyone chips in13:05
efriedcatmando I assume there's some way to specialize what gets run in there.13:06
catmandoas would i, but there's nothing that i can see in the docs13:06
*** efried is now known as fried_rice13:22
*** svenkat has joined #openstack-powervm13:22
fried_ricesvenkat Greetings.  Do you know anything about cloud-init?13:25
svenkatGood morning! yes. I can help. If not, i will get answers from my team… i have set up cloudinit before.13:27
fried_riceGreat.  catmando ^ care to repeat your question for svenkat?13:28
fried_ricesvenkat You can see the previous discussion on eavesdrop:
catmandosvenkat: cloud-init defines datastores, one of which is config drive13:29
fried_ricesvenkat Including /me being snotty.13:29
svenkatlet me look. yes it defines config drive…13:29
catmandomy cloud-init guests are coming up fine, but i don't understand why they're looking (for 2-3 minutes) for a config drive service at address .254 on the network they come up on13:30
catmandonor do i understand the docs that refer to a cd drive and how that (virtual) drive plays into it13:30
fried_ricecatmando We ought to fix the docs, but you can just mentally map CD=>config13:30
fried_rice(svenkat, if you know who owns those docs, perhaps we can get 'em fixed?)13:31
svenkatIst his AIX, ubuntu, rhel, sles?13:31
svenkatsure. I will work with ID team.. good point.13:31
catmandoin this initial case, ubuntu 16.0413:31
svenkatok… thanks, i will ping our ID team13:32
catmandoi still don't understand what i'm supposed to do13:33
catmandodo i want / need to set up a config drive service at that IP? how do I do that? I understand
*** svenkat_ has joined #openstack-powervm13:35
*** svenkat has quit IRC13:36
*** svenkat_ is now known as svenkat13:36
catmandobut is that implemented in powervm/vc? there's nothing in the documentation that I can find about it, but I may be looking in the wrong place...13:37
svenkatin your cloud.cfg, you should have an entry like : datasource_list: ['ConfigDrive']13:38
svenkatdo you see it?13:38
*** apearson has joined #openstack-powervm13:40
svenkatok… i guess our troubleshooting wiki needs update. let me get it updated. I will work with ID team.13:40
*** apearson has quit IRC13:44
*** svenkat has quit IRC13:46
*** svenkat has joined #openstack-powervm13:48
-openstackstatus- NOTICE: gerrit has been restarted to get it back to its normal speed.13:52
*** svenkat_ has joined #openstack-powervm13:52
*** svenkat has quit IRC13:53
*** svenkat_ is now known as svenkat13:53
*** AlexeyAbashkin has quit IRC14:02
*** esberglu has quit IRC14:10
*** esberglu has joined #openstack-powervm14:10
*** catmando has quit IRC14:18
mdrabefried_rice: burgerk is the cloud-init guy if that's what you're looking for14:45
fried_ricemdrabe Good to know, thanks.  Next time we see catmando...14:45
svenkati emailed burgerk and Eric, I copied you…14:53
fried_riceoh, yeah, saw that, thanks svenkat15:04
fried_riceesberglu ?15:13
esberglufried_rice: whats up?15:44
*** thorst has quit IRC15:45
fried_riceesberglu I want to get at that log server and fiddle with the htmlification bits.15:45
fried_riceesberglu First of all, is os-loganalyze up to date there?15:45
esberglufried_rice: Should be. I think it's in /opt/os-loganalyze15:46
esbergluI can ping you creds15:46
fried_riceesberglu Okay, I have a patch for it.  Then we can turn log coloring back on in journalctl.15:47
fried_riceMeh, you were one patch behind.15:54
esberglufried_rice: Link to the patch?15:59
fried_riceJust posted it:
fried_riceTwo different options in there.  They produce slightly different results, and I haven't dug into why yet.15:59
fried_riceesberglu Okay, the patch is installed on the log server.  Do we need to restart things (like apache) to pick it up?  First thing I want to do is see if/how it changes what the regular logs look like.16:04
esberglufried_rice: You pip installed from /opt/os-loganalyze? Or you just have the patch there16:06
fried_riceesberglu I put the patch there.  Assumed it was already pip installed or whatever.  Not so?16:08
fried_riceesberglu Oh, no, looks like it's picking up out of the lib16:09
esberglufried_rice: I'm looking at how that works right now, we just run a puppet manifest that sets up all the logserver stuff16:09
fried_riceLet me pip install the local os-loganalyze.  Might just work.16:09
esberglufried_rice: If not I can probably just rerun the manifest16:10
esbergluAs long as that doesn't pull down master over your change16:10
fried_riceesberglu eeek, I'm now getting perm errors...16:10
esberglufried_rice: hmm16:11
fried_riceI'm sure it's apache being safe.16:12
fried_riceesberglu I can hack the patch into the regular location.16:12
fried_riceesberglu Dangit, I don't know how to un-pip-install this directory.16:13
* fried_rice broke the log server.16:14
*** thorst has joined #openstack-powervm16:15
esberglufried_rice: We can always just pip uninstall it and rerun the playbook if things get too messed up16:18
esberglufried_rice: You're trying to switch back to using the version in the lib?16:18
fried_riceesberglu Yeah.  Which seems to be gone now.16:18
*** thorst has quit IRC16:20
esberglufried_rice: /usr/local/lib/python2.7/dist-packages/os-loganalyze/16:20
esbergluIts there?16:20
fried_riceI just put it there.16:20
fried_riceBut I copied the git repo16:20
fried_riceSo it's wrong.16:20
fried_riceesberglu Where are the apache configs?16:22
fried_riceNyaha, I got it.16:22
esberglu /etc/apache2/sites-enabled/16:22
fried_riceesberglu Okay, this is weird; it's showing up, but it's showing the old style content, unchanged.16:23
fried_riceThe changes I made should at least have altered the background colors and stuff.16:23
*** thorst has joined #openstack-powervm16:24
fried_riceMaybe restart apached...16:24
fried_riceesberglu How do we do ^16:24
esberglusudo systemctl restart apache2.service16:24
esberglufried_rice: Heh it's all showing up as 1 line now16:25
esbergluI'm digging the black background though16:25
fried_ricebolz, I'm gonna have to restart apached for every delta.16:27
*** thorst has quit IRC16:28
fried_riceesberglu Better now.  I need to tweak the default colors.16:29
esberglufried_rice: Sweet. And yeah the info lines are tough to see16:31
fried_riceesberglu Okay, that's nicer.16:33
fried_riceCan we turn log colors back on for one run?16:33
esberglufried_rice: Nope. I could put a run through manually though16:34
fried_riceesberglu Whatever's easiest.  I want you to see what it looks like, and then we can decide what to do f'real.16:34
fried_riceesberglu Actually also want to make sure it renders right.  I had some trouble with newlines (as you can tell)16:35
esberglufried_rice: Yeah I can push a manual through, I'll ping you when its done16:35
fried_riceesberglu Thanks!16:35
*** thorst has joined #openstack-powervm16:41
*** thorst has quit IRC16:50
*** AlexeyAbashkin has joined #openstack-powervm17:13
fried_riceesberglu JFYI, the reaction from -nova and -infra is such that this change is not likely to get beyond our CI.17:13
fried_riceWhich is fine by me17:13
*** fried_rice is now known as fried_rolls17:14
*** AlexeyAbashkin has quit IRC17:17
*** thorst has joined #openstack-powervm17:21
esberglufried_rolls: Yeah that's cool with me. Once we know it's working I'll figure out the best way to carry it in the deployment playbooks17:21
*** thorst has quit IRC17:26
*** thorst has joined #openstack-powervm17:28
*** thorst has quit IRC17:33
*** thorst has joined #openstack-powervm17:34
*** AlexeyAbashkin has joined #openstack-powervm17:37
*** AlexeyAbashkin has quit IRC17:41
esbergluMost of the files aren't displaying anything17:57
*** chhavi has quit IRC18:10
*** AlexeyAbashkin has joined #openstack-powervm20:11
*** AlexeyAbashkin has quit IRC20:15
*** thorst has quit IRC20:19
*** fried_rolls is now known as fried_rice20:24
fried_riceesberglu Is there an apache log where exceptions from os-loganalyze might show up?20:24
esberglufried_rice: /var/log/apache2/...20:27 maybe?20:29
esbergluOh you must be looking at it already20:29
fried_riceyup, looking.20:29
fried_riceNot very helpful.20:30
fried_ricewas hoping for a stack trace.20:30
fried_riceOkay, let me do this: I'm just gonna patch it to remove the color codes entirely for now, and play with it more when I have some time.20:31
esberglufried_rice: Fine by me20:31
fried_riceesberglu I think something else is wrong here - those files really are empty.20:34
fried_riceesberglu Anyway, I patched it up and restarted apache2.  So if we can get some real logs in there...20:35
esberglufried_rice: No color at all now20:42
esberglufried_rice: nvm back to the way it was before all of this20:42
fried_riceesberglu link?20:42
fried_riceWait, what?  Now there's no color.20:43
fried_riceDid I miss something?20:43
fried_riceno color until partway down, then all red from then on.20:44
fried_riceThat's effed up.20:44
fried_riceat least the escape sequences are gone.20:45
fried_riceah, I found a bug from earlier.20:46
*** thorst has joined #openstack-powervm20:50
*** thorst has quit IRC20:55
fried_riceI also know wtf is happening with the new shit.  The regexes are getting applied before the color codes are coming off, so they're not matching properly.21:01
fried_riceesberglu There, quick and dirty and simple.  Now we can re-enable ansi colors in the CI runs.21:05
fried_ricerun git diff from /usr/local/lib/python2.7/dist-packages/os-loganalyze21:06
*** AlexeyAbashkin has joined #openstack-powervm21:11
esberglufried_rice: I still don't get color codes when I download the files21:13
fried_riceI'm not getting anything now.21:13
esberglufried_rice: When you wget using that URL you get the modified version21:13
esberglufried_rice: Try now21:14
fried_riceesberglu modified as in html?21:14
fried_riceesberglu Oh.  It must be running the filter even when it's not running the htmlview.  That's a pain.21:15
*** AlexeyAbashkin has quit IRC21:15
esberglufried_rice: yep21:15
fried_riceesberglu Sigh.  I can still make this work, but it's going to be more annoying.21:16
fried_riceesberglu I'll hack at it some more once I fix this here bug...21:16
esberglufried_rice: Sounds good21:20
*** thorst has joined #openstack-powervm21:20
*** thorst has quit IRC21:25
fried_riceesberglu THERE.21:29
*** thorst has joined #openstack-powervm21:31
*** thorst has quit IRC21:36
fried_riceesberglu Oh.  There was already an attempt to strip control characters.21:45
fried_riceIt just wasn't working very well, because the regex was too restrictive.21:46
fried_ricewhich is actually probably the culprit of my mismatchings21:46
esberglufried_rice: Where's that at?21:46 line 12621:46
fried_rice'cept actually that's switched off.21:47
fried_riceAnd I don't see anywhere it's switched on.21:47
*** thorst has joined #openstack-powervm21:49
*** thorst has quit IRC22:03
*** svenkat has quit IRC22:16
fried_riceesberglu I implemented themes :)22:36
*** thorst has joined #openstack-powervm22:40
*** thorst has quit IRC22:45
esberglufried_rice: That's sweet22:55
fried_riceThe feedback I got from the nova devs was "I use solarized" and "Colors are a distraction - get rid of them".22:55
*** fried_rice is now known as efried22:59
esbergluI wholeheartedly disagree23:05
*** thorst has joined #openstack-powervm23:14
*** thorst has quit IRC23:19
*** thorst has joined #openstack-powervm23:45
*** thorst has quit IRC23:50

Generated by 2.15.3 by Marius Gedminas - find it at!