ego trouble

dhudson · November 26, 2017

I Know this worked a few weeks ago but now I get error when i do,

~$ ego query versions ego
Traceback (most recent call last):
  File "/usr/bin/ego", line 102, in <module>
    EgoModule.run_ego_module(action, econfig, args, VERSION)
  File "/usr/share/ego/python/ego/module.py", line 64, in run_ego_module
    ego_module(*args)
  File "/usr/share/ego/python/ego/module.py", line 49, in __call__
    self.handle(**options)
  File "/usr/share/ego/modules/query.ego", line 41, in handle
    handler(**options)
  File "/usr/share/ego/modules/query.ego", line 56, in handle_versions_subcommand
    slot = ebuild.vars.get('SLOT', '0')
  File "/usr/lib64/python3.4/site-packages/appi/ebuild.py", line 122, in vars
    self._parse_ebuild_file()
  File "/usr/lib64/python3.4/site-packages/appi/ebuild.py", line 180, in _parse_ebuild_file
    make_conf = Profile.get_system_make_conf()
  File "/usr/lib64/python3.4/site-packages/appi/conf/profile.py", line 136, in get_system_make_conf
    context = Profile._parse_make_conf_file(path, context)
  File "/usr/lib64/python3.4/site-packages/appi/conf/profile.py", line 117, in _parse_make_conf_file
    r'^\s*(?:export\s+)?([a-z][a-z0-9_]*)=', f.read(), re.M | re.I
  File "/usr/lib/python-exec/python3.4/../../../lib64/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2220: ordinal not in range(128)

here is ego vers.

~$ eix ego
[I] app-admin/ego
     Available versions:  2.0.9^m **2.0.10^m 2.0.14^m **2.1.1^m 2.2.0^m 2.2.1^m 2.3.1^m **2.3.2^m **9999^m {zsh-completion PYTHON_SINGLE_TARGET="python3_4 python3_5 python3_6" PYTHON_TARGETS="python3_4 python3_5 python3_6"}
     Installed versions:  2.3.1^m(11:54:30 11/08/17)(-zsh-completion PYTHON_SINGLE_TARGET="python3_4 -python3_5 -python3_6" PYTHON_TARGETS="python3_4 -python3_5 -python3_6")
     Homepage:            http://www.funtoo.org/Package:Ego
     Description:         Funtoo's configuration tool: ego, epro, edoc.

also ego kit list does work so maybe the query part I am doing wrong or was removed.

no other issues with funtoo ego --sync and eix ect all work.

if I need to supply other info let me know..

thanks

dhudson · December 13, 2017

just to update this matter, turns out I had put

LC_ALL=C

IIRC .bashrc or .Xdefaults or somewhere ?

once removed it all went away.

No idea why it matters, or why or what I was trying to do.

palica · November 26, 2017

python 3.6 needed for ego query

dhudson · November 26, 2017

that's interesting...didn't know that..

complains about 0xb4 which is the unicode character for the backtick.

I guess I need to update python

thanks.

drobbins · November 27, 2017

@palica, ego query should be patched to work with python 3.4+.

dhudson · November 28, 2017

thanks for the reply,

I thought it strange my laptop runs the query module fine and also note that

the query origin works fine, but im still investigating this unicode error

i noticed a warning or notice during the conky update this morning

something related to U+xxx stuff but cant find it in my emerge logs.

in fact I have no emerge log for today. HA!

gissf1 · February 23, 2018

I seem to have this issue too (nearly the same stack trace as the OP) and was wondering what else I could look for to diagnose it?

I've had this issue since I first noticed ego offering the query option, probably back around November, and simply assumed my system was just outdated and that's why it wasn't working. After just recently updating my entire system with the latest ego, portage, and nearly everything else (emerge -uD @system @world) this issue is still persisting. I'm running Python 3.4, I don't have LC_ALL defined in my environment, and I included the version info for ego itself below.

Stack trace:

# ego query v ego
Traceback (most recent call last):
  File "/usr/bin/ego", line 96, in <module>
    EgoModule.run_ego_module(action, econfig, args, VERSION)
  File "/usr/share/ego/python/ego/module.py", line 78, in run_ego_module
    ego_module(*args)
  File "/usr/share/ego/python/ego/module.py", line 63, in __call__
    self.handle(**options)
  File "/usr/share/ego/modules/query.ego", line 42, in handle
    handler(**options)
  File "/usr/share/ego/modules/query.ego", line 59, in handle_versions_subcommand
    slot = ebuild.vars.get('SLOT', '0')
  File "/usr/lib64/python3.4/site-packages/appi/ebuild.py", line 122, in vars
    self._parse_ebuild_file()
  File "/usr/lib64/python3.4/site-packages/appi/ebuild.py", line 180, in _parse_ebuild_file
    make_conf = Profile.get_system_make_conf()
  File "/usr/lib64/python3.4/site-packages/appi/conf/profile.py", line 136, in get_system_make_conf
    context = Profile._parse_make_conf_file(path, context)
  File "/usr/lib64/python3.4/site-packages/appi/conf/profile.py", line 117, in _parse_make_conf_file
    r'^\s*(?:export\s+)?([a-z][a-z0-9_]*)=', f.read(), re.M | re.I
  File "/usr/lib/python-exec/python3.4/../../../lib64/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2220: ordinal not in range(128)

ego version info:

# emerge --info app-admin/ego
Portage 2.3.18 (python 3.4.5-final-0, funtoo/1.0/linux-gnu/arch/x86-64bit, gcc-5.4.0, glibc-2.23-r6, 
4.9.16-gentoo_shard x86_64) 
... SNIP ...
sh bash 4.3_p48
... SNIP ...
dev-lang/python:          2.7.12::python-kit, 3.4.5::python-kit
... SNIP ...
Unset:  CPPFLAGS, CTARGET, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS

=================================================================
                        Package Settings
=================================================================

app-admin/ego-2.3.3-r1::core-kit was built with the following:
USE="-zsh-completion" PYTHON_SINGLE_TARGET="python3_4 -python3_5 -python3_6" PYTHON_TARGETS="python3_4 -python3_5 -python3_6"

Any ideas?

cardinal · February 24, 2018

@gissf1

Runs as expected on my system:

Install strace, run the commands below and compare your output to mine linked to bpaste.

Differences between the two may point to a solution.

strace ego query v ego

strace -e open ego query v ego

strace -e read ego query v ego

gissf1 · February 24, 2018

@cardinal

Thanks for that strace, that did help alot!

I noticed one major difference while initializing. It seems that on my system its using:

/usr/lib/python-exec/python3.4/../../../lib64/python3.4/encodings/ascii.py

and on your system its using:

 /usr/lib/python-exec/python3.4/../../../lib64/python3.4/encodings/utf_8.py

Using ascii instead of utf_8 in python is not completely unexpected on my system since I try to avoid building anything outside en_US locale with ISO-8859-1 encoding.

Later, (on both yours and mine) it reads the file:

/var/git/meta-repo/kits/core-kit/profiles/base/make.defaults

In that file, it finds non-ascii (code points >0x7e) characters in the file contents as part of a utf-8 sequence in one of the first author's names in a comment, and mine dies immediately after, because the python ascii parser does not attempt to handle any extended characters and throws exceptions instead.

So this brings up the next question of why my system is using ascii instead of utf_8. I do have utf-8 support enabled in some places, and I have the referenced "utf_8.py" file (the one yours loaded earlier) in my system, but it's not choosing to load it for some reason.

I guess I'll dig in deeper and see where the rabbit hole leads. Thanks again cardinal!

gissf1 · February 24, 2018

OK, The mystery is solved!

I dug in and looked into where "ego query versions" opens the file. This search led me to /usr/lib64/python3.4/site-packages/appi/conf/profile.py (part of dev-python/appi-0.1.6::python-kit in my system) which is opening the file as text without specifying an encoding, and then throwing the trace when it reads the files and realizes there are non-ascii characters.

I'm not sure what the best approach is, as I can imagine several options:

use errors='ignore' on open, but I think this drops the entire input data when it dislikes even 1 character in the data
use errors='replace' on open, which replaces unwanted characters with an encoding-specific character
specify an encoding on open that does not throw an exception on non-ascii characters
leave the open alone and catch the exception somewhere else
others?

I went with option 2 and it seemed to fix my problem. The patch for profile.py is as follows:

--- profile.py  2018-02-17 06:12:06.000000000 -0600
+++ profile.py.modified 2018-02-24 15:52:43.000000000 -0600
@@ -112,7 +112,7 @@
             k: v for k, v in context.items()
             if k in constant.INCREMENTAL_PORTAGE_VARS
         }
-        with open(str(path), 'r') as f:
+        with open(str(path), 'r', errors='replace') as f:
             output_vars = set(re.findall(
                 r'^\s*(?:export\s+)?([a-z][a-z0-9_]*)=', f.read(), re.M | re.I
             ))

Since appi is on gitlab, I created an issue https://gitlab.com/apinsard/appi/issues/15 ticket and cross referenced this forum page. I may submit a patch/pull request at some point, but thought I'd elicit some feedback before doing so.

Any thoughts on what the best solution is?

Edited February 27, 2018 by gissf1
Update with appi issue ticket link and correct some typos

pytony · March 4, 2018

Can someone try with `appi-0.1.9999` and confirm it fixes the issue? If it does, I'll release it as appi-0.1.7.

pytony · March 4, 2018

On 11/26/2017 at 1:11 PM, palica said:

python 3.6 needed for ego query

ego query should work fine with python 3.4+, did you face any issue about it?

gissf1 · March 6, 2018

@pytony Thanks for the official patch! I installed appi-0.1.9999 (post-0.1.6, pre-0.1.7), and everything worked out of the box (without additional patching), so I would call that a fix!

Also, in case it wasn't evident to others on this thread, I'm using Python 3.4 at the moment, so it absolutely can work with Python 3.4.

pytony · March 7, 2018

Thanks, I released appi 0.1.7. You can mask 0.1.9999 back and install 0.1.7 when it hit tree. :)

pytony · March 7, 2018

@gissf1 for the record, can you show me the output of the command locale in your system that failed with appi 0.1.6? Thanks

gissf1 · March 7, 2018

@pytony I've done a few package updates since I last saw the issue, but didn't change any major configuration, so its probably still the same. Here you go:

# locale
LANG=C
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE=POSIX
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=

I was using "set | grep LANG" and "set | grep LC" before to check my locale settings. I just learned something new. Thanks! :)

dhudson · March 7, 2018

I think the LANG=C could be a problem, I have this:

 :~$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE=POSIX
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

and have no problems since my previous post.

pytony · March 10, 2018

Yes the problem was that appi used the default system encoding to read files, which is ascii when LANG=C. I assumed encoding should always be utf-8 for these files and explicitly read them with this encoding. If it happens that they may be encoded with a different charset on some systems, I'll have to handle this...

ego trouble

Question

Link to comment

Share on other sites

17 answers to this question

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites