ego trouble

[dhudson]

I Know this worked a few weeks ago but now I get error when i do,

~$ ego query versions ego
Traceback (most recent call last):
  File "/usr/bin/ego", line 102, in <module>
    EgoModule.run_ego_module(action, econfig, args, VERSION)
  File "/usr/share/ego/python/ego/module.py", line 64, in run_ego_module
    ego_module(*args)
  File "/usr/share/ego/python/ego/module.py", line 49, in __call__
    self.handle(**options)
  File "/usr/share/ego/modules/query.ego", line 41, in handle
    handler(**options)
  File "/usr/share/ego/modules/query.ego", line 56, in handle_versions_subcommand
    slot = ebuild.vars.get('SLOT', '0')
  File "/usr/lib64/python3.4/site-packages/appi/ebuild.py", line 122, in vars
    self._parse_ebuild_file()
  File "/usr/lib64/python3.4/site-packages/appi/ebuild.py", line 180, in _parse_ebuild_file
    make_conf = Profile.get_system_make_conf()
  File "/usr/lib64/python3.4/site-packages/appi/conf/profile.py", line 136, in get_system_make_conf
    context = Profile._parse_make_conf_file(path, context)
  File "/usr/lib64/python3.4/site-packages/appi/conf/profile.py", line 117, in _parse_make_conf_file
    r'^\s*(?:export\s+)?([a-z][a-z0-9_]*)=', f.read(), re.M | re.I
  File "/usr/lib/python-exec/python3.4/../../../lib64/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2220: ordinal not in range(128)

 

here is ego vers.

~$ eix ego
[I] app-admin/ego
     Available versions:  2.0.9^m **2.0.10^m 2.0.14^m **2.1.1^m 2.2.0^m 2.2.1^m 2.3.1^m **2.3.2^m **9999^m {zsh-completion PYTHON_SINGLE_TARGET="python3_4 python3_5 python3_6" PYTHON_TARGETS="python3_4 python3_5 python3_6"}
     Installed versions:  2.3.1^m(11:54:30 11/08/17)(-zsh-completion PYTHON_SINGLE_TARGET="python3_4 -python3_5 -python3_6" PYTHON_TARGETS="python3_4 -python3_5 -python3_6")
     Homepage:            http://www.funtoo.org/Package:Ego
     Description:         Funtoo's configuration tool: ego, epro, edoc.

also ego kit list does work so maybe the query part I am doing wrong or was removed.

no other issues with funtoo ego --sync and eix ect all work.

if I need to supply other info let me know..

thanks

 

[dhudson]

just to update this matter, turns out I had put

LC_ALL=C

IIRC .bashrc or .Xdefaults  or somewhere ?

once removed it all went away.

No idea why it matters, or why or what I was trying to do.

[palica]

python 3.6 needed for ego query

[dhudson]

that's interesting...didn't know that..

 complains about 0xb4 which is the unicode character for the backtick.

I guess I need to update python

 

thanks.

 

[drobbins]

@palica, ego query should be patched to work with python 3.4+.

[dhudson]

thanks for the reply,

I thought it strange my laptop runs the query module fine and also note that

the query origin works fine, but im still investigating this unicode error

i noticed a warning or notice during the conky update this morning

something related to U+xxx stuff but cant find it in my emerge logs.

in fact I have no emerge log for today. HA!

[gissf1]

I seem to have this issue too (nearly the same stack trace as the OP) and was wondering what else I could look for to diagnose it?

I've had this issue since I first noticed ego offering the query option, probably back around November, and simply assumed my system was just outdated and that's why it wasn't working.  After just recently updating my entire system with the latest ego, portage, and nearly everything else (emerge -uD @system @world) this issue is still persisting.  I'm running Python 3.4, I don't have LC_ALL defined in my environment, and I included the version info for ego itself below.

Stack trace:

# ego query v ego
Traceback (most recent call last):
  File "/usr/bin/ego", line 96, in <module>
    EgoModule.run_ego_module(action, econfig, args, VERSION)
  File "/usr/share/ego/python/ego/module.py", line 78, in run_ego_module
    ego_module(*args)
  File "/usr/share/ego/python/ego/module.py", line 63, in __call__
    self.handle(**options)
  File "/usr/share/ego/modules/query.ego", line 42, in handle
    handler(**options)
  File "/usr/share/ego/modules/query.ego", line 59, in handle_versions_subcommand
    slot = ebuild.vars.get('SLOT', '0')
  File "/usr/lib64/python3.4/site-packages/appi/ebuild.py", line 122, in vars
    self._parse_ebuild_file()
  File "/usr/lib64/python3.4/site-packages/appi/ebuild.py", line 180, in _parse_ebuild_file
    make_conf = Profile.get_system_make_conf()
  File "/usr/lib64/python3.4/site-packages/appi/conf/profile.py", line 136, in get_system_make_conf
    context = Profile._parse_make_conf_file(path, context)
  File "/usr/lib64/python3.4/site-packages/appi/conf/profile.py", line 117, in _parse_make_conf_file
    r'^\s*(?:export\s+)?([a-z][a-z0-9_]*)=', f.read(), re.M | re.I
  File "/usr/lib/python-exec/python3.4/../../../lib64/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2220: ordinal not in range(128)

ego version info:

# emerge --info app-admin/ego
Portage 2.3.18 (python 3.4.5-final-0, funtoo/1.0/linux-gnu/arch/x86-64bit, gcc-5.4.0, glibc-2.23-r6, 
4.9.16-gentoo_shard x86_64) 
... SNIP ...
sh bash 4.3_p48
... SNIP ...
dev-lang/python:          2.7.12::python-kit, 3.4.5::python-kit
... SNIP ...
Unset:  CPPFLAGS, CTARGET, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS

=================================================================
                        Package Settings
=================================================================

app-admin/ego-2.3.3-r1::core-kit was built with the following:
USE="-zsh-completion" PYTHON_SINGLE_TARGET="python3_4 -python3_5 -python3_6" PYTHON_TARGETS="python3_4 -python3_5 -python3_6"

Any ideas?

[cardinal]

@gissf1

Runs as expected on my system:

rj@funtoo ~ $ ego query v ego
 app-admin/ego| slot|               repo
--------------+-----+-------------------
         2.3.3|    0| core-kit/1.0-prime
    * 2.3.3-r1|     | core-kit/1.0-prime
          9999|     | core-kit/1.0-prime

Install strace, run the commands below and compare your output to mine linked to bpaste.

Differences between the two may point to a solution.

strace ego query v ego

strace -e open ego query v ego

strace -e read ego query v ego

[gissf1]

@cardinal

Thanks for that strace, that did help alot!

I noticed one major difference while initializing.  It seems that on my system its using:

/usr/lib/python-exec/python3.4/../../../lib64/python3.4/encodings/ascii.py

and on your system its using:

 /usr/lib/python-exec/python3.4/../../../lib64/python3.4/encodings/utf_8.py

Using ascii instead of utf_8 in python is not completely unexpected on my system since I try to avoid building anything outside en_US locale with ISO-8859-1 encoding.

Later, (on both yours and mine) it reads the file:

/var/git/meta-repo/kits/core-kit/profiles/base/make.defaults

In that file, it finds non-ascii (code points >0x7e) characters in the file contents as part of a utf-8 sequence in one of the first author's names in a comment, and mine dies immediately after, because the python ascii parser does not attempt to handle any extended characters and throws exceptions instead.

So this brings up the next question of why my system is using ascii instead of utf_8.  I do have utf-8 support enabled in some places, and I have the referenced "utf_8.py" file (the one yours loaded earlier) in my system, but it's not choosing to load it for some reason.

I guess I'll dig in deeper and see where the rabbit hole leads.  Thanks again cardinal!

[gissf1]

OK, The mystery is solved!

I dug in and looked into where "ego query versions" opens the file.  This search led me to /usr/lib64/python3.4/site-packages/appi/conf/profile.py (part of dev-python/appi-0.1.6::python-kit in my system) which is opening the file as text without specifying an encoding, and then throwing the trace when it reads the files and realizes there are non-ascii characters.

I'm not sure what the best approach is, as I can imagine several options:

1. use errors='ignore' on open, but I think this drops the entire input data when it dislikes even 1 character in the data
2. use errors='replace' on open, which replaces unwanted characters with an encoding-specific character
3. specify an encoding on open that does not throw an exception on non-ascii characters
4. leave the open alone and catch the exception somewhere else
5. others?

I went with option 2 and it seemed to fix my problem.  The patch for profile.py is as follows:

--- profile.py  2018-02-17 06:12:06.000000000 -0600
+++ profile.py.modified 2018-02-24 15:52:43.000000000 -0600
@@ -112,7 +112,7 @@
             k: v for k, v in context.items()
             if k in constant.INCREMENTAL_PORTAGE_VARS
         }
-        with open(str(path), 'r') as f:
+        with open(str(path), 'r', errors='replace') as f:
             output_vars = set(re.findall(
                 r'^\s*(?:export\s+)?([a-z][a-z0-9_]*)=', f.read(), re.M | re.I
             ))

Since appi is on gitlab, I created an issue https://gitlab.com/apinsard/appi/issues/15 ticket and cross referenced this forum page.  I may submit a patch/pull request at some point, but thought I'd elicit some feedback before doing so.

Any thoughts on what the best solution is?

Edited February 27, 2018 by gissf1
Update with appi issue ticket link and correct some typos

[pytony]

Can someone try with `appi-0.1.9999` and confirm it fixes the issue? If it does, I'll release it as appi-0.1.7.

[pytony]

On 11/26/2017 at 1:11 PM, palica said:

python 3.6 needed for ego query

ego query should work fine with python 3.4+, did you face any issue about it?

[gissf1]

@pytony Thanks for the official patch!  I installed appi-0.1.9999 (post-0.1.6, pre-0.1.7), and everything worked out of the box (without additional patching), so I would call that a fix!

Also, in case it wasn't evident to others on this thread, I'm using Python 3.4 at the moment, so it absolutely can work with Python 3.4.

[pytony]

Thanks, I released appi 0.1.7. You can mask 0.1.9999 back and install 0.1.7 when it hit tree. :)

[pytony]

@gissf1 for the record, can you show me the output of the command locale in your system that failed with appi 0.1.6? Thanks

[gissf1]

@pytony I've done a few package updates since I last saw the issue, but didn't change any major configuration, so its probably still the same.  Here you go:

# locale
LANG=C
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE=POSIX
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=

I was using "set | grep LANG" and "set | grep LC" before to check my locale settings.  I just learned something new.  Thanks! :)

[dhudson]

I think the LANG=C could be a problem, I have this:

 :~$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE=POSIX
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

and have no problems since my previous post.

[pytony]

Yes the problem was that appi used the default system encoding to read files, which is ascii when LANG=C. I assumed encoding should always be utf-8 for these files and explicitly read them with this encoding. If it happens that they may be encoded with a different charset on some systems, I'll have to handle this...