Советы и рекомендации по обновлению с Python 2 до Python 3

2. Manual changes

The auto conversion doesn’t do the complete porting. The below common problems might be experienced during the porting process than the common syntax changes done by the auto conversion 2to3 tool. 

Run the test class and see if any errors and fix appropriately whether to switch from bytes to str or str to bytes or some sort/comparison issue where one has to fix the key name in the sorted function. This is an iterative process until all the code runtime has been validated.

Once a common pattern for sure is clear, then you can do grep and sed to replace across many class files. If you are not sure on other code until runtime, then defer until that test class is executed. 

There might be issues with third party libraries/modules might have changed, those need to be searched on the web and use appropriately.

Make sure all the code path is covered by running across all supported platforms and parameters.

3. Runtime Validation and Fix

Once the conversion is done, then perform a lot of code runtime as Python is a dynamic language. Otherwise, the changes can break the things if you do just visual static code inspection/changes. You can start with basic sanity tests, acceptance tests and then select full tests from a single module tests.

Once comfortable, and then go with all other modules one by one. Keep checkin the changes into new repository. In addition, you need to make sure no regressions with ported changes from this new repository by running sanity tests on the newer builds. Also, the validation should include all the supported platforms with Python 3.

Python 3 Ported Code and Status

Below is the new repository for Python 3 ported code until it is merged to the main repository. The plan is to do one cycle of porting or intermediately take the changes from main repo and do manual merge to this.

https://github.com/couchbaselabs/testrunner-py3/

(Branch: master)

Many common changes were already done but not completed as there might be some other runtime issues. Fixes in common can also be regressed the earlier fixes because of assumptions on input value type conversions. There is still some more ported code to be validated with Python 3 and the effort is in still in progress.

Now, let me show you the common issues happened during the runtime validation. You can use this as a reference when you hit an issue to see if you are having the similar issue. You can apply the same solution and see if it works for you. Any new ideas, you can put in comments.

Common Runtime Problems

1. Problem(s):

• You might get some of the below TypeErrors during runtime like str instead of bytes and bytes instead of str
• Error#1. TypeError: can’t concat str to bytes
• Error#2. TypeError: must be str, not bytes

File "lib/mc_bin_client.py", line 53, in __init__    
if msg: supermsg += ":  " + str(msg)
TypeError: must be str, not bytes

File "lib/mc_bin_client.py", line 141, in _recvMsg    
response += data
TypeError: must be str, not bytes

File "lib/remote/remote_util.py", line 3038, in log_command_output    
if "Warning" in line and "hugepages" in line:
TypeError: a bytes-like object is required, not 'str'

File "lib/tasks/task.py", line 1167, in run_high_throughput_mode    
raise Exception(rv["err"])
Exception: a bytes-like object is required, not 'str'

File "lib/mc_bin_client.py", line 936, in _set_vbucket    
self.vbucketId = (((zlib.crc32(key)) >> 16) & 0x7fff) & (self.vbucket_count - 1)
TypeError: a bytes-like object is required, not 'str'


File "lib/mc_bin_client.py", line 148, in _recvMsg    
magic = struct.unpack(">B", response[0:1])[0]
TypeError: a bytes-like object is required, not 'str'

File "lib/remote/remote_util.py", line 4560, in check_cmd    
if out and command_output in out[0]:
TypeError: a bytes-like object is required, not 'str'

File "lib/mc_bin_client.py", line 126, in _sendMsg    
self.s.send(msg + extraHeader + key + val + extended_meta_data)
TypeError: can't concat str to bytes

File "/usr/lib64/python3.6/urllib/parse.py", line 120, in _coerce_args    
raise TypeError("Cannot mix str and non-str arguments")
TypeError: Cannot mix str and non-str arguments

if o[0] != b"":                    
   o = o[0].split(b" ")

 import traceback                  
 try:  

 ..                  

 except Exception as e:                     
   log.info("{}".format(e))                     
   traceback.print_exc()                     
   exc_type, exc_obj, exc_tb = sys.exc_info()                     
   fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]                     
   print(exc_type, fname, exc_tb.tb_lineno)

for line in output:   
	try:   
    	line=line.decode()   
    except AttributeError:   
        pass

File "lib/membase/api/rest_client.py", line 4178, in multiscan_count_for_gsi_index_with_rest    
content = content.split("[]")[0]
TypeError: a bytes-like object is required, not 'str'

content = content.split(b'[]')[0].decode()

AttributeError: type object 'XDCRAdvFilterTests' has no attribute 'suite_setUp'

11a12,18
> 
>     def suite_setUp(self):
>         print("*** XDCRAdvFilterTests : suite_Setup() ***")
> 
>     def suite_tearDown(self):
>         print("*** XDCRAdvFilterTests : suite_tearDown() ***")
>

File "./testrunner.py", line 416, in main    
result.errors = [(name, e.message)]
AttributeError: 'AttributeError' object has no attribute 'message'

result.errors = [(name, str(e))]

File "./testrunner.py", line 529, in main    
t._Thread__stop()
AttributeError: 'Transport' object has no attribute '_Thread__stop'

File "pytests/view/viewquerytests.py", line 45, in stop    
self._Thread__stop()
AttributeError: 'StoppableThread' object has no attribute '_Thread__stop'    

self._stop()
TypeError: 'Event' object is not callable

def buildGoodSet(goodChars=string.printable, badChar='?'):    
143     """Build a translation table that turns all characters not in goodChars    
144     to badChar"""    
145     allChars = '\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./0123        456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x9     3\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\x bb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\        xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff'    
146     badchars = str.maketrans(allChars, allChars, goodChars)    
147     badchars1=str.translate(allChars,badchars)    
148     rv = str.maketrans(badchars1, badChar * len(badchars1))    
149     return rv

File "pytests/security/audittest.py", line 396    
shell = RemoteMachineShellConnection(self.master)                                                    
^
TabError: inconsistent use of tabs and spaces in indentation

sed -i 's/\t/        /g' pytests/security/audittest.py

File "lib/couchbase_helper/documentgenerator.py", line 83, in __next__    
value = arg[seed % len(arg)]
TypeError: list indices must be integers or slices, not float

File "lib/membase/helper/bucket_helper.py", line 517, in load_some_data    
keys = ["key_%s_%d" % (testuuid, i) for i in range(number_of_buckets)]
TypeError: 'float' object cannot be interpreted as an integer 

File "lib/membase/helper/bucket_helper.py", line 372, in verify_data    
test.assertEqual(value, key, msg='values dont match')
AssertionError: b'key_d918f450-5858-4430-a016-230e1f45bcf9_0' != 'key_d918f450-5858-4430-a016-230e1f45bcf9_0' : values dont match

File "pytests/setgettests.py", line 90, in set_get_test    
self.test.fail("value mismatch for key {0}".format(key))
AssertionError: value mismatch for key 9fcbd36f-e34d-477b-9fc5-0a5d067dff4b

File "pytests/security/auditmain.py", line 320, in returnFieldsDef    
if (isinstance((particulars['mandatory_fields'][items.encode('utf-8')]), dict)):
KeyError: b'bucket_name'

File "lib/tasks/task.py", line 2370, in _check_ddoc_revision    
new_rev_id = self._parse_revision(meta['rev'])
KeyError: 'rev'

File "pytests/tuqquery/tuq_dml.py", line 455, in test_insert_with_select    
expected_result = sorted([{bucket.name: {'name': doc['name']}} for doc in values[:num_docs]])
TypeError: '<' not supported between instances of 'dict' and 'dict'

expected_result = sorted(expected_result,key=(lambda x: x[bucket.name]['name']))

File "pytests/tuqquery/tuq_2i_index.py", line 1057, in test_simple_array_index    
self.assertTrue(sorted(actual_result_within['results']) == sorted(expected_result['results']))
TypeError: '<' not supported between instances of 'dict' and 'dict'

-                self.assertTrue(sorted(actual_result_within['results']) == sorted(expected_result['results']))
+                self.assertTrue(sorted(actual_result_within['results'], key=(lambda x: x['name'])) == \
+                                sorted(expected_result['results'], key=(lambda x: x['name'])))

File "pytests/tuqquery/tuq.py", line 1221, in _verify_results    
self.fail("Results are incorrect.Actual num %s. Expected num: %s.\n" % (len(actual_result), len(expected_result)))
AssertionError: Results are incorrect.Actual num 0. Expected num: 72.

File "lib/tasks/task.py", line 3638, in filter_emitted_rows    
reverse=descending_set)
TypeError: 'cmp' is an invalid keyword argument for this function

expected_rows = sorted(self.emitted_rows, key=(lambda x: (x['key'],x['id'])),reverse=descending_set)

 File "lib/tasks/task.py", line 3675, in <listcomp>    
expected_rows = [row for row in expected_rows if row['key'] >= start_key and row['key'] <= end_key]
TypeError: '>=' not supported between instances of 'int' and 'NoneType'

hasattr(items,’iteritems’) doesn’t return true

@@ -754,7 +755,7 @@ class MemcachedClient(object):         
# If this is a dict, convert it to a pair generator         
collection = self.collection_name(collection) 
-        if hasattr(items, 'iteritems'):
+        if hasattr(items, 'items'):             
items = iter(items.items())

if hasattr(items, 'items'):

 File "lib/crc32.py", line 78, in crc32_hash    
crc = (crc >> 8) ^ crc32tab[int((crc ^ ord(ch)) & 0xff)]
TypeError: ord() expected string of length 1, but int found

try: 
  key = key.decode()
except 
  AttributeError: pass

File "lib/remote/remote_util.py", line 1714, in create_directory    
if e[0] == 2:
TypeError: 'FileNotFoundError' object is not subscriptable
-            if e[0] == 2:
+            if e.errno == 2:

Traceback (most recent call last):  
File "lib/couchbase_helper/tuq_helper.py", line 521, in run_query_and_verify_result    
self._verify_results(sorted_actual_result, sorted_expected_result)  
File "lib/couchbase_helper/tuq_helper.py", line 114, in _verify_results    
raise Exception(msg)
Exception: The number of rows match but the results mismatch, please check

File "scripts/populateIni.py", line 52, in main    
data[i] = string.replace(data[i], 'dynamic', servers[0])
AttributeError: module 'string' has no attribute 'replace'

data[i].replace( 'dynamic', servers[0])

TypeError: '>' not supported between instances of 'int' and 'str'

  if where_clause:
+            where_clause = where_clause.replace('if  t > "', 'if str(t) > "') # to fix the type error between int, str comparison

  File "lib/couchbase_helper/tuq_helper.py", line 782, in verify_indexes_redistributed    
if cmp(items_count_before_rebalance, items_count_after_rebalance) != 0:
NameError: name 'cmp' is not defined

Use deepdiff module and DeepDiff class to do object comparison.
-        if cmp(index_state_before_rebalance, index_state_after_rebalance) != 0:
+        if DeepDiff(index_state_before_rebalance, index_sFile "lib/couchbase_helper/documentgenerator.py", line 19, in has_next    
return self.itr < self.end
TypeError: '<' not supported between instances of 'int' and 'str'

return int(self.itr) < int(self.end)