Fix scanner force-scanning #511

uzlonewolf · 2024-06-23T06:37:29Z

The move to argparse in the last release broke the force-scan IP list.

jasonacox · 2024-06-23T20:50:18Z

tinytuya/scanner.py

@@ -164,6 +164,7 @@ def __init__( self, ip, deviceinfo, options, debug ):
        self.timeo = 0
        self.resets = 0
        self.step = FSCAN_NOT_STARTED
+        self.try_v35 = False


Hey @uzlonewolf is this really only v35 or v34_and_v35? I'm trying to follow the logic/flow it seems like both, but I'm probably missing something.

It's only looked at when the device version is set to 3.4, so I guess try_v35_with_v34 would be a more accurate name.

Got it! Yes, that makes sense. Thanks @uzlonewolf !

jasonacox · 2024-06-23T20:51:00Z

tinytuya/scanner.py

@@ -334,7 +342,7 @@ def timeout( self, forced=False ):
                print('ForceScannedDevice: Debug sock', self.ip, 'connect timed out!')
        elif self.step == FSCAN_INITIAL_CONNECT:
            if self.debug:
-                print('ForceScannedDevice: Debug sock', self.ip, 'socket send failed,', 'no data received' if forced else 'receive timed out')
+                print('ForceScannedDevice: Debug sock', self.ip, 'socket send failed,', 'no data received,' if forced else 'receive timed out,', 'current retry:', self.retries)


Good addition.

jasonacox · 2024-06-23T20:58:24Z

Not sure if we want to address this because it is correct, but I suspect some (ok, even me) will specify a host address with netmask:

$ python3 -m tinytuya scan -f 10.0.1.1/24                                                                        pr/511

TinyTuya (Tuya device scanner) [1.14.1]

[Loaded devices.json - 43 devices]

Scanning on UDP ports 6666 and 6667 and 7000 for devices for 18 seconds...

    Option: Network force scanning requested.


    Running Scan...
ERROR: Unable to get network for '10.0.1.1/24', ignoring.
Traceback (most recent call last):
  File "/Users/jason/Code/tinytuya/tinytuya/scanner.py", line 931, in _generate_ip
    network = ipaddress.ip_network(netblock)
  File "/Users/jason/miniforge3/lib/python3.10/ipaddress.py", line 74, in ip_network
    return IPv4Network(address, strict)
  File "/Users/jason/miniforge3/lib/python3.10/ipaddress.py", line 1507, in __init__
    raise ValueError('%s has host bits set' % self)
ValueError: 10.0.1.1/24 has host bits set

MarkCupitt · 2024-06-24T04:14:47Z

@jasonacox @uzlonewolf

Hi Guys, I grabbed server.py and scanner.py (I think it was only the two files) from @uzlonewolf's commits and ran them here, they did not seem to fix my issue, ie, did not see any v 3.5 devices, have attached log and devices file files for you, hope it helps

What I did notice last night I that 2 of the three devices that were not being seen by tinytuya eventually showed up on the webui, this morning, which was interesting.

tinytuya scan -d >debug.log 2>&1
tinytuya scan -f 192.168.2.0/24 >force.log 2>&1

debug.log
devices.json
force.log

jasonacox · 2024-06-24T04:20:46Z

tinytuya/core.py

@@ -1654,6 +1654,7 @@ def add_dps_to_request(self, dp_indicies):
            self.dps_to_request.update({str(index): None for index in dp_indicies})

    def set_version(self, version): # pylint: disable=W0621
+        version = float(version)


Good call! Yes, this surfaces as an issue every 3-6mo or so.

Yeah, I thought I'd sneak that one in there :) I'm not marking it as 'closes #507' as I think there's more we can do to type-safe everything.

uzlonewolf · 2024-06-24T04:21:23Z

@MarkCupitt that force.log is still using the old scanner.py, not this modified one, as shown by:

Traceback (most recent call last):
  File "/home/mark/.local/lib/python3.11/site-packages/tinytuya/scanner.py", line 902, in _generate_ip
    network = ipaddress.ip_network(netblock)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ipaddress.py", line 83, in ip_network
    raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 network')
ValueError: ['192.168.2.0/24'] does not appear to be an IPv4 or IPv6 network

Replace /home/mark/.local/lib/python3.11/site-packages/tinytuya/scanner.py and try it again.

jasonacox · 2024-06-24T04:30:34Z

@MarkCupitt if you want to pull the full repo down you could do this to test the PR:

# Clone
git clone https://github.com/jasonacox/tinytuya.git
cd tinytuya

# Grab PR
git pull origin pull/511/head

# Test
python3 -m tinytuya scan -f

MarkCupitt · 2024-06-24T05:09:03Z

@jasonacox @uzlonewolf sorry, my bad, I was using the wrong lib .. so per @jasonacox

From https://github.com/jasonacox/tinytuya
 * branch            refs/pull/511/head -> FETCH_HEAD
Already up to date.

python -m tinytuya scan -d >debug.log 2>&1
python -m tinytuya scan -f 192.168.2.0/24 >force.log 2>&1

So, Im seeing 3.5 Devices now in the Force scan .. fantastic .. but some still show off line in the server web and I dont see any 3.5 Devices in the Debug log. Im not sure whether they are not sending packets, or tinytuya is missing them

force.log
devices.json
debug.log

uzlonewolf · 2024-06-24T13:28:05Z

The Wireshark capture you posted to #510 show that the devices are not sending broadcasts. Since the devices are not sending packets there is nothing for TinyTuya to receive.

As the device in the Wireshark capture immediately tries to find your phone after it is rebooted, I suspect Smart Life is still running in the background on your phone.

jasonacox · 2024-06-24T23:23:03Z

Try to stop the SmartLife app and see if the devices start broadcasting.

MarkCupitt · 2024-06-25T03:03:29Z

@jasonacox @uzlonewolf Yep, I actually had disabled wifi for my phone with Smartlife on it, restarted all the device.

The pull request does address the 3.5 issue which was the root cause of my issue, and I accept that the devices are "different" and do something different with Broadcast packets.

Ill adjust my use case to work with this approach as I will know the IP addresses and can access them directly. Im comfortable with that, so I would say the pR is Ok

Thanks for the prompt fixes and responses, they are very much appreciated

MarkCupitt · 2024-06-25T03:04:19Z

I'm going to call this issue successfully closed .. as any remaining issues appear to be device-related. Thanks again ..

jasonacox · 2024-06-25T03:16:17Z

I sure hope this isn't the start of a new device series/trend that don't broadcast discovery packets.

Thanks for your help with this, @MarkCupitt !

uzlonewolf · 2024-06-25T03:16:51Z

@jasonacox Do you want to add a "Start Force-scan" button to the server?

jasonacox · 2024-06-25T03:18:12Z

@jasonacox Do you want to add a "Start Force-scan" button to the server?

Oh, that's a great idea. Hum. Let me poke at that tonight. Might be worth including in this PR.

uzlonewolf · 2024-06-25T03:19:58Z

I also fixed the "ValueError: 10.0.1.1/24 has host bits set" thing but it looks like I forgot to push it. 1 sec

jasonacox · 2024-06-25T03:27:24Z

tinytuya/scanner.py

@@ -928,7 +928,7 @@ def _generate_ip(networks, verbose, term):
        if tinytuya.IS_PY2 and type(netblock) == str:
            netblock = netblock.decode('latin1')
        try:
-            network = ipaddress.ip_network(netblock)
+            network = ipaddress.ip_network(netblock, strict=False)


So easy. ;-) thanks! 🙏

MarkCupitt · 2024-06-25T04:42:48Z

@jasonacox @uzlonewolf

As a humble suggestion, If you had a force scan update snapshot.json, perhaps add a key to the JSON like this:

poll_method: "broadcast"
or
poll_method: "poll"

with broadcast being the default, any other code that uses snapshot.json would know that the device was there and that it needed to poll the device and not wait for a broadcast.

Secondly, adding a timestamp recording the time it was last successfully accessed would allow for a last accessed age, thus allowing logic for retries on individual devices, should it pass a threshold, or then showing them as going offline. Because of high disk io, if running this on a pi, it may be better to keep this in memory and just serve an exact copy of snapshot.json out an API, for use in other systems.

Lastly, if you were to add an optional new file, say "local_ips.json` with something like this

[{
    "deviceid": "eb6cfead8a7ce1de12wvaf",
    "ip": "192.168.2.x",
    "key": "akdfjf",
    "version": "3.5"
  }]

You could build in a poll (on a timer) like monitor.py into the scan only for non-broadcast devices shown in snapshot.json, and be able to handle most edge cases I would think

Updating the UI to indicate if it is a broadcast or poll and the age [ eg: formatted DateTime (12 seconds ago) ] would also be very useful

Lastly, a wishlist request, In thee UI, update the Offline devices from info in Devices.json so they are the same display as the Online Table display.

This covers pretty much all the UI and oddball devices I can see from where I sit and may make Tinytuya more flexible

I'm quite happy to contribute, but Python is not my first language, but it is 15 years old, however feel free to ask, dev, testing etc

If you do agree to the above, I would be happy to write a separate MQTT server that uses the above (basically reads snapshot.json, from server.py) to publish values to MQTT on a regular basis, my feeling is that this is best left out of server.py, as its use case is very different, although I see there has been some thought on MQTT already, happy to contribute to that .. if you deem that a better approach

The end result would be a pretty cool setup where a process server.py was running in the background keeping snapshot.json up to date, then a second process that read snapshot.json, ( looked for file changes and reload would be nice) and polled the devices (threaded or asyncio) for status and emitted MQTT publish events

added #512 as an idea

…into pr/511

jasonacox · 2024-06-25T05:14:09Z

tinytuya/scanner.py

-def _print_device_info( result, note, term, extra_message=None ):
+def _print_device_info( result, note, term, extra_message=None, verbose=True ):
+    if not verbose:
+        return


I noticed while testing that verbose=False was not honored.

scanner.devices(verbose=False, color=False, poll=False, forcescan="10.0.0.0/24",discover=False)

Printed all the devices. I updated scanner to ignore prints if verbose=False.

…into pr/511

jasonacox · 2024-07-07T18:34:46Z

I added a "Force Scan" to the server which causes it to launch a scanner.devices(forcescan=True, ...) call. I have the 3.5 Smartbulb on my network and force scan finds it, but not via discover (as we already found out). I will add the 3.5 broadcast packet (7000) to the server once we have it in the scanner.

…rcescan-fix

jasonacox · 2024-07-07T19:41:22Z

tinytuya/scanner.py

+    for bcast in client_bcast_addrs:
+        addr = client_bcast_addrs[bcast]
+        client_ip_broadcast_list[addr] = { 'broadcast': bcast }
+


Would it be simpler to remove this logic and have send_discovery_request() build the broadcast list itself (assume we always want it to broadcast to all interfaces)? Or do you think we will want to alter / prune the list it would use?

Either is fine... just selfishly wondering if I could just call scanner.send_discovery_request() without worrying about sending it any arguments. 😋

Yep, 2 reasons: so we only need to build the interface list/encrypt the payload/open the socket once, and so we can expand the list later if needed (i.e. we're using the single getmyIP() address and want to add any address we receive a broadcast on).

It shouldn't be hard to make the send_discovery_request() argument optional.

…ilable

uzlonewolf · 2024-07-07T20:51:06Z

Ok, I think that's everything on my list for now.

jasonacox · 2024-07-07T23:21:38Z

tinytuya/Cloud.py

+        Args: 
+            verbose - Returns raw JSON data from Tuya Cloud
+            oldlist - List of devices from previous run
+            include_map - Include the DPS mapping in the device list


I discovered I needed this better description of the args while adding the cloud sync button to the proxy. :-)

jasonacox · 2024-07-08T01:23:08Z

LGTM - I'm going to do a few final tests and then merge and release v1.14.1.

jasonacox · 2024-07-08T05:37:03Z

@uzlonewolf We may have a problem. I just tested on a system without netifaces or psutil and I'm getting an error for the basic scan. After installing psutil, it works without any issue. We should add psutil to the setup requirements. I'll tests on a few other OS variants to make sure that works.

$ python3 -m tinytuya scan

TinyTuya (Tuya device scanner) [1.14.1]

[Loaded devices.json - 43 devices]

Scanning on UDP ports 6666 and 6667 and 7000 for devices for 18 seconds...

Traceback (most recent call last):                 
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/jason/.local/lib/python3.10/site-packages/tinytuya/__main__.py", line 124, in <module>
    scanner.scan( scantime=args.max_time, color=(not args.nocolor), forcescan=args.force, discover=(not args.no_broadcasts), assume_yes=args.yes )
  File "/home/jason/.local/lib/python3.10/site-packages/tinytuya/scanner.py", line 1027, in scan
    devices(verbose=True, scantime=scantime, color=color, poll=True, forcescan=forcescan, discover=discover, assume_yes=assume_yes)
  File "/home/jason/.local/lib/python3.10/site-packages/tinytuya/scanner.py", line 1636, in devices
    send_discovery_request( client_ip_broadcast_list )
  File "/home/jason/.local/lib/python3.10/site-packages/tinytuya/scanner.py", line 218, in send_discovery_request
    iface['socket'].bind( (address,0) )
socket.gaierror: [Errno -5] No address associated with hostname


$ python3 -m tinytuya scan -f

TinyTuya (Tuya device scanner) [1.14.1]

[Loaded devices.json - 43 devices]

Scanning on UDP ports 6666 and 6667 and 7000 for devices for 18 seconds...

    Option: Network force scanning requested.

    NOTE: neither module netifaces nor module psutil are available, multi-interface machines will be limited.
           (Requires: `pip install netifaces` or `pip install psutil`)


    Running Scan...
Traceback (most recent call last):                      
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/jason/.local/lib/python3.10/site-packages/tinytuya/__main__.py", line 124, in <module>
    scanner.scan( scantime=args.max_time, color=(not args.nocolor), forcescan=args.force, discover=(not args.no_broadcasts), assume_yes=args.yes )
  File "/home/jason/.local/lib/python3.10/site-packages/tinytuya/scanner.py", line 1027, in scan
    devices(verbose=True, scantime=scantime, color=color, poll=True, forcescan=forcescan, discover=discover, assume_yes=assume_yes)
  File "/home/jason/.local/lib/python3.10/site-packages/tinytuya/scanner.py", line 1636, in devices
    send_discovery_request( client_ip_broadcast_list )
  File "/home/jason/.local/lib/python3.10/site-packages/tinytuya/scanner.py", line 218, in send_discovery_request
    iface['socket'].bind( (address,0) )
socket.gaierror: [Errno -5] No address associated with hostname

jasonacox · 2024-07-08T05:47:48Z

Trying:

pip3 install tinytuya==1.15.0.dev0

installs psutil as a dependency.

jasonacox · 2024-07-08T06:13:21Z

So it seems that pstuil doesn't work well on Windows. I switched to netifaces.

pip3 install tinytuya==1.15.0.dev1

Successful tests on MacOS, Linux, RPi and Windows.

jasonacox · 2024-07-08T07:17:59Z

Library: https://pypi.org/project/tinytuya/1.15.0

Server (in a can): jasonacox/tinytuya:1.15.0p13 - Run via docker, where HOST is the IP address of the host running the server (required to prevent force scan from scanning the /8 docker IP space :)

docker run \
    -d \
    -p 8888:8888 \
    -p 6666:6666/udp \
    -p 6667:6667/udp \
    -p 7000:7000/udp \
    --network host \
    -e DEBUGMODE='no' \
    -e HOST='192.168.0.100' \
    -v $PWD/devices.json:/app/devices.json \
    -v $PWD/tinytuya.json:/app/tinytuya.json \
    --name tinytuya \
    --restart unless-stopped \
    jasonacox/tinytuya

uzlonewolf · 2024-07-08T18:34:37Z

socket.gaierror: [Errno -5] No address associated with hostname

Fixed in #519 . getmyIP() actually returned a network (i.e. "192.168.1.0/24") and not an IP, so I added getmyIPaddr() which returns the actual IP.

jasonacox · 2024-07-09T05:00:11Z

Nice! I'm wondering if we should reverse the dependency of netifaces. I tend to favor that to keep it clean, but having it install by default does give more functionality. Thoughts?

uzlonewolf · 2024-07-09T22:05:16Z

I keep waffling on the netifaces dependency. On one hand it's practically a requirement for multi-interface machines, but on the other how many of our users are actually using multi-interface machines? I could go either way on it.

Fix scanner force-scanning

d3be0ac

uzlonewolf marked this pull request as ready for review June 23, 2024 06:37

uzlonewolf and others added 4 commits June 22, 2024 23:46

Allow a string as the force-scan list

d3bc3d0

Add force-scanning v3.5 devices to scanner

d088823

Add force-scanning v3.5 devices to scanner

e9fd881

v1.14.2 Notes

0690e8c

jasonacox reviewed Jun 23, 2024

View reviewed changes

uzlonewolf added 2 commits June 23, 2024 21:11

Rename scanner variable try_v35 to a more descriptive try_v35_with_v34

7259b8d

Make sure set_version() is given a float (jasonacox#507)

9420ec6

jasonacox reviewed Jun 24, 2024

View reviewed changes

Allow host bits in the force-scan network list

133fd35

jasonacox reviewed Jun 25, 2024

View reviewed changes

jasonacox added 2 commits June 25, 2024 01:10

Refactor _print_device_info function to include verbose flag

da6d01b

Merge branch 'forcescan-fix' of https://github.com/uzlonewolf/tinytuya …

51f7653

…into pr/511

jasonacox reviewed Jun 25, 2024

View reviewed changes

uzlonewolf mentioned this pull request Jun 26, 2024

Cannot connect to Smart Switch #507

Open

uzlonewolf and others added 4 commits July 7, 2024 09:25

Allow device discovery packets on port 7000

7d160a0

Add get_ip_to_broadcast() function to scanner

ff6abf2

Add "Force Scan" button and UI updates

5f38b49

Merge branch 'forcescan-fix' of https://github.com/uzlonewolf/tinytuya …

59a013f

…into pr/511

uzlonewolf and others added 5 commits July 7, 2024 12:15

Make scanner send broadcasts to port 7000

831f113

Update control panel image

4d46538

Add port 7000 support to tools/pcap_parse.py

6073a8c

Merge branch 'forcescan-fix' of github-lw:uzlonewolf/tinytuya into fo…

3610184

…rcescan-fix

Fix incorrect message log level

a20b080

jasonacox reviewed Jul 7, 2024

View reviewed changes

uzlonewolf added 3 commits July 7, 2024 13:13

Allow calling send_discovery_request() without an argument

2fbb691

Rewrite tinytuya.find_device() to use the scanner

0b19990

Also use psutil to get the force-scan IP list if netifaces is not ava…

c6dba30

…ilable

uzlonewolf and others added 2 commits July 7, 2024 16:10

Close the broadcast sockets once we're done with them

b00882a

Add broadcast for 3.5 devices to server

728779d

jasonacox reviewed Jul 7, 2024

View reviewed changes

Release notes

3aea261

jasonacox merged commit 828d762 into jasonacox:master Jul 8, 2024
16 checks passed

Fix scanner force-scanning #511

Fix scanner force-scanning #511

Conversation

uzlonewolf commented Jun 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasonacox commented Jun 23, 2024

MarkCupitt commented Jun 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uzlonewolf commented Jun 24, 2024

jasonacox commented Jun 24, 2024

MarkCupitt commented Jun 24, 2024 • edited Loading

uzlonewolf commented Jun 24, 2024

jasonacox commented Jun 24, 2024

MarkCupitt commented Jun 25, 2024

MarkCupitt commented Jun 25, 2024

jasonacox commented Jun 25, 2024

uzlonewolf commented Jun 25, 2024

jasonacox commented Jun 25, 2024

uzlonewolf commented Jun 25, 2024

Choose a reason for hiding this comment

MarkCupitt commented Jun 25, 2024 • edited Loading

Choose a reason for hiding this comment

jasonacox commented Jul 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uzlonewolf commented Jul 7, 2024

Choose a reason for hiding this comment

jasonacox commented Jul 8, 2024

jasonacox commented Jul 8, 2024

jasonacox commented Jul 8, 2024 • edited Loading

jasonacox commented Jul 8, 2024

jasonacox commented Jul 8, 2024

uzlonewolf commented Jul 8, 2024

jasonacox commented Jul 9, 2024

uzlonewolf commented Jul 9, 2024

MarkCupitt commented Jun 24, 2024 •

edited

Loading

MarkCupitt commented Jun 25, 2024 •

edited

Loading

jasonacox commented Jul 7, 2024 •

edited

Loading

jasonacox commented Jul 8, 2024 •

edited

Loading