Porting 3.5 into tuyapi

[nospam2k]

I am trying to implement 3.5 protocol into tuyapi. I have created code to make a connection and decode the return (I believe). I will include the code just for visual. If you could cobble together a python quick test of connecting to a device, getting status, and turning it off or on, without using the full tinytuya library (like you did in your 3.5 notes for the first test) I would really appreciate it. I know this isn't technically an issue, but I would really appreciate the help. I hate to see tuyapi die when it only needs to update to 3.5. Here is what I have so far:

const net = require('net');
const crypto = require('crypto');

const ip = '<ip>'; // Add the correct IP address
const key = Buffer.from('<local key>', 'utf-8'); // Add local key
var stime;

for (let i = 0; i < 2; i++) {
    const client = new net.Socket();

    client.setTimeout(5000);

    client.connect(6668, ip, () => {
        console.log('connected!');

        stime = Date.now();
        const localNonce = Buffer.from('0123456789abcdef', 'utf-8'); // not-so-random random key
        const localIV = localNonce.slice(0, 12); // not-so-random random iv

        const pkt = Buffer.alloc(18);
        pkt.writeUInt32BE(0x6699, 0);
        pkt.writeUInt16BE(0, 4);
        pkt.writeUInt32BE(1, 6);
        pkt.writeUInt32BE(3, 10);
        pkt.writeUInt32BE(localNonce.length + localIV.length + 16, 14);

        const cipher = crypto.createCipheriv('aes-128-gcm', key, localIV);
        cipher.setAAD(pkt.slice(4));
        const encrypted = Buffer.concat([cipher.update(localNonce), cipher.final()]);
        const tag = cipher.getAuthTag();

        const message = Buffer.concat([pkt, localIV, encrypted, tag, Buffer.from([0x00, 0x00, 0x99, 0x66])]);

        client.write(message);
    });

    client.on('data', (data) => {
        //console.log('data:', data.toString('hex'));
        //console.log('prefix:', data.slice(0, 4).toString('hex'));
        //console.log('unknown:', data.slice(4, 6).toString('hex'));
        //console.log('sequence:', data.slice(6, 10).toString('hex'));
        //console.log('command:', data.slice(10, 14).toString('hex'));
        //console.log('length:', data.slice(14, 18).toString('hex'));
        //console.log('iv:', data.slice(18, 30).toString('hex'));
        //console.log('payload:', data.slice(18 + 12, plen + 18 - 16).toString('hex'));
        //console.log('tag:', data.slice(plen + 18 - 16, plen + 18 - 16 + 16).toString('hex'));
        //console.log('footer:', data.slice(plen + 18).toString('hex'));
        const header = data.slice(4, 18)
        const prefix = data.slice(0, 4);
        const unknown = data.slice(4, 6);
        const sequence = data.slice(6, 10);
        const command = data.slice(10, 14);
        const length = data.slice(14, 18);
        const iv = data.slice(18, 30);
        const plen = parseInt(data.slice(14, 18).toString('hex'), 16);
        const payload = data.slice(18 + 12, plen + 2); // 18 - 16
        const tag = data.slice(plen + 2, plen + 18); // 18 - 16
        const footer = data.slice(plen + 18);

        client.destroy();

        const decipher = crypto.createDecipheriv('aes-256-gcm', key.toString('hex'), iv.toString('hex'));
        decipher.setAAD(header);
        decipher.setAuthTag(tag);
        let raw = decipher.update(payload, 'binary', 'utf-8');
        console.log(raw);
    });

    client.on('timeout', () => {
        console.log('socket timeout');
        client.destroy();
    });

    client.on('error', (err) => {
        console.log('socket error:', err.message);
        client.destroy();
    });

    client.on('close', () => {
        console.log('connection closed');
    });
}

What I get back:

connected!
connected!
�����(\<}�5W(�M�'��W�y�� ��}���z�_�
                                   �d��v{ey�f
connection closed

�?�3&3݄~>�O8����놡�Ý*����7�:�����:��J��{4
connection closed

[uzlonewolf]

Yeah, that is decidedly non-trivial. v3.5 devices require a 3-way handshake to negotiate the session key before status can be queried or commands sent. To do this without using any of the existing functions is going to require rewriting the packet assembler, packet parser, and crypto routines and will most likely be at least a couple hundred lines of code. I think it would be more productive for both of us if I were to just write this example in javascript.

[nospam2k]

Yeah, that is decidedly non-trivial. v3.5 devices require a 3-way handshake to negotiate the session key before status can be queried or commands sent. To do this without using any of the existing functions is going to require rewriting the packet assembler, packet parser, and crypto routines and will most likely be at least a couple hundred lines of code. I think it would be more productive for both of us if I were to just write this example in javascript.

@uzlonewolf - If I had an example of negotiate key, query, turn on, turn off for bulb, I'd be happy to put in the work to implement into tuyapi. I'm starting to do that now and I've had some minor success, but an example would be great. Tuyapi support 3.4 so there aren't a whole lot of changes but the key negotiation is where I'm stuck right now. Thanks for the help!

[uzlonewolf]

How long does it wait before the connection closes? It's been a while since I looked at it, but IIRC the device does not send a response to the final sequence.

That said, your sessionKey calculation is wrong; it should be localNonce ^ deviceNonce (not client.key).

[nospam2k]

@uzlonewolf - Thank you!!! I really appreciate your efforts. I will go through your code and try to implement it for tuyapi!!!

I'm very happy right now!!!

[uzlonewolf]

A few more updates and turn on / turn off implemented:

const net = require('net');
const crypto = require('crypto');

function sleep(ms) {
  return new Promise((resolve) => {
    setTimeout(resolve, ms);
  });
}

class PacketBuilder
{
    #seq = 1;
    #buf;
    key;

    constructor(key)
    {
	this.key = Buffer.from(key);
	this.#buf = Buffer.alloc(18);
	this.#buf.writeUInt32BE(0x6699, 0); // Prefix
	this.#buf.writeUInt16BE(0, 4); // Unknown
    }

    setKey(key)
    {
	this.key = Buffer.from(key);
    }

    build(cmd, data)
    {
	if(cmd == 0x0d)
	{
	    // Prepend 'version header' to the data
	    data = '3.5\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' + data;
	}

	this.#buf.writeUInt32BE(this.#seq, 6); // Sequence
	this.#buf.writeUInt32BE(cmd, 10); // Command
	this.#buf.writeUInt32BE(12 + data.length + 16, 14); // IV length + Data length + Tag length
	this.#seq++;

	let localIV = Buffer.from((Date.now() * 10).toString().substring(0, 12));
	console.log('localIV', localIV.toString('hex'));

	// Encrypt payload
	let cipher = crypto.createCipheriv('aes-128-gcm', this.key, localIV);
	cipher.setAAD(this.#buf.slice(4, 18));
        let encrypted = Buffer.concat([cipher.update(data), cipher.final()]);

	// Concat message
	let enc = Buffer.concat([this.#buf, localIV, encrypted, cipher.getAuthTag(), Buffer.from([0x00, 0x00, 0x99, 0x66])]);
	console.log('command:', cmd.toString(16), data);
	console.log('encrypted packet:', enc.toString('hex'));
	return enc;
    }
}

class PacketParser
{
    constructor(data, key)
    {
	// Create prefix var
	this.prefix = data.slice(0, 4);
	console.log('prefix:', this.prefix.toString('hex'));

	// Create header var
	this.header = data.slice(4, 18);
	//console.log('header', this.header.toString('hex'));

	// Create unknown var
	this.unknown = data.slice(4, 6);
	console.log('unknown:', this.unknown.toString('hex'));

	// Create sequence var
	this.sequence = data.slice(6, 10);
	console.log('sequence:', this.sequence.toString('hex'));

	// Create command var
	this.command = data.slice(10, 14);
	console.log('command:', this.command.toString('hex'));

	// Create per-packet remoteIV var
	this.remoteIV = data.slice(18, 30);
	console.log('remoteIV:', this.remoteIV.toString('hex'));

	// Create length var
	this.len = parseInt(data.slice(14, 18).toString('hex'), 16);
	console.log('len:', this.len);

	// Create payload var
	this.payload = data.slice(18 + 12, this.len + 2); // 18 - 16
	console.log('payload:', this.payload.toString('hex'));

	// Update tag from packet
	this.tag = data.slice(this.len + 2, this.len + 18); // 18 - 16
	console.log('tag:', this.tag.toString('hex'));

	// Create footer
	this.footer = data.slice(this.len + 18);
	console.log('footer:', this.footer.toString('hex'));

	// Decrypt payload
	let decipher = crypto.createDecipheriv('aes-128-gcm', key, this.remoteIV);
	decipher.setAAD(this.header);
	decipher.setAuthTag(this.tag);
	let decrypt = Buffer.concat([decipher.update(this.payload), decipher.final()]);

	this.retcode = decrypt.slice(0, 4);
	this.decrypt = decrypt.slice(4); // Remove return code
    }
}

class SocketClient
{
    constructor(ip, port)
  {
    this.ip = ip;
    this.port = port;
    this.client = new net.Socket();
    this.connected = false;

    this.client.on('data', (data) => this.handleData(data));
    this.client.on('timeout', () =>
    {
      console.log('socket timeout');
      this.client.destroy();
    });
    this.client.on('error', (err) =>
    {
      console.log('socket error:', err.message);
      this.client.destroy();
    });
    this.client.on('close', () =>
    {
      console.log('connection closed');
      this.connected = false;
    });
  }

  connect()
  {
    return new Promise((resolve, reject) =>
    {
      this.client.connect(this.port, this.ip, () =>
      {
        console.log('connected!');
        this.connected = true;
        resolve();
      });
    });
  }

    write(message)
    {
	this.client.write(message);
    }

  send(message)
  {
    return new Promise((resolve, reject) =>
    {
      this.client.write(message, (err) =>
      {
        if (err) {
            reject(err);
        } else {
            this.resolveResponse = resolve;
        }
      });
    });
  }

    recv()
    {
	return new Promise((resolve) =>
	    {
		this.resolveResponse = resolve;
	    });
    }


  handleData(data)
  {
    // Process the data as needed, e.g., decrypt, parse, etc.
    if (this.resolveResponse)
    {
      this.resolveResponse(data);
      this.resolveResponse = null;
    }
  }

    close()
    {
	this.client.end();
    }
}

(async () =>
{
    /****** Create client, connect and send first sequence packet ******/
    const client = new SocketClient(
	'172.20.10.136',
	6668
    );
    await client.connect();

    let devKey = "<snip>";

    // Session key neg start
    let localSesKey = Buffer.from('0123456789abcdef', 'utf-8');

    // Get a PacketBuilder
    let pkt = new PacketBuilder(devKey);

    // Build the message
    let message = pkt.build(3, localSesKey);

    // Send first sequence
    console.log('Sending:', message.toString('hex') + '\n');
    let data = await client.send(message);


    /******* Receive sequence response packet *******/
    console.log('received:', data.toString('hex'));


    let decrypt = new PacketParser(data, pkt.key);

    // Get deviceSesKey
    let deviceSesKey = Buffer.from(decrypt.decrypt).slice(0, 16);
    console.log('deviceSesKey', deviceSesKey.toString('hex'));

    // Get hmac
    let hmac = Buffer.from(decrypt.decrypt).slice(16);
    console.log('hmac', hmac.toString('hex'));

    //Create hmac for compare
    let calcLocalHmac = crypto.createHmac('sha256', pkt.key).update(localSesKey, 'utf8').digest();
    console.log('calcnonce', calcLocalHmac.toString('hex'));

    // Check hmacs match
    if(calcLocalHmac.toString('hex') === hmac.toString('hex')) console.log('nonces match'); else console.log('nonces FAILED!');

    // Calculate the HMAC of the device session key, and send it
    let deviceSesKeyHmac = crypto.createHmac('sha256', pkt.key).update(deviceSesKey, 'utf8').digest();
    message = pkt.build(5, deviceSesKeyHmac);
    client.write(message);

    // Calculate session encryption key and store it in deviceSesKey
    for (let i = 0; i < localSesKey.length; i++) {
	deviceSesKey[i] ^= localSesKey[i];
    }

    let cipher = crypto.createCipheriv('aes-128-gcm', pkt.key, localSesKey.slice(0, 12));
    deviceSesKey = Buffer.concat([cipher.update(deviceSesKey), cipher.final()]);

    console.log('final session key:', deviceSesKey.toString('hex'));

    // Update encryption key
    pkt.key = deviceSesKey;

    // Query device status
    message = pkt.build(0x10, '{}');
    data = await client.send(message);

    // Print the result
    console.log('received packet:', data.toString('hex') + '\n');
    decrypt = new PacketParser(data, pkt.key);
    console.log('received:', decrypt);
    console.log('received payload:', decrypt.decrypt.toString('utf8'));


    // Turn On (power is DP 20 for this bulb)
    message = pkt.build(0x0d, '{"protocol":5,"t":' + Date.now().toString() + ',"data":{"dps":{"20":true}}}');
    data = await client.send(message);

    // Print the new DP status
    console.log('received packet:', data.toString('hex') + '\n');
    decrypt = new PacketParser(data, pkt.key);
    console.log('received:', decrypt);
    console.log('received payload:', decrypt.decrypt.toString('utf8'));

    data = await client.recv();
    // Print the DP Set result
    console.log('received packet:', data.toString('hex') + '\n');
    decrypt = new PacketParser(data, pkt.key);
    console.log('received:', decrypt);
    console.log('received payload:', decrypt.decrypt.toString('utf8'));




    await sleep(3000);



    // Turn Off (power is DP 20 for this bulb)
    message = pkt.build(0x0d, '{"protocol":5,"t":' + Date.now().toString() + ',"data":{"dps":{"20":false}}}');
    data = await client.send(message);

    // Print the new DP status
    console.log('received packet:', data.toString('hex') + '\n');
    decrypt = new PacketParser(data, pkt.key);
    console.log('received:', decrypt);
    console.log('received payload:', decrypt.decrypt.toString('utf8'));

    data = await client.recv();
    // Print the DP Set result
    console.log('received packet:', data.toString('hex') + '\n');
    decrypt = new PacketParser(data, pkt.key);
    console.log('received:', decrypt);
    console.log('received payload:', decrypt.decrypt.toString('utf8'));


    client.close();
})();

[nospam2k]

So, so, so happy ;) Many thanks again. I've been working on this non stop for days!

[nospam2k]

@uzlonewolf Ok, so I've been able to implement 3.5 in tuyapi and put in PR's (with yours and Apollon77's help). I've got my head around 3.5.

Now I'm trying to work out how to handle a 3.4 device (device is returning 55AA using tinytuya) that is a sub device on a 3.3 version gateway. If I could trouble you (if you have time) for a code example like you did for 3.5 above (with the turn on, turn off) I'd really appreciate it.

[nospam2k]

Is cids [] acceptable? Here is my code:

const TuyAPI = require('tuyapi');
const fs = require('fs');
const path = require('path');

let endpoints = JSON.parse(fs.readFileSync(path.join(__dirname, 'endpoints.json')));

let gws = {};

// Loop through 40 devices and group them by gateway IP
for (let i = 0; i < 40; i++) {
  let devName = Object.keys(endpoints)[i];
  let {
    tuya_gwid: gwId,
    tuya_ip: gwIp,
    tuya_key: gwKey,
    tuya_cid: devCid,
    tuya_version: devVer
  } = Object.values(endpoints)[i];

  //console.log(`Processing device: ${devName}`);
  //console.log(`GW ID: ${gwId}, GW IP: ${gwIp}, GW Key: ${gwKey}, Dev CID: ${devCid}`);

  if (!gws[gwIp]) {
    gws[gwIp] = {
      gw: new TuyAPI({gwId: gwId, ip: gwIp, key: gwKey, version: '3.3'}),
      devCid: []
    };
  }

  if (!gws[gwIp].devCid.includes(devCid)) {
    gws[gwIp].devCid.push(devCid);
  }
}

//console.log(JSON.stringify(gws, null, 2));

(async () => {

  for (let gwIp in gws) {
    let { gw, devCid } = gws[gwIp];

    console.log(gw);
    
    await gw.connect();
    console.log('Connected');
    
    let devCidString = devCid.map(cid => JSON.stringify(cid)).join(', ');

    gw.set({shouldWaitForResponse: false, multiple: true, cid: [' + devCidString + '], data: {"1": true}});
  }
})();

I'm getting an error:

Connected
node:internal/errors:496
    ErrorCaptureStackTrace(err);
    ^

Error [ERR_UNHANDLED_ERROR]: Unhandled error. ('Timeout waiting for status response from device id: undefined')
    at new NodeError (node:internal/errors:405:5)
    at TuyaDevice.emit (node:events:506:17)
    at /home/church/Src/node/tuyapi/node_modules/tuyapi/index.js:434:12
    at Timeout._onTimeout (/home/church/Src/node/tuyapi/node_modules/p-timeout/index.js:25:13)
    at listOnTimeout (node:internal/timers:569:17)
    at process.processTimers (node:internal/timers:512:7) {
  code: 'ERR_UNHANDLED_ERROR',
  context: 'Timeout waiting for status response from device id: undefined'
}

Device dump:

TuyaDevice {
  _events: [Object: null prototype] {},
  _eventsCount: 0,
  _maxListeners: undefined,
  device: {
    ip: '192.168.0.23',
    port: 6668,
    id: undefined,
    gwID: undefined,
    key: 'xxxxxxxxxxxxxxx,
    productKey: undefined,
    version: '3.3',
    parser: MessageParser {
      version: '3.3',
      cipher: [TuyaCipher],
      key: 'xxxxxxxxxxx'
    }
  },

[uzlonewolf]

No, turning it into one big comma-delimited string will not work, it needs to be a proper array. Using the above code you should be able to just:

    gw.set({shouldWaitForResponse: false, multiple: true, cid: devCid, data: {"1": true}});

If that doesn't work then I'm going to need the Cloud logs or a local packet capture.

[nospam2k]

I'm getting the same error after making the change as you suggested.

node:internal/errors:496
    ErrorCaptureStackTrace(err);
    ^

Error [ERR_UNHANDLED_ERROR]: Unhandled error. ('Timeout waiting for status response from device id: undefined')
    at new NodeError (node:internal/errors:405:5)
    at TuyaDevice.emit (node:events:506:17)
    at /home/church/src/node/tuyapi/node_modules/tuyapi/index.js:434:12
    at Timeout._onTimeout (/home/church/src/node/tuyapi/node_modules/p-timeout/index.js:25:13)
    at listOnTimeout (node:internal/timers:569:17)
    at process.processTimers (node:internal/timers:512:7) {
  code: 'ERR_UNHANDLED_ERROR',
  context: 'Timeout waiting for status response from device id: undefined'
}

[uzlonewolf]

In that case I'm going to need a local packet capture of the group command.

[nospam2k]

I was able to work out the 3.3 protocol so I won't need that. It seems pretty clear that using an object for cid isn't allowed. Also tried just putting cid: '0028', cid: '0038' but just times out.

[nospam2k]

@uzlonewolf - I've had a thought about groups I thought I would bounce off of you. Groups have their own key and id's. I tried different combinations with no success but I think it's because I need to provide a parameter like id, cid, gwId... I have a feeling this may work.

[nospam2k]

D00D! Thx. I just did a test and mbid is the local_id for the group and it doesn't matter which cid so long as it's one of the devices in the group. Also, it isn't specific to the ip of the gateway (any gateway ip seems to work to send to all group devices in all gateways), it will send to all gateways! I haven't extensively tested this yet, but I will. Awesome stuff @uzlonewolf!! I appreciate your perseverance. If you can teach me how you were able to capture the packets, that would be awesome!

[nospam2k]

Ok, so now, how to send a group command for non-sub devices. I have a bunch of 3.5 bulbs that I have made a group and need to send a group command. I noticed there is no local_id for this type of group. This would be a good time to learn how to see packets! Thanks for all you help!

[uzlonewolf]

Sorry, I've been really busy this week.

Capturing traffic between a phone and a device is tricky due to how WiFi works. The 2 methods I've used are:

1. Have the phone connected to a different access point (AP) than the device, and bridge them using a router you can capture traffic on.

2. My current setup, use an AP you can capture traffic on directly. I have a fleet of Ubiquity's Unifi APs, and with them you just SSH in and run tcpdump on the appropriate physical interface, i.e. tcpdump -p -n -X -s0 -i ath1 -w /tmp/btlog.pcap. Anything running OpenWRT or similar should work as well. I then just scp -O <ap>:/tmp/btlog.pcap ./ to copy the file to my local machine and call https://github.com/jasonacox/tinytuya/blob/master/tools/pcap_parse.py on it to decrypt the traffic.

I've never used them, but a couple other options may be:

3. Use an enterprise AP that allows sending all traffic to a router instead of relaying it directly.

4. Use a WiFi card that can capture raw frames and crack the per-client key. It's been may years since I've tried this so I don't know if it can still be done with modern WPA versions. You could always switch to WEP encryption or even turn off encryption completely and reconnect a device to that new AP.

[nospam2k]

I was able to give you the capture.pcap because I have an openwrt router as you mentioned and did exactly the same as what you said, but how did you find out about the mbid? BTW I found out that mbid stands for message body id according to chatgpt. Here is chatgpts explaination: 1. MBID (Message Body ID) Purpose: The mbid is a unique identifier used in packets to track specific messages between devices and gateways or apps. Usage: It helps identify and match responses to requests or commands sent to devices.

[uzlonewolf]

All I know is that posted pcap does not contain any local traffic to/from a phone, it only has a few packets from the Cloud.

When I'm testing stuff sometimes I shut off mobile data on my phone and block internet access at the router to force it to use a local connection instead of the cloud. Depending on what you're doing that may or may not work (some commands require access to the cloud), but generally if you can force a local connection then it will prefer that even if internet/cloud access is later restored.