Remote control app for iOS devices

[guanzhangrtk]

Hi @nanoscopic thank you for this project.

I'm looking for an Open Source solution that will allow me to control iOS devices from Windows/Mac/Linux and stumbled upon your project.

I understand that the main project is geared towards device farms but I don't see why it can't be used by a single user who just want to remotely control their iOS devices, and perhaps create some macros for automation. What I'm looking for can be summarized in this Reddit post.

Can you please confirm/verify whether this could be possible? Thanks a lot for your attention and Happy New Year!

[nanoscopic]

You're welcome.

The project overall was made with a few different goals:

1. To bring open source code that works to the scene of controlling iOS devices. This has been sorely lacking for many years. Before this project there was some scattering of information and ways to potentially do this but nothing concrete or reasonably use existed.

2. To pay my own salary, as I find this work interesting and want to spend all my time doing this. Dry Ark LLC exists mainly for this purpose, and it has just recently become a company of a whopping 2 employees now! ( myself and Brian Krznarich ) As a result of becoming dependent on making a living from the project, I've recently taken all the updates this project and its many components closed source. The reason being that the only people contributing money to the project have been companies.

3. To introduce some sane competition to the existing market players offering similar things in a closed source and uncooperative fashion.

Keeping these goals in mind, I'm not opposed to continuing to allow individual hobbyists to use the project for whatever purpose they wish. I would of course appreciate if individuals contribute some small amount to the project as well per the value gained from its use, but those amounts are unlikely to add up to much.

As a result of this, what will be coming soon is binary releases of the further updated components and a licensing system that allows individuals such as yourself to get a license for your handful of devices for free.

I would still like to make all the updates open source, but I need to first ensure the profitability of the company I have formed around this.

In regard to using the system to play games on a larger screen and control them there, the following are relevant:

1. The video feed speed and latency should be adequate, especially if the device is local directly plugged into the same machine.
2. Clicking and swiping work well with low latency.
3. Swiping does NOT take effect till after the swipe is finished, so this will interfere with the ability to play games where that extra delay will make you lose the game. This is a restriction of how Apple automation works. In order to get around this, there are a variety of options that have yet to be implemented. The first is use a virtual bluetooth mouse. Since you are local to the device, you could use a real bluetooth ( or USB ) mouse as well.

If you are using a bluetooth or USB mouse connected to your iOS device, you probably don't even need this system to do what you want, as you could just use Apple Airplay with an Airplay compatible screen.

I am still unclear what problem you would be attempting to solve with use of this software.

It sounds like you want to use the system to automate certain tedious actions ( perhaps to mine items that can be sold in online game markets?? ) I know of at least one person making money by doing this... The system could/can be used in that way... I'm not terribly fond of this as game makers don't generally want their games being automated this way and consider it cheating.

Despite the cheaty nature of it, it isn't immoral in my mind. If you want to cheat at games, that is on you. Also, it is interesting from a technology point of view. As a result, I'm not opposed to the system being used in that way. Just don't be surprised when game makers try to prevent this. We will be taking care not to intentionally offer the software with that goal, as it could run afoul of the rules with how Xcode Xctest automation is supposed to be used. Generally though, if you are doing it, you would be using your own Apple developer account with this software, in which case it is your own responsibility to abide by Apple rules.

[guanzhangrtk]

You're welcome.

The project overall was made with a few different goals:

1. To bring open source code that works to the scene of controlling iOS devices. This has been sorely lacking for many years. Before this project there was some scattering of information and ways to potentially do this but nothing concrete or reasonably use existed.
2. To pay my own salary, as I find this work interesting and want to spend all my time doing this. Dry Ark LLC exists mainly for this purpose, and it has just recently become a company of a whopping 2 employees now! ( myself and Brian Krznarich ) As a result of becoming dependent on making a living from the project, I've recently taken all the updates this project and its many components closed source. The reason being that the only people contributing money to the project have been companies.
3. To introduce some sane competition to the existing market players offering similar things in a closed source and uncooperative fashion.

Keeping these goals in mind, I'm not opposed to continuing to allow individual hobbyists to use the project for whatever purpose they wish. I would of course appreciate if individuals contribute some small amount to the project as well per the value gained from its use, but those amounts are unlikely to add up to much.

As a result of this, what will be coming soon is binary releases of the further updated components and a licensing system that allows individuals such as yourself to get a license for your handful of devices for free.

I would still like to make all the updates open source, but I need to first ensure the profitability of the company I have formed around this.

Thank you for the detailed response! I am happy to contribute my time in terms of testing/coding if it aligns with my interests but unfortunately will not be able to contribute any money into this, sorry about that.

In regard to using the system to play games on a larger screen and control them there, the following are relevant:

1. The video feed speed and latency should be adequate, especially if the device is local directly plugged into the same machine.
2. Clicking and swiping work well with low latency.
3. Swiping does NOT take effect till after the swipe is finished, so this will interfere with the ability to play games where that extra delay will make you lose the game. This is a restriction of how Apple automation works. In order to get around this, there are a variety of options that have yet to be implemented. The first is use a virtual bluetooth mouse. Since you are local to the device, you could use a real bluetooth ( or USB ) mouse as well.

If you are using a bluetooth or USB mouse connected to your iOS device, you probably don't even need this system to do what you want, as you could just use Apple Airplay with an Airplay compatible screen.

Thanks for these useful info!

It sounds like you want to use the system to automate certain tedious actions ( perhaps to mine items that can be sold in online game markets?? ) I know of at least one person making money by doing this... The system could/can be used in that way... I'm not terribly fond of this as game makers don't generally want their games being automated this way and consider it cheating.

Despite the cheaty nature of it, it isn't immoral in my mind. If you want to cheat at games, that is on you. Also, it is interesting from a technology point of view. As a result, I'm not opposed to the system being used in that way. Just don't be surprised when game makers try to prevent this. We will be taking care not to intentionally offer the software with that goal, as it could run afoul of the rules with how Xcode Xctest automation is supposed to be used. Generally though, if you are doing it, you would be using your own Apple developer account with this software, in which case it is your own responsibility to abide by Apple rules.

Botting in games is certainly in the grey area. Most game developers have it in their ToS that states that it is not allowed and it would be within their rights to ban your account if they determine that you have been using third party software to do so. However, the reality is that most of them do not care about this to the extent that they would actually ban you, unless it's some really competitive game like PUBGM. In fact macro/scripting are popular features which Android Emulators such as BlueStacks provide to their users, they are so popular to the extent that developers create special rooms for players who play on emulators such as to provide a more "fair" competition between the different groups of players (mobile vs PC players).

For my case, I am merely interested in automating tedious tasks, and I'm not here for monetary gain. I mean I can certainly see people using it to farm items to sell on the market which you mentioned but it is not my goal at all. So you have to decide whether this is something you are okay with when your software is being used for such tasks.

In fact for me, I will be happy to just have a nice and polished application on Windows or Mac where I can just screen mirror my iOS device and control it, such that I can play the game on the big screen instead of on my tiny phone. Bonus points for using a controller with it. As mentioned in my Reddit post most of the solutions out there are paid software like Wormhole which didn't work particularly well for my case.

In fact if this sounds interesting to you, you can potentially sell the software as an individual app on Mac/Microsoft store, I think there could be a lot of interest in it. Assuming you can find a balance between providing Open Source code and selling it at the same time.

You mentioned that this may potentially run afoul with Apple's rules on Xcode Xctest automation -- do you happen to have a link to such rules? I would like to review it for my own information if possible.

Thanks again and I hope this clarifies my interest and goals on this project.

[mankoffs]

Hello @nanoscopic. Is there any news regarding your project and iOS device support? Perhaps something appeared paid? We need a stable mechanism for a remote control app for iOS devices. And in general, is it supported, and will it be supported?