How draw password on remote smartphone with mouse?

[flashcoder]

I'm not understood very fine your example above, then i tried follow your hint in my actual code, but without success, until now.

How update correctly PO coordenates on code below?

Delphi-Quellcode:

 private
    { Private declarations }
    PO, LP: TPoint;
    Draw: boolean;
  public
    { Public declarations }
  end;
  
  ...

procedure TForm2.Image1MouseDown(Sender: TObject; Button: TMouseButton;
  Shift: TShiftState; X, Y: Integer);
var
  Index, XTouch, YTouch, RXCoord, RYCoord: Integer;
  List: TStrings;
  RScreen: String;
begin
  Index := Form1.ListView1.ItemIndex;
  if Index = -1 then
    Exit;

    List := TStringList.Create;
    RScreen := Form1.ListView1.Selected.SubItems[6]; // Remote screen resolution

    try
      ExtractStrings(['x'], [], PChar(RScreen), List); // Ex: my smartphone is 1920x1080
      RYCoord := StrToInt(List[0]);                    // 1920 (height)
      RXCoord := StrToInt(List[1]);                    // 1080 (width)
    finally
      List.Free;
    end;

    XTouch := Round((X / Image1.Width) * RXCoord);
    YTouch := Round((Y / Image1.Height) * RYCoord);

    PO.X := XTouch;
    PO.Y := YTouch;
    LP.X := XTouch;
    LP.Y := YTouch;
   
    Draw := true;

    { Form1.ServerSocket1.Socket.Connections[Index].SendText('touch' + IntToStr(XTouch) + '<|>' + 
                                                      IntToStr(YTouch) + #13#10); 
    }

  end;
end;

procedure TForm2.Image1MouseMove(Sender: TObject; Shift: TShiftState;
  X, Y: Integer);
var
  Index, XTouch, YTouch, RXCoord, RYCoord: Integer;
  List: TStrings;
  RScreen: String;
begin

  Index := Form1.ListView1.ItemIndex;
  if Index = -1 then
    Exit;

  List := TStringList.Create;
  RScreen := Form1.ListView1.Selected.SubItems[6]; // Remote screen resolution

  try
    ExtractStrings(['x'], [], PChar(RScreen), List); // Ex: my smartphone is 1920x1080
    RYCoord := StrToInt(List[0]);                    // 1920 (height)
    RXCoord := StrToInt(List[1]);                    // 1080 (width)
  finally
    List.Free;
  end;

  XTouch := Round((X / Image1.Width) * RXCoord);
  YTouch := Round((Y / Image1.Height) * RYCoord);

  if Draw then
  begin
    LP.X := XTouch;
    LP.Y := YTouch;

    Form1.ServerSocket1.Socket.Connections[Index].SendText('swipescreen' + IntToStr(PO.X) + '<|>' + 
                                                            IntToStr(PO.Y) + '<|>' + 
                                                            IntToStr(LP.X) + '<|>' + 
                                                            IntToStr(LP.Y) + #13#10);

    PO.X := LP.X;
    PO.Y := LP.Y;
  end;

end;

[Whookie]

Zitat von flashcoder:

How update correctly PO coordenates on code below?

Have a look at my code, I store the positions of all 9 dots in my fDots[0..2,0..2] - array (.Pos is the center point, .Bounds is the bounding rectangle). This is done once in the CalcDotPositions method.

In the Paintbox MouseDown event I use the MouseNearDots()-function to see if the current mouse position (X,Y) is near of one of the nine dots. Drawing does not start from anywhere, you need to be near of one of the dots (= bounding rectangle + MOUSE_SLACK).

If drawing starts I save the x/y-index of the starting dot within my fPO variable.

In PBMouseMove I also use MouseNearDots() to decide, if another dot (except already selected ones) is reached. If another dot is reached its fDots[].Selected member is set to TRUE, it is linked to the previous dot (.LinkTo := fPO) and fPO is set to be the new starting point (fPO := PointIdx())!
Otherwise just the current mouse position is stored (fCurPos).

Note that all drawing is done in the OnPaint event (first the nine dots are drawn, then all the segments wich are already connected and finally the current segment).

So I think you dont need just the size of your mobile devices screen, you also will need the positions of the nine dots to provide the correct coordinates for your .SendText function.


Another option might be the use of relative movement. In MouseDown you could use 'moveto' to provide the start coordinates and in MouseMove some sort of 'swipeto' which only sends the relative movement changes:

Code:

procedure TForm1.PBMouseDown(Sender: TObject; Button: TMouseButton; Shift: TShiftState; X, Y: Integer);
begin
  fPO := Point(X,Y);
  fDown := TRUE;
  Form1.ServerSocket1.Socket.Connections[Index].SendText(format('moveto%d<|>%d'#13#10, [X,Y]));
end;

procedure TForm1.PBMouseMove(Sender: TObject; Shift: TShiftState; X, Y: Integer);
begin
  if fDown then
  begin
    Form1.ServerSocket1.Socket.Connections[Index].SendText(format('swipeto%d<|>%d'#13#10, [X-fPO.X,Y-fPO.Y]));
    fPO := Point(X,Y);
  end;
end;

I can not test this, nor do I know if you are able to implement such commands on the mobile device. The main point here is to not provide a line with always the same starting point but to just send the current mouse position (either relative like in the code above or may be even absolute).

[flashcoder]

@Whookie,

[flashcoder]

Considering the last suggestion i made this following code in my project but generates a exception on Android code:

Zitat:

java.lang.IllegalStateException: Attempting to add too many strokes to a gesture

in this line:

Code:

gestureBuilder.addStroke(new GestureDescription.StrokeDescription(path, 100, time));

Delphi-Quellcode:

 private
    { Private declarations }
    fDown: Boolean;
    fPO: TPoint;
  public
    { Public declarations }
  end;
  
  ...

procedure TForm2.Image1MouseDown(Sender: TObject; Button: TMouseButton;
  Shift: TShiftState; X, Y: Integer);
var
  Index, XTouch, YTouch, RXCoord, RYCoord: Integer;
  List: TStrings;
  RScreen, MoveTo: String;
begin
  Index := Form1.ListView1.ItemIndex;
  if Index = -1 then
    Exit;

    List := TStringList.Create;
    RScreen := Form1.ListView1.Selected.SubItems[6]; // Remote screen resolution

    try
      ExtractStrings(['x'], [], PChar(RScreen), List); // Ex: my smartphone is 1920x1080
      RYCoord := StrToInt(List[0]);                    // 1920 (height)
      RXCoord := StrToInt(List[1]);                    // 1080 (width)
    finally
      List.Free;
    end;

    XTouch := Round((X / Image1.Width) * RXCoord);
    YTouch := Round((Y / Image1.Height) * RYCoord);

    fPO := Point(XTouch, YTouch);
    MoveTo := 'moveto';
    fDown := true;

    Form1.ServerSocket1.Socket.Connections[Index].SendText(format('mouseswipescreen%d<|>%d<|>%s'#13#10, [fPO.X, fPO.Y, MoveTo]));

end;

procedure TForm2.Image1MouseMove(Sender: TObject; Shift: TShiftState;
  X, Y: Integer);
var
  Index, XTouch, YTouch, RXCoord, RYCoord: Integer;
  List: TStrings;
  RScreen, SwipeTo: String;
begin

  Index := Form1.ListView1.ItemIndex;
  if Index = -1 then
    Exit;

  List := TStringList.Create;
  RScreen := Form1.ListView1.Selected.SubItems[6]; // Remote screen resolution

  try
    ExtractStrings(['x'], [], PChar(RScreen), List); // Ex: my smartphone is 1920x1080
    RYCoord := StrToInt(List[0]);                    // 1920 (height)
    RXCoord := StrToInt(List[1]);                    // 1080 (width)
  finally
    List.Free;
  end;

  XTouch := Round((X / Image1.Width) * RXCoord);
  YTouch := Round((Y / Image1.Height) * RYCoord);

  if fDown then
  begin

    SwipeTo := 'swipeto';

    Form1.ServerSocket1.Socket.Connections[Index].SendText(format('mouseswipescreen%d<|>%d<|>%s'#13#10, [fPO.X, fPO.Y, SwipeTo]));

    fPO := Point(XTouch, YTouch);

  end;

end;

procedure TForm2.Image1MouseUp(Sender: TObject; Button: TMouseButton;
  Shift: TShiftState; X, Y: Integer);
begin
  if fDown then
    fDown := false;
end;

Android:

Code:

if (xline.contains("swipescreen")) {

    String coordenates = xline.replace("swipescreen", "");

    String[] tokens = coordenates.split(Pattern.quote("<|>"));

    float x = parseFloat(tokens[0]);
    float y = parseFloat(tokens[1]);
    String cmd = tokens[2];

    MyAccessibility.instance.Swipte((int) x, (int) y, 50, cmd);

}

Code:

GestureDescription.Builder gestureBuilder;
Path path;

public void Swipte(int x, int y, int time, String command) {

    if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.N) {
        System.out.println(" ======= Swipte =======");

        if (command.equalsIgnoreCase("moveto")) {

            gestureBuilder = new GestureDescription.Builder();
            path = new Path();

            path.moveTo(x, y);

        } else if (command.equalsIgnoreCase("swipeto")) {

            path.lineTo(x, y);
            gestureBuilder.addStroke(new GestureDescription.StrokeDescription(path, 100, time));

            dispatchGesture(gestureBuilder.build(), new GestureResultCallback() {
                @Override
                public void onCompleted(GestureDescription gestureDescription) {
                    System.out.println("SWIPTE Gesture Completed :D");
                    super.onCompleted(gestureDescription);
                }
            }, null);
        }
      }
    }

}

[Whookie]

Hi,
its hard to tell, if this could work on android...

You could reduce the number of pixels (path segements) by just adding another position if it differes from the last one by a reasonable amount.

But thinking about this I must admit that I'm not sure if this will ever work. Adding all the mouse movement will end up in a drawing (like using Paint and moving the mouse around).

What you are trying would need another set of functions where you could emulate touching the screen and let android do all the work (in Windows you could send WM_MOUSEDOWN/WM_MOUSEMOVE/... messages yourself and letting the application beneath the mouse do, what ever it would do).

If that's not possible you might need to go back to my example and implement something like that...

[flashcoder]

@Wookie,

thank you very much by help and code example, i concluded that:

My approach above is right and unfortunately until this moment was saw that, not is possible achieve this goal using AccessibilityService, because exists limitation on storage of gestures, here is the official reference:

Code:

addStroke(new GestureDescription.StrokeDescription(path, 0, time));

And also, until now (in this case) AccessibilityService not is able to perform a gesture of touch "only press and not loose" in some point (of 9 points that compose the password screen) when 2 fast mouse clicks are perform and sent on:

Delphi-Quellcode:

procedure TForm2.Image1MouseDown(Sender: TObject; Button: TMouseButton;
  Shift: TShiftState; X, Y: Integer);

before start and during drawing the password (dragging mouse), and release only when password be drawn completely and:

Delphi-Quellcode:

procedure TForm2.Image1MouseUp(Sender: TObject; Button: TMouseButton;
  Shift: TShiftState; X, Y: Integer);

be sent to remote device.

---------------------------------------------------------------------------------------------------------------------------------------

Note: Team View QuickSupport is able to this because use code of root gave to he by several vendors.