How draw password on remote smartphone with mouse?

[flashcoder]

I'm working in a remote assistence software to android smartphones and i wish draw the password (similar to Team View for mobile) using the mouse and onmousemove event (or other event) of a Image component (that is where the remote screen is showed).

I'm using this Delphi code below but happens a detour on trajetory and draw a wrong password.
The image in attachment and video (following link) show how this is in pratice.

DEMOSTRATION IN VIDEO

Someone have some idea how solve this trouble?

Delphi-Quellcode:

TForm2 = class(TForm)
    procedure Image1MouseDown(Sender: TObject; Button: TMouseButton;
      Shift: TShiftState; X, Y: Integer);
    procedure Image1MouseMove(Sender: TObject; Shift: TShiftState; X, Y: Integer);
    procedure Image1MouseUp(Sender: TObject; Button: TMouseButton;
      Shift: TShiftState; X, Y: Integer);
  private
   PO,LP: TPoint;
   draw: boolean;
  public
    { Public declarations }
  end;

procedure TForm2.Image1MouseDown(Sender: TObject; Button: TMouseButton;
  Shift: TShiftState; X, Y: Integer);
begin
 PO.X:= X;
 PO.Y:= Y;
 LP.X:= X;
 LP.Y:= Y;
 draw:= true;
// Canvas.Pen.Mode:= pmNotXor;
end;

procedure TForm2.Image1MouseMove(Sender: TObject; Shift: TShiftState; X,
  Y: Integer);
begin
 
 if draw then
 
  begin
  
   if (LP.X <> PO.X) or (LP.Y <> PO.Y) then
    begin

    { Canvas.MoveTo(PO.X,PO.Y);
     Canvas.LineTo(LP.X,LP.Y); }
    end;
   
   LP.X:= X;
   LP.Y:= Y;

   { Canvas.MoveTo(PO.X,PO.Y);
   Canvas.LineTo(LP.X,LP.Y); }
  end;
end;

procedure TForm2.Image1MouseUp(Sender: TObject; Button: TMouseButton;
  Shift: TShiftState; X, Y: Integer);
begin

// Sending throught Socket: PO.X, PO.Y, LP.X, LP.Y 

 if draw then draw:= false;
end;

SocketBackgroungService class:

Code:

if (xline.contains("swipescreen")) {

    String coordenates = xline.replace("swipescreen", "");

    String[] tokens = coordenates.split(Pattern.quote("<|>"));

    float x1 = parseFloat(tokens[0]);
    float y1 = parseFloat(tokens[1]);
    float x2 = parseFloat(tokens[2]);
    float y2 = parseFloat(tokens[3]);

    MyAccessibility.instance.Swipte((int) x1, (int) y1, (int) x2, (int) y2, 50);

}

MyAccessibility class (Swipte() routine):

Code:

public void Swipte(int x1,int y1,int x2,int y2,int time){
        if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.N) {
            System.out.println(" ======= Swipte =======");

            GestureDescription.Builder gestureBuilder = new GestureDescription.Builder();
            Path path = new Path();
            path.moveTo(x1,y1);
            path.lineTo(x2,y2);

            gestureBuilder.addStroke(new GestureDescription.StrokeDescription(path, 0, time));
            dispatchGesture(gestureBuilder.build(), new GestureResultCallback() {
                @Override
                public void onCompleted(GestureDescription gestureDescription) {
                    System.out.println("SWIPTE Gesture Completed :D");
                    super.onCompleted(gestureDescription);
                }
            }, null);
        }

    }

[freimatz]

https://stackoverflow.com/questions/...mouse/52033672

[flashcoder]

Zitat:

Zitat von freimatz (Beitrag 1411790)

https://stackoverflow.com/questions/...mouse/52033672

Yes, but noone gave a definitive solution in StackOverflow.

[Whookie]

Hi, I'm no expert with mobile apps but as far as I can see you need to update the first position (PO) every time you move to another 'dot'. otherwise you are always drawing from the first spot (upper, left in your video) to where the mouse is.

In plain Delphi that could be implemented like this:

Delphi-Quellcode:

unit frmMain;

interface

uses
  Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants, System.Classes, Vcl.Graphics,
  Vcl.Controls, Vcl.Forms, Vcl.Dialogs, Vcl.ExtCtrls, acPNG;

const
  DOT_RADIUS = 8;
  maxXDots = 3;
  maxYDots = 3;
  MOUSE_SLACK = 16;

type
  TPointIdx = Record
    XIdx: Integer;
    YIdx: Integer;
  End;

  TDot = Record
    Pos:    TPoint;
    Bounds: TRect;
    Selected: Boolean;
    LinkTo: TPointIdx;
  End;

  TForm1 = class(TForm)
    PB: TPaintBox;
    procedure FormCreate(Sender: TObject);
    procedure PBMouseDown(Sender: TObject; Button: TMouseButton; Shift: TShiftState; X, Y: Integer);
    procedure PBMouseMove(Sender: TObject; Shift: TShiftState; X, Y: Integer);
    procedure PBMouseUp(Sender: TObject; Button: TMouseButton; Shift: TShiftState; X, Y: Integer);
    procedure PBPaint(Sender: TObject);
  private
    fDots: Array[0..maxXDots-1,0..maxYDots-1] Of TDot;
    fDown: Boolean;
    fPO: TPointIdx;
    fCurPos: TPoint;
    procedure CalcDotPositions;
    procedure ResetDotSelection;
    function MouseNearDots(X,Y: Integer; Var DXIdx, DYIdx: Integer): Boolean;
    function HasLinkTo(ADot: TDot): Boolean;
  public
    { Public-Deklarationen }
  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}


function PointIdx(XIdx,YIdx: Integer): TPointIdx;
begin
  Result.XIdx := XIdx;
  Result.YIdx := YIdx;
end;

procedure TForm1.FormCreate(Sender: TObject);
begin
  CalcDotPositions;
end;

procedure TForm1.CalcDotPositions;
var
  LCntrHorz, LCntrVert: Integer;
  LLeftHorz, LTopVert: Integer;
  LRghtHorz, LBotVert: Integer;
  ix: Integer;
  iy: Integer;
begin
  LCntrHorz := PB.Width Div 2;
  LLeftHorz := DOT_RADIUS + Round(LCntrHorz*0.1);
  LRghtHorz := PB.Width - LLeftHorz;
  LCntrVert := PB.Height Div 2;
  LTopVert := LCntrVert - (LRghtHorz - LCntrHorz);
  LBotVert := LCntrVert + (LRghtHorz - LCntrHorz);

  fDots[0,0].Pos := Point(LLeftHorz, LTopVert);
  fDots[0,1].Pos := Point(LLeftHorz, LCntrVert);
  fDots[0,2].Pos := Point(LLeftHorz, LBotVert);

  fDots[1,0].Pos := Point(LCntrHorz, LTopVert);
  fDots[1,1].Pos := Point(LCntrHorz, LCntrVert);
  fDots[1,2].Pos := Point(LCntrHorz, LBotVert);

  fDots[2,0].Pos := Point(LRghtHorz, LTopVert);
  fDots[2,1].Pos := Point(LRghtHorz, LCntrVert);
  fDots[2,2].Pos := Point(LRghtHorz, LBotVert);

  for ix := 0 to maxXDots-1 do
  begin
    for iy := 0 to maxYDots-1 do
    begin
      fDots[ix,iy].Selected := FALSE;
      fDots[ix,iy].Bounds := Rect(
        fDots[ix,iy].Pos.X - DOT_RADIUS,
        fDots[ix,iy].Pos.Y - DOT_RADIUS,
        fDots[ix,iy].Pos.X + DOT_RADIUS,
        fDots[ix,iy].Pos.Y + DOT_RADIUS
      );
    end;
  end;
end;

procedure TForm1.ResetDotSelection;
var
  ix,iy: Integer;
begin
  for ix := 0 to maxXDots-1 do
  begin
    for iy := 0 to maxYDots-1 do
    begin
      fDots[ix,iy].Selected := FALSE;
    end;
  end;
end;

function TForm1.MouseNearDots(X,Y: Integer; Var DXIdx, DYIdx: Integer): Boolean;
var
  ix,iy: Integer;
  LRect: TRect;
begin
  Result := FALSE;
  for ix := 0 to maxXDots-1 do
  begin
    for iy := 0 to maxYDots-1 do
    begin
      LRect := fDots[ix, iy].Bounds;
      InflateRect(LRect, MOUSE_SLACK, MOUSE_SLACK);
      if PtInRect(LRect, Point(X,Y)) then
      begin
        DXIdx := ix;
        DYIdx := iy;
        Result := TRUE;
        Break;
      end;
    end;
  end;
end;

function TForm1.HasLinkTo(ADot: TDot): Boolean;
begin
  Result := (ADot.LinkTo.XIdx >= 0) And (ADot.LinkTo.YIdx >= 0);
end;

procedure TForm1.PBMouseDown(Sender: TObject; Button: TMouseButton; Shift: TShiftState; X, Y: Integer);
var
  dx, dy: Integer;
begin
  ResetDotSelection;
  if MouseNearDots(X,Y, dx,dy) then
  begin
    fDown := TRUE;
    fPO := PointIdx(dx,dy);
    fDots[dx, dy].Selected := TRUE;
    fDots[dx, dy].LinkTo := PointIdx(-1, -1);
  end;
end;

procedure TForm1.PBMouseMove(Sender: TObject; Shift: TShiftState; X, Y: Integer);
var
  dx: Integer;
  dy: Integer;
begin
  if fDown then
  begin
    if MouseNearDots(X,Y, dx, dy) and not fDots[dx, dy].Selected then
    begin
      fCurPos := fDots[dx,dy].Pos;
      fDots[dx, dy].Selected := TRUE;
      fDots[dx, dy].LinkTo := fPO;
      fPO := PointIdx(dx, dy);
    end
    else
      fCurPos := Point(X,Y);

    Invalidate;
  end;
end;

procedure TForm1.PBMouseUp(Sender: TObject; Button: TMouseButton; Shift:
    TShiftState; X, Y: Integer);
begin
  fDown := FALSE;
  PB.Invalidate;
end;


procedure TForm1.PBPaint(Sender: TObject);
var
  ix,iy: Integer;
begin
  // paint dots
  PB.Canvas.Brush.Color := clSilver;
  PB.Canvas.Brush.Style := bsSolid;
  PB.Canvas.Pen.Color := clBlack;
  PB.Canvas.Pen.Style := psSolid;
  PB.Canvas.Pen.Width := 1;
  for iy := 0 to maxYDots-1 do
  begin
    for ix := 0 to maxXDots-1 do
    begin
      PB.Canvas.Ellipse(fDots[ix,iy].Bounds);
    end;
  end;

  // draw fixed segemts
  PB.Canvas.Pen.Color := clYellow;
  PB.Canvas.Pen.Width := 6;
  for iy := 0 to maxYDots-1 do
  begin
    for ix := 0 to maxXDots-1 do
    begin
      if fDots[ix,iy].Selected And HasLinkTo(fDots[ix,iy]) then
      begin
       PB.Canvas.MoveTo( fDots[ix, iy].Pos.X, fDots[ix, iy].Pos.Y );
       PB.Canvas.LineTo( fDots[fDots[ix, iy].LinkTo.XIdx, fDots[ix, iy].LinkTo.YIdx].Pos.X,
                         fDots[fDots[ix, iy].LinkTo.XIdx, fDots[ix, iy].LinkTo.YIdx].Pos.Y  );
      end;
    end;
  end;


  // draw current segment
  if fDown then
  begin
    PB.Canvas.Pen.Color := clYellow;
    PB.Canvas.Pen.Width := 6;
    PB.Canvas.MoveTo( fDots[fPO.XIdx, fPO.YIdx].Pos.X, fDots[fPO.XIdx, fPO.YIdx].Pos.Y );
    PB.Canvas.LineTo( fCurPos.X, fCurPos.Y );
  end;
end;

end.

with a Form like this:

Delphi-Quellcode:

object Form1: TForm1
  Left = 0
  Top = 0
  Caption = 'Form1'
  ClientHeight = 450
  ClientWidth = 250
  Color = clBtnFace
  Font.Charset = DEFAULT_CHARSET
  Font.Color = clWindowText
  Font.Height = -11
  Font.Name = 'Tahoma'
  Font.Style = []
  OldCreateOrder = False
  OnCreate = FormCreate
  PixelsPerInch = 96
  TextHeight = 13
  object PB: TPaintBox
    Left = 8
    Top = 8
    Width = 233
    Height = 427
    OnMouseDown = PBMouseDown
    OnMouseMove = PBMouseMove
    OnMouseUp = PBMouseUp
    OnPaint = PBPaint
  end
end

[flashcoder]

@Whookie,

thank you very much, this is a good example!:-D

But this code also will work to 9 (nine) dots?

I saw that you drawn only 3 dots on PaintBox.

[flashcoder]

I'm not understood very fine your example above, then i tried follow your hint in my actual code, but without success, until now.

How update correctly PO coordenates on code below?

Delphi-Quellcode:

 private
    { Private declarations }
    PO, LP: TPoint;
    Draw: boolean;
  public
    { Public declarations }
  end;
  
  ...

procedure TForm2.Image1MouseDown(Sender: TObject; Button: TMouseButton;
  Shift: TShiftState; X, Y: Integer);
var
  Index, XTouch, YTouch, RXCoord, RYCoord: Integer;
  List: TStrings;
  RScreen: String;
begin
  Index := Form1.ListView1.ItemIndex;
  if Index = -1 then
    Exit;

    List := TStringList.Create;
    RScreen := Form1.ListView1.Selected.SubItems[6]; // Remote screen resolution

    try
      ExtractStrings(['x'], [], PChar(RScreen), List); // Ex: my smartphone is 1920x1080
      RYCoord := StrToInt(List[0]);                    // 1920 (height)
      RXCoord := StrToInt(List[1]);                    // 1080 (width)
    finally
      List.Free;
    end;

    XTouch := Round((X / Image1.Width) * RXCoord);
    YTouch := Round((Y / Image1.Height) * RYCoord);

    PO.X := XTouch;
    PO.Y := YTouch;
    LP.X := XTouch;
    LP.Y := YTouch;
   
    Draw := true;

    { Form1.ServerSocket1.Socket.Connections[Index].SendText('touch' + IntToStr(XTouch) + '<|>' + 
                                                      IntToStr(YTouch) + #13#10); 
    }

  end;
end;

procedure TForm2.Image1MouseMove(Sender: TObject; Shift: TShiftState;
  X, Y: Integer);
var
  Index, XTouch, YTouch, RXCoord, RYCoord: Integer;
  List: TStrings;
  RScreen: String;
begin

  Index := Form1.ListView1.ItemIndex;
  if Index = -1 then
    Exit;

  List := TStringList.Create;
  RScreen := Form1.ListView1.Selected.SubItems[6]; // Remote screen resolution

  try
    ExtractStrings(['x'], [], PChar(RScreen), List); // Ex: my smartphone is 1920x1080
    RYCoord := StrToInt(List[0]);                    // 1920 (height)
    RXCoord := StrToInt(List[1]);                    // 1080 (width)
  finally
    List.Free;
  end;

  XTouch := Round((X / Image1.Width) * RXCoord);
  YTouch := Round((Y / Image1.Height) * RYCoord);

  if Draw then
  begin
    LP.X := XTouch;
    LP.Y := YTouch;

    Form1.ServerSocket1.Socket.Connections[Index].SendText('swipescreen' + IntToStr(PO.X) + '<|>' + 
                                                            IntToStr(PO.Y) + '<|>' + 
                                                            IntToStr(LP.X) + '<|>' + 
                                                            IntToStr(LP.Y) + #13#10);

    PO.X := LP.X;
    PO.Y := LP.Y;
  end;

end;

[Whookie]

Zitat:

Zitat von flashcoder (Beitrag 1412039)

@Whookie,

thank you very much, this is a good example!:-D

But this code also will work to 9 (nine) dots?

I saw that you drawn only 3 dots on PaintBox.

My code is drawing 9 dots (as the paintbox is of fixed size, you might need to resize the form...):

Anhang 49856

[Whookie]

Zitat:

Zitat von flashcoder (Beitrag 1412068)

How update correctly PO coordenates on code below?

Have a look at my code, I store the positions of all 9 dots in my fDots[0..2,0..2] - array (.Pos is the center point, .Bounds is the bounding rectangle). This is done once in the CalcDotPositions method.

In the Paintbox MouseDown event I use the MouseNearDots()-function to see if the current mouse position (X,Y) is near of one of the nine dots. Drawing does not start from anywhere, you need to be near of one of the dots (= bounding rectangle + MOUSE_SLACK).

If drawing starts I save the x/y-index of the starting dot within my fPO variable.

In PBMouseMove I also use MouseNearDots() to decide, if another dot (except already selected ones) is reached. If another dot is reached its fDots[].Selected member is set to TRUE, it is linked to the previous dot (.LinkTo := fPO) and fPO is set to be the new starting point (fPO := PointIdx())!
Otherwise just the current mouse position is stored (fCurPos).

Note that all drawing is done in the OnPaint event (first the nine dots are drawn, then all the segments wich are already connected and finally the current segment).

So I think you dont need just the size of your mobile devices screen, you also will need the positions of the nine dots to provide the correct coordinates for your .SendText function.


Another option might be the use of relative movement. In MouseDown you could use 'moveto' to provide the start coordinates and in MouseMove some sort of 'swipeto' which only sends the relative movement changes:

Code:

procedure TForm1.PBMouseDown(Sender: TObject; Button: TMouseButton; Shift: TShiftState; X, Y: Integer);
begin
  fPO := Point(X,Y);
  fDown := TRUE;
  Form1.ServerSocket1.Socket.Connections[Index].SendText(format('moveto%d<|>%d'#13#10, [X,Y]));
end;

procedure TForm1.PBMouseMove(Sender: TObject; Shift: TShiftState; X, Y: Integer);
begin
  if fDown then
  begin
    Form1.ServerSocket1.Socket.Connections[Index].SendText(format('swipeto%d<|>%d'#13#10, [X-fPO.X,Y-fPO.Y]));
    fPO := Point(X,Y);
  end;
end;

I can not test this, nor do I know if you are able to implement such commands on the mobile device. The main point here is to not provide a line with always the same starting point but to just send the current mouse position (either relative like in the code above or may be even absolute).

[flashcoder]

@Whookie,

[flashcoder]

Considering the last suggestion i made this following code in my project but generates a exception on Android code:

Zitat:

java.lang.IllegalStateException: Attempting to add too many strokes to a gesture

in this line:

Code:

gestureBuilder.addStroke(new GestureDescription.StrokeDescription(path, 100, time));

Delphi-Quellcode:

 private
    { Private declarations }
    fDown: Boolean;
    fPO: TPoint;
  public
    { Public declarations }
  end;
  
  ...

procedure TForm2.Image1MouseDown(Sender: TObject; Button: TMouseButton;
  Shift: TShiftState; X, Y: Integer);
var
  Index, XTouch, YTouch, RXCoord, RYCoord: Integer;
  List: TStrings;
  RScreen, MoveTo: String;
begin
  Index := Form1.ListView1.ItemIndex;
  if Index = -1 then
    Exit;

    List := TStringList.Create;
    RScreen := Form1.ListView1.Selected.SubItems[6]; // Remote screen resolution

    try
      ExtractStrings(['x'], [], PChar(RScreen), List); // Ex: my smartphone is 1920x1080
      RYCoord := StrToInt(List[0]);                    // 1920 (height)
      RXCoord := StrToInt(List[1]);                    // 1080 (width)
    finally
      List.Free;
    end;

    XTouch := Round((X / Image1.Width) * RXCoord);
    YTouch := Round((Y / Image1.Height) * RYCoord);

    fPO := Point(XTouch, YTouch);
    MoveTo := 'moveto';
    fDown := true;

    Form1.ServerSocket1.Socket.Connections[Index].SendText(format('mouseswipescreen%d<|>%d<|>%s'#13#10, [fPO.X, fPO.Y, MoveTo]));

end;

procedure TForm2.Image1MouseMove(Sender: TObject; Shift: TShiftState;
  X, Y: Integer);
var
  Index, XTouch, YTouch, RXCoord, RYCoord: Integer;
  List: TStrings;
  RScreen, SwipeTo: String;
begin

  Index := Form1.ListView1.ItemIndex;
  if Index = -1 then
    Exit;

  List := TStringList.Create;
  RScreen := Form1.ListView1.Selected.SubItems[6]; // Remote screen resolution

  try
    ExtractStrings(['x'], [], PChar(RScreen), List); // Ex: my smartphone is 1920x1080
    RYCoord := StrToInt(List[0]);                    // 1920 (height)
    RXCoord := StrToInt(List[1]);                    // 1080 (width)
  finally
    List.Free;
  end;

  XTouch := Round((X / Image1.Width) * RXCoord);
  YTouch := Round((Y / Image1.Height) * RYCoord);

  if fDown then
  begin

    SwipeTo := 'swipeto';

    Form1.ServerSocket1.Socket.Connections[Index].SendText(format('mouseswipescreen%d<|>%d<|>%s'#13#10, [fPO.X, fPO.Y, SwipeTo]));

    fPO := Point(XTouch, YTouch);

  end;

end;

procedure TForm2.Image1MouseUp(Sender: TObject; Button: TMouseButton;
  Shift: TShiftState; X, Y: Integer);
begin
  if fDown then
    fDown := false;
end;

Android:

Code:

if (xline.contains("swipescreen")) {

    String coordenates = xline.replace("swipescreen", "");

    String[] tokens = coordenates.split(Pattern.quote("<|>"));

    float x = parseFloat(tokens[0]);
    float y = parseFloat(tokens[1]);
    String cmd = tokens[2];

    MyAccessibility.instance.Swipte((int) x, (int) y, 50, cmd);

}

Code:

GestureDescription.Builder gestureBuilder;
Path path;

public void Swipte(int x, int y, int time, String command) {

    if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.N) {
        System.out.println(" ======= Swipte =======");

        if (command.equalsIgnoreCase("moveto")) {

            gestureBuilder = new GestureDescription.Builder();
            path = new Path();

            path.moveTo(x, y);

        } else if (command.equalsIgnoreCase("swipeto")) {

            path.lineTo(x, y);
            gestureBuilder.addStroke(new GestureDescription.StrokeDescription(path, 100, time));

            dispatchGesture(gestureBuilder.build(), new GestureResultCallback() {
                @Override
                public void onCompleted(GestureDescription gestureDescription) {
                    System.out.println("SWIPTE Gesture Completed :D");
                    super.onCompleted(gestureDescription);
                }
            }, null);
        }
      }
    }

}

[Whookie]

Hi,
its hard to tell, if this could work on android...

You could reduce the number of pixels (path segements) by just adding another position if it differes from the last one by a reasonable amount.

But thinking about this I must admit that I'm not sure if this will ever work. Adding all the mouse movement will end up in a drawing (like using Paint and moving the mouse around).

What you are trying would need another set of functions where you could emulate touching the screen and let android do all the work (in Windows you could send WM_MOUSEDOWN/WM_MOUSEMOVE/... messages yourself and letting the application beneath the mouse do, what ever it would do).

If that's not possible you might need to go back to my example and implement something like that...

[flashcoder]

@Wookie,

thank you very much by help and code example, i concluded that:

My approach above is right and unfortunately until this moment was saw that, not is possible achieve this goal using AccessibilityService, because exists limitation on storage of gestures, here is the official reference:

Code:

addStroke(new GestureDescription.StrokeDescription(path, 0, time));

And also, until now (in this case) AccessibilityService not is able to perform a gesture of touch "only press and not loose" in some point (of 9 points that compose the password screen) when 2 fast mouse clicks are perform and sent on:

Delphi-Quellcode:

procedure TForm2.Image1MouseDown(Sender: TObject; Button: TMouseButton;
  Shift: TShiftState; X, Y: Integer);

before start and during drawing the password (dragging mouse), and release only when password be drawn completely and:

Delphi-Quellcode:

procedure TForm2.Image1MouseUp(Sender: TObject; Button: TMouseButton;
  Shift: TShiftState; X, Y: Integer);

be sent to remote device.

---------------------------------------------------------------------------------------------------------------------------------------

Note: Team View QuickSupport is able to this because use code of root gave to he by several vendors.