/* * * euan - Exploration into using Java robot. * * Author: Doug Rice, * doug.h.rice@btinternet.com * * Open EUA and move to the Top Left of the screen. * * usage: * java euan defaults to "java euan eua 0 0 10 10" * * java euan filename x y w d - EUA top left corner * * java euan eua 100 100 10 10 - if this area changes, end the script. e.g cmd maximised or editor maximised. * * EUAN - End User App Nerd * EUAN - End User App Norman * EUAN - End User App Numpty * EUAN - End User App Nellie * * Try and automate a gui to replace manual actions e.g. auto join and reconnect the EUA app. * * Watch the screen where the gui is and when changes are noticed use these to trigger mouse moves, clicks and key presses. * * Use JavaRobot to sample rectangles of the screen and read the colours of pixels. * * monitor areas for changes and generate events to be serviced by a service routines. * monitor pixels for changes and generate events to be serviced by a service routines. * consider this to be like a peripheral in a MicroChip microcontroller. * * A simple real time system works by a simple loop. * * init() * while( true ){ * poll_and_make_events_and_set_InterruptFlags * service_events() * sleep() * } * * In a MicropChip 16F microcontroller, there are a number of peripherals like timers, uarts, etc. * You can consider each periperal is a co-processor specific to what is good at. * Each "peripheral" has a flag bit that is set to indicate an interupt is required. * * The main loop or Interupt service routine polls these bits and runs an interupt service routine, which clears the flag * * This program looks at areas of the screen which are polled and each has an interupt flag which is set to indicate an event. * * It is possible to monitor indiviual pixels and also set interupt flags. * * To automate the GUI, click on buttons. * If the user moves the mouse, disable the events for a period of time. * * usage: * java euan defaults to "java euan eua 100 100 10 10" * * java euan filename x y w d - area on screen used to exit the program. * * java euan eua 100 100 10 10 - if this area changes, end the script. e.g cmd maximised or editor maximised. * * * Inspired by: * https://alvinalexander.com/java/java-robot-class-example-mouse-keystroke * http://www.dougrice.plus.com/hp/Theory/Index.htm * https://docs.oracle.com/javase/7/docs/api/java/awt/Robot.html * * Possible ideas considered:- * * Grab small area of screen and watch the screen for a Change. * Exit on screen change. * * Extend this to poll multiples areas and pixels. * * A simple real time system works by a simple loop. * * init() * while( true ){ * poll_and_make_events_and_set_InterruptFlags * service_events() * sleep() * } * Java robot allows areas of screen to be captured, read colour of pixels, keys to be pressed, move the mouse and press mouse buttons. e.g. // Set up list of images Rectangle rect = new Rectangle( rx , ry , rw , rh ); BufferedImage firstImage = robot.createScreenCapture( rect ); BufferedImage image = robot.createScreenCapture( rect ); // set up a rectangle to be used to take a snapshot of where the mouse has been moved to. Rectangle rect_m = new Rectangle( rx , ry , rw , rh ); BufferedImage image_m = robot.createScreenCapture( rect_m ); Unfortunately, they are screen co-ordinate and it is difficult to map application GUI co-ordinates to Screen Co-ordinates. poll_and make_events:- monitor many areas and generate events. When small areas of screen change, generate events. Watch icons for change and buttons being disabled. Saves screen capture as .png Also consider analysing a small image captured off the screen We need to process it to classify it. We need to do some image arithmatics. Compare the middle row of pixels between first capture of the image and current image to work out dImage/dt. Other Tests: Is the image blank? For each pixel, are they equal? Is the image complex? For each pixel, work out the mean, standard deviation Is the image complex? Each Pixel is RGB, Is the image dark? For each pixel sum the pixels. What colour? for each PIXEL work out which is max R,G or B and report R G B = " | diffs: " + (r-g) + ", " + (g-b) + ", " + (b-r) * * * * * * * * * This program can grab some of the screen, but does not know what it is. The window is focused, but we assume it is where we think it is. If it moves we have problems. We could check if it is where we think it is with a special test. Look for the edges. Grab a thin letterbox and from the center look for the edges. Grab a thin Portrate and from the center look for the edges. Using these coordinates grab the image of interest. define the areas to watch. wait till an area changes and return. capture before and after screen shots Issues: eua starts in middle of the screen eua is different on Linux and Windows so mouse positions vary across installations Need some deadmans handle if user moves mouse. If the mouse is not used, ensure EUA is focused. If the mouse position moves unexpectedly, disable mouse and keypresses. If the mouse position remains stable, click on EUA titlebar to focus it. Need to disable mouse and key presses if EUA is not in focus. When focused header is different to unfocused. Java object references. To take the differential we use lastX = x; dX/dt = ( x-lastX); There is a problem with java when copying object references. a = new Object b = a a = new Object now a and B point to same object 2) base point One of the problems is that the EUA starts in the middle of the screen. Screens are different a method to centralise is required. */ /* ==================================================================== SEC: x.x imports ===================================================================== */ /* ==================================== */ import java.io.*; import java.awt.AWTException; import java.awt.Robot; import java.awt.event.InputEvent; import java.awt.event.KeyEvent; /* ==================================== */ import java.awt.Color; import java.awt.Rectangle; import java.awt.image.BufferedImage; /* ==================================== */ import javax.imageio.*; import java.awt.MouseInfo; import java.awt.Point; import java.awt.PointerInfo; /* ==================================== */ //https://docs.oracle.com/javase/7/docs/api/java/awt/GraphicsDevice.html /* ==================================== */ /* ==================================================================== SEC: x.x class ===================================================================== */ public class euan { Robot robot = new Robot(); /* use PrtScrn and paint to get co-ordinates */ /* base point:282,115 top left corner on 1366x728 */ /* base point:399,161 top left corner on 1600x900 */ //static int lx =( 282 ); //static int ly =( 115 ); /* assume NGT Lite is running in top left corner, at 0,0 */ static Point base = new Point(0,0); // Top left of EUA GUI window int ic = 0 ; boolean d = true; int count =0; int count_images =1; int count_pixels =1; int count_buttons =1; int i ; /* does not work GraphicsDevice gd = MouseInfo.getDevice(); */ //PointerInfo PointerInfo info = MouseInfo.getPointerInfo(); Point p = info.getLocation(); Point last_p = new Point(0,0); int safeCounter = 15; // if mouse have moves by use set safeCounter. int safeCounterInit = 15; // if mouse have moves by use set safeCounter. boolean safeMouseMove = false; // if script moves mouse set this String flags; /* * * Watch these rectangles on the screen to see if they change. * Make an event out of it, * use a Flag to indicate an ISR "interrupt service routine" needs to be run * Get an ISR interrupt service routine to handle the events. * */ // need #define for number of polled areas Rectangle[] rects = new Rectangle[ 20 ] ; BufferedImage[] firstImages = new BufferedImage[ 20 ]; BufferedImage[] lastImages = new BufferedImage[ 20 ]; BufferedImage[] images = new BufferedImage[ 20 ]; Point[] polledPixels = new Point[ 20 ] ; // Points on the screen. Color[] polledPixelsColour = new Color[ 20*2 ] ; // Color getPixelColor(int x, int y) Point[] buttonPoints = new Point[ 20 ] ; // Points on the buttons. //Returns the color of a pixel at the given screen coordinates. // set up a list of pixels to monitor /* set up a rectangle to be used to take a snapshot of where the mouse has been moved to */ Rectangle rect_m = new Rectangle( 0, 0, 10, 10 ); BufferedImage image_m = robot.createScreenCapture( rect_m ); /* define where to click buttons here */ /*base_set( 554, 294 );*/ /* these are updated later */ Point titleBarHome = new Point( 0,0); // Park mouse at left of title bar - useful to check Registration of the GUI on the screen. /************************************************************** * button co-ordinates used for mouse clicks **************************************************************/ Point buttonJoin = new Point( 0,0 ); // JoinButton Point buttonReconnect = new Point( 0,0 ); //791,600,116,22 ); // 8- reconnect button Point buttonCalls = new Point( 0,0 ); // Call Button int x; int y; Color cmd; Color colourAtMouse; int matchRank = 0; int matchRank2 = 0; int[] matchRanks = new int[ 20 ]; int[] lastMatchRanks = new int[ 20 ]; boolean[] matchRankFlags = new boolean[ 20 ]; public static void main(String[] args) throws AWTException { System.out.println( "usage: java GrabsScreen img x y w h - waits for image to change" ); System.out.println( "usage: java GrabsScreen - mouse co-ordinates" ); System.out.println(" Make sure EUA is positioned at the top left of the screen..." ); /* if no parameters show mouse co-ords */ if ( args.length > 4 ) { base.x = Integer.parseInt(args[2]); base.y = Integer.parseInt(args[3]); System.out.println("starting...."); System.out.println("args[0]: " + args[0] ); System.out.println("args[1]: " + args[1] ); System.out.println("args[2]: " + args[2] ); System.out.println("args[3]: " + args[3] ); System.out.println("args[4]: " + args[4] ); new euan( args[0], Integer.parseInt(args[1]), Integer.parseInt(args[2]), Integer.parseInt(args[3]), Integer.parseInt(args[4]) ); } else { /* start up */ new euan( "eua", 0,0,20,20); } } public euan( String fileName , int rx, int ry, int rw, int rh ) throws AWTException { robot.setAutoDelay(40); robot.setAutoWaitForIdle(true); /* set up a rectangle to be used to take a snapshot of where the mouse has been moved to */ Rectangle rect_m = new Rectangle( 0, 0, 10, 10 ); BufferedImage image_m = robot.createScreenCapture( rect_m ); Color cmd; Color colourAtMouse; int[] matchRanks = new int[ 20 ]; int[] lastMatchRanks = new int[ 20 ]; boolean[] matchRankFlags = new boolean[ 20 ]; boolean[] polledPixelsFlag = new boolean[ 20 ]; /* Set up list of images */ Rectangle rect = new Rectangle( rx , ry , rw , rh ); BufferedImage firstImage = robot.createScreenCapture( rect ); BufferedImage image = robot.createScreenCapture( rect ); /* save image */ saveImage( fileName , image ); /* options rectangle */ count = 1; /* * * Watch these rectangles on the screen to see if they change. * Make an event out of it, * use a Flag to indicate an ISR "interrupt service routine" needs to be run * Get an ISR interrupt service routine to handle the events. * */ /************************************************************** * Rectangles to be captured and monitored for changed **************************************************************/ /* use PrtScrn and paint to get co-ordinates */ /*base:399,161 */ /* cropped. 0,0 = top left of EUA. */ count = 1; rects[ count++ ] = new Rectangle(8,34, 34,24); // 1- option rects[ count++ ] = new Rectangle(758,34, 34,27); // 2- calls rects[ count++ ] = new Rectangle(174,129, 454,279); // 3- Calls Dialogue rects[ count++ ] = new Rectangle(386,34, 30,30); // 4- green rects[ count++ ] = new Rectangle(228,191, 350,155); // 5- Reconnect Dialogue box rects[ count++ ] = new Rectangle(128,99, 541,22); // 6- Options Dialog titles rects[ count++ ] = new Rectangle(129,61, 544,416); // 7- Options Dialog area rects[ count++ ] = new Rectangle(239,306, 116,22); // 8- reconnect button rects[ count++ ] = new Rectangle(342, 3, 121,15); // 9- watch the screen title to see if there is a move of the screen. rects[ count++ ] = new Rectangle(350,495, 104,20); // 10 - mute button rects[ count++ ] = new Rectangle(697,495, 97,22); // 11 - keypad button rects[ count++ ] = new Rectangle(187,367, 118,26); // 12 join rects[ count++ ] = new Rectangle(500,367, 116,25); // 13 cancel /*base:399,161 */ /* cropped. 0,0 = top left of EUA. */ count_pixels=1; polledPixels[ count_pixels++ ] = new Point(407,52 ); // 1 polledPixels[ count_pixels++ ] = new Point( 3, 3 ); // 2 Park mouse /************************************************************** * button co-ordinates used for mouse clicks **************************************************************/ polledPixels[ count_pixels++ ] = new Point(245,380 ); // 3 JoinButton polledPixels[ count_pixels++ ] = new Point(300,320 ); // 4 Reconnect button polledPixels[ count_pixels++ ] = new Point(775, 48 ); // 5 Call button /* adjust the positions, add position of top left corner */ for( i=1 ; i < count ; i++){ rects[ i ].x += base.x; rects[ i ].y += base.y; System.out.println( "rects[ count++ ] = new Rectangle(" + rects[ i ].x + "," + rects[ i ].y + "," + rects[ i ].width + "," + rects[ i ].height + ");// " ); } for( i=1 ; i < count_pixels ; i++){ polledPixels[ i ].x += base.x; polledPixels[ i ].y += base.y; System.out.println( "polledPixels[ count_pixels++ ] = new Point(" + polledPixels[ i ].x + "," + polledPixels[ i ].y + " ); // " ); } System.out.println( "count:" + count +" count_pixels:" + count_pixels ); // Add another rectangle of the Join Button in grey state. /* base_set( 554, 294 );*/ /* base_set( 290, 100 ); */ /* 399,161 */ /************************************************************** * Sample the defined rectangles **************************************************************/ for( i=1 ; i < count ; i++){ firstImages[i] = robot.createScreenCapture( rects[i] ); lastImages[i] = robot.createScreenCapture( rects[i] ); saveImage( (fileName+"_"+i) , firstImages[i] ); } /************************************************************** * Sample the defined pixels **************************************************************/ for( i=1 ; i < count_pixels ; i++){ int x; int y; x = polledPixels[ i ].x; y = polledPixels[ i ].y; polledPixelsColour[ 2*i ] = robot.getPixelColor( x, y ); polledPixelsColour[ 2*i + 1 ] = robot.getPixelColor( x, y ); } /* define where to click buttons here */ /*base_set( 554, 294 );*/ count_pixels = 2; Point titleBarHome = polledPixels[ count_pixels++ ]; // Park mouse at left of title bar - useful to check Registration of the GUI on the screen. /************************************************************** * button co-ordinates used for mouse clicks **************************************************************/ Point buttonJoin = polledPixels[ count_pixels++ ]; // JoinButton Point buttonReconnect = polledPixels[ count_pixels++ ]; // Reconnect button Point buttonCalls = polledPixels[ count_pixels++ ]; // Call Button /* we may want to sample the screens in know states so we can test if an icon is displayed */ /* activate Calls and sample Join Button when grey*/ /* Join Call - click Join Button*/ x=buttonCalls.x; y=buttonCalls.y; robot.mouseMove( x,y); robot.delay(500); leftClick(); robot.delay(500); Rectangle rect_join_grey = rects[ 8 ] ; // 14 - Join Button - grey System.out.println( rect_join_grey ); BufferedImage firstImage_join_grey = robot.createScreenCapture( rect_join_grey ); BufferedImage lastImage_join_grey = robot.createScreenCapture( rect_join_grey ); robot.delay(500); type( KeyEvent.VK_ESCAPE ); // int matchRank = testImage( image ); matchRank = testImage( image, firstImage ); System.out.println( "M:" + matchRank ); //reportImage( image ); System.out.println( "count:" + count ); /* Run a timer to disable key and mouse presses if the mouse position has been moved by the user instead of the script. If the mouse position is stable, then enable keypresses and mouse moves If the script moves the mouse disable triggering this, or shorten the timeout. */ matchRank2 = testImage( image, firstImage ); matchRank2 = 0; while( matchRank2 == 0 ){ image = robot.createScreenCapture( rect ); /************************************************************** * Safety feature - disable the automated mouse moves and clicks and key presses if the user moves the mouse. **************************************************************/ /* get pointer position to help */ info = MouseInfo.getPointerInfo(); p = info.getLocation(); colourAtMouse = robot.getPixelColor(p.x,p.y); //System.out.println( " [] "+colourAtMouse ); if ( ( p.x != last_p.x ) || ( p.y != last_p.y ) ){ /* mouse moved possibly by user, so disable events */ if ( safeMouseMove ){ safeCounter = 0; } else { safeCounter = safeCounterInit; } } safeMouseMove = false; /************************************************************** * Safety feature - Monitor a screen area and end script if it changes. **************************************************************/ /* capture the area where the mouse is pointing at */ /* This was to try out an idea of using colour to say where to move to */ /* rect_m = new Rectangle( p.x , p.y , 50 , 10 ); image_m = robot.createScreenCapture( rect_m ); cmd = colourMouse( image_m ); robot.mouseMove( cmd.getRed(), cmd.getGreen() ); */ /* print out the mouse position */ /* consider using pause to generate event to sample the mouse and output rectangle co-ordinates */ if ( safeCounter==0 ){ System.out.print( "--- "); } else { System.out.print( "-!!!: " + safeCounter ); } //System.out.println( " @ " + p.x + "," + p.y + " last:"+ last_p.x + "," + last_p.y +" w:" + (p.x-last_p.x) + ",h:" + (p.y-last_p.y) + " -------------------" ); last_p.x = p.x; last_p.y = p.y; /************************************************************** * Poll screen areas and test for changes. * If an area has changed set a flag. * The Service routine resets these flags **************************************************************/ /* disable event processing if the mouse has been moved by the user, so the user can kill the script */ for( i = 1; i < count ; i++){ /* https://docs.oracle.com/javase/7/docs/api/java/awt/image/BufferedImage.html */ /* save image for dImage/dt */ /* how do you copy the objects in java ?*/ //lastImages[ i ] = images[ i ].getSubimage(0,0,images[ i ].getWidth(),images[ i ].getHeight()); // getSubimage(int x, int y, int w, int h) images[ i ] = robot.createScreenCapture( rects[ i ] ); } if( safeCounter == 0 ) { for( i = 1; i < count ; i++){ /* https://docs.oracle.com/javase/7/docs/api/java/awt/image/BufferedImage.html */ /* save image for dImage/dt */ /* how do you copy the objects in java ?*/ //lastImages[ i ] = images[ i ].getSubimage(0,0,images[ i ].getWidth(),images[ i ].getHeight()); // getSubimage(int x, int y, int w, int h) images[ i ] = robot.createScreenCapture( rects[ i ] ); matchRanks[ i ] = testImage( images[ i ], firstImages[ i ] ); //matchRanks[ i ] = testImage( images[ i ], lastImages[ i ] ); /* images are not equal */ /* testImage returns differences from when the program was started to now */ /* take the differential */ if ( ( matchRanks[ i ] - lastMatchRanks[ i ] ) > 0 ){ matchRankFlags[ i ] = true; /* Interupt Service Routines have to poll these and reset these flags */ } lastMatchRanks[ i ] = matchRanks[ i ]; } /* is Join Call Grey displayed */ lastImage_join_grey = robot.createScreenCapture( rect_join_grey ); matchRank = testImage( firstImage_join_grey, lastImage_join_grey ); /************************************************************** * Poll screen Pixles and test for changes. * If a monitored pixel has changed, set a flag. * The Service routine resets these flags **************************************************************/ for( i=1 ; i < count_pixels ; i++){ int x; int y; x = polledPixels[ i ].x; y = polledPixels[ i ].y; /* in java objects are created on the heap and the line below is a reference */ polledPixelsColour[ 2*i+1 ] = polledPixelsColour[ 2*i ]; polledPixelsColour[ 2*i ] = robot.getPixelColor( x, y ); //System.out.println( "polledPixel:" + polledPixelsColour[ 2*i ] + "," + polledPixelsColour[ 2*i +1 ] ); /* if the colours are not equal raise a flag */ if ( ! polledPixelsColour[ 2*i ].equals( polledPixelsColour[ 2*i + 1 ] ) ) { polledPixelsFlag[ i ] = true; }; } } /********************************************************************** * Safety feature - if mouse moved by user safeCounter set to disable events. **********************************************************************/ /* if mouse moved by user safeCounter set to disable events. */ if( safeCounter > 0 ) { /* try and click on GUI to focus it */ if( safeCounter == 1 ) { /*move the mouse around to each rectangle defined to see if any rouge rectangles defined.*/ /* for( i = 1; i < count ; i++){ robot.mouseMove( rects[i].x, rects[i].y ); robot.delay(250); }; */ /* Park the mouse to the top left of the title bar */ robot.mouseMove(titleBarHome.x, titleBarHome.y); robot.delay(500); /* click it to focus the GUI */ leftClick(); robot.delay(500); safeMouseMove = true; } safeCounter--; } robot.delay(500); /************************************************************** * Poll Interupt Flags and display Flags **************************************************************/ flags = ""; for( i = 1; i < count ; i++){ if( matchRankFlags[ i ] == true ){ flags += " "+i; } else { flags += " _"; } } flags += "|"; for( i = 1; i < count_pixels ; i++){ if( polledPixelsFlag[ i ] == true ){ flags += " "+i; } else { flags += " _"; } } flags += "|"; flags += isPixelGreen( polledPixelsColour[2*1] ); flags += " "; flags += isPixelRed( polledPixelsColour[2*1] ); flags += "|"; flags += "|(" + p.x + "," + p.y +")|("+ (p.x-base.x) + "," + (p.y-base.y) + ")" ; System.out.println( flags ); /************************************************************** * Poll Interupt Flags and run routine to service each Interupt. **************************************************************/ for( i = 1; i < count ; i++){ if( matchRankFlags[ i ] == true ){ /* appended to the image file name*/ //count_images++; matchRankFlags[ i ] = false; // System.out.println( "ISR[" +i +"]:"+matchRanks[ i ] +","+ lastMatchRanks[ i ]); saveImage( (fileName+"_"+i+"_end_"+count_images ) , images[i] ); if( i==1 ){ System.out.println( "ISR[" +i +"] Options.. Called Ended! "); } if ( ( i==2 )||( i==3 ) ){ /* Join Call - click Join Button*/ System.out.println( "ISR[" +i +"] Joining Call... "); x=buttonJoin.x; y=buttonJoin.y; serviceEvent( i, fileName, x,y ); } if( i==6 ){ System.out.println( "ISR[" +i +"] options titles "); } if( i==7 ){ System.out.println( "ISR[" +i +"] options Dialogue "); } if( i==8 ){ System.out.println( "ISR[" +i +"] reconnecting... "); /* reconnect - click Reconnect Button*/ x=buttonReconnect.x; y=buttonReconnect.y; serviceEvent( i, fileName, x,y ); } /* This did not work as it reacted to the focus */ if( i==9 ){ /* We need a way to detect when the window has moved */ System.out.println( "ISR[" +i +"] GUI window moved!!! "); } if( i==10 ){ System.out.println( "ISR[" +i +"] Mute Button "); } if( i==11 ){ System.out.println( "ISR[" +i +"] Keypad button "); } } } for( i = 1; i < count_pixels ; i++){ if( polledPixelsFlag[ i ] == true ){ //System.out.println( "pixel_ISR[" +i +"] raised"); //System.out.println( "**********PixelISR routine running "); polledPixelsFlag[ i ] = false; } } // int matchRank = testImage( image ); matchRank2 = testImage( image, firstImage ); matchRank2 = 0; } /************************************************************** * At the end save a capture of each rectangle defined. **************************************************************/ for( i=1 ; i < count ; i++){ saveImage( (fileName+"_"+i+"_end") , images[i] ); } System.exit(0); } /* ==================================================================== SEC: x.x Service Routines ===================================================================== */ void serviceEvent( int i, String fileName, int x, int y ){ rect_m = new Rectangle( x-25,y-25 , 50 , 50 ); image_m = robot.createScreenCapture( rect_m ); saveImage( ( fileName+"_"+i+"_press_" + count_images ) , image_m ); robot.mouseMove( x,y); robot.delay(500); leftClick(); robot.delay(500); type( KeyEvent.VK_ESCAPE ); /* we need to monitor the Options icon during a call */ safeMouseMove = true; } /* ==================================================================== SEC: x.x methods - images ===================================================================== */ private int testImage( BufferedImage image, BufferedImage firstImage ) { /* * look at images find middle and compare to firstImage. */ int y = image.getHeight() / 2; Color firstC = new Color( image.getRGB( 0, y )); Color c = new Color( image.getRGB( 0 ,y )); int sumEquals = 0; int sumDeltas = 0; /* * to work out match, * work out the difference between * the row of pixels in the middle of the images * If working out dImage/dt we */ for ( y = 0; y < image.getHeight(); y +=3 ) { for (int x = 0; x < image.getWidth(); x++) { firstC = new Color( firstImage.getRGB( x, y ) ); c = new Color( image.getRGB( x, y ) ); /* work out a statistic */ if ( c.equals( firstC ) ) { sumEquals += 1; } else { sumEquals += -1; }; /* work out dImage/dt */ /* image - firstImage */ if ( !c.equals( firstC ) ) { sumDeltas += 1; }; // System.out.println( x + "," + y + "," + c.equals( firstC ) ); } } //return sumEquals; return sumDeltas; } boolean isPixelRed( Color pxl ){ int r = pxl.getRed(); int g = pxl.getGreen(); int b = pxl.getBlue(); //System.out.println( r+"|"+g+"|"+b+"|"+pxl); return (( r > g ) && ( r > b )); }; boolean isPixelGreen( Color pxl ){ int r = pxl.getRed(); int g = pxl.getGreen(); int b = pxl.getBlue(); //System.out.println( r+"|"+g+"|"+b+"|"+pxl); return (( g > r ) && ( g > b )); }; private Color colourMouse( BufferedImage image ) { /* * look at images find colours and use this to move the mouse to the encoded co-ordinates. */ int y = image.getHeight() / 2; Color cmd = new Color( image.getRGB( 5 , y ) ); Color mx = new Color( image.getRGB( 10 , y ) ); Color my = new Color( image.getRGB( 15 , y ) ); // System.out.println( "cmd: "+ cmd +" mx: " + mx + " my: "+ my ); System.out.print( "cmd: "+ cmd ); return cmd; } private void saveImage( String fileName , BufferedImage image ) { System.out.println( "filename:" + fileName ); try { // retrieve image // File outputfile = new File("saved.png"); File outputfile = new File( fileName + ".png"); ImageIO.write(image, "png", outputfile); } catch (IOException e) { System.out.println("image save failed..." ); } } /* ==================================================================== SEC: x.x methods - keyboard and mouse ===================================================================== */ private void leftClick() { robot.mousePress(InputEvent.BUTTON1_MASK); robot.delay(200); robot.mouseRelease(InputEvent.BUTTON1_MASK); robot.delay(200); } private void type(int i) { robot.delay(10); robot.keyPress(i); robot.keyRelease(i); } private void type(String s) { byte[] bytes = s.getBytes(); for (byte b : bytes) { int code = b; // keycode only handles [A-Z] (which is ASCII decimal [65-90]) if (code > 96 && code < 123) code = code - 32; // robot.delay(40); robot.delay(10); robot.keyPress(code); robot.keyRelease(code); } } private void script() { /* robot.delay(4000); robot.mouseMove(40, 130); robot.delay(500); // select App to bring to the front. leftClick(); robot.delay(500); type("\n\n\n\n"); robot.mouseMove(40, 130); robot.delay(500); leftClick(); robot.delay(500); leftClick(); robot.delay(500); type("Hello, world"); robot.mouseMove(40, 160); robot.delay(500); leftClick(); robot.delay(500); leftClick(); robot.delay(500); type("This is a test of the Java Robot class"); robot.delay(50); type(KeyEvent.VK_DOWN); robot.delay(1000); */ } /* ==================================================================== SEC: x.x methods - Program to log key presses ===================================================================== */ /************************************************ * * This routine reports the cursor position. * * If the mouse pauses, the co-ordinates are output. * * x,y, rel_x, rel_y, @ , base_x,base_y * * It is possible to set a base point, by moving the cursor to 0,0 * then to the new base point * By moving the cursot to the top left or 0,0, * waiting and then to the new point. * * Output the last x,y,w,h * & *************************************************/ public static int mouseCapture( ) { PointerInfo info = MouseInfo.getPointerInfo(); Point p = info.getLocation(); Point last_last_last_p = new Point(0,0); Point last_last_p = new Point(0,0); Point last_p = new Point(0,0); Point base_p = new Point(0,0); Point delta_p = new Point(0,0); Point last_pause_p = new Point(0,0); Point last_delta_p = new Point(0,0); int w = 0; int h = 0; int count = 0; int img_count = 0; boolean base_p_set = false; base_p.x = 0; base_p.y = 0; last_p.x = 0; last_p.y = 0; /* move cursor to point , cannot work out how to press so use time */ /* if mouse stops digitise base point */ /* if mouse pauses digitise delta point */ /* to set new base move mouse to left of screen */ while (true) { last_last_last_p.x = last_last_p.x; last_last_p.x = last_p.x; last_p.x = p.x; last_last_last_p.y = last_last_p.y; last_last_p.y = last_p.y; last_p.y = p.y; count ++; //PointerInfo info = MouseInfo.getPointerInfo(); p = info.getLocation(); if ( p.equals(last_last_p) ){ /* mouse paused so capture */ if ( ( p.x == 0 ) || ( p.y == 0 )) { System.out.format("\n /* Move mouse to base, and wait */\n" ); base_p.x = 0; base_p.y = 0; base_p_set = false ; } /* trigger event */ /* set base */ if ( count == 3 ){ if ( base_p_set == false){ base_p_set = true; base_p.x = p.x; base_p.y = p.y; System.out.format("\n/*base_set( %d, %d );*/\n", base_p.x , base_p.y ); } base_p_set = true; /* output delta */ delta_p.x = ( p.x- base_p.x ); delta_p.y = ( p.y- base_p.y ); //System.out.format("\ndelta:, %d, %d\n", delta_p.x , delta_p.y ); //System.out.println( "delta:" + p.x+","+p.y+" :"+delta_p.x + "," + delta_p.y +" @" + base_p.x +"," + base_p.y ); w = (delta_p.x-last_delta_p.x); h = (delta_p.y-last_delta_p.y); /* System.out.println( " delta(" + p.x+","+p.y+", "+delta_p.x + "," + delta_p.y + ","+ base_p.x +"," + base_p.y +","+ // img_count + ","+ last_delta_p.x +"," + last_delta_p.y +","+ w + "," + h + "," + ");" ); */ last_delta_p.x = delta_p.x ; last_delta_p.y = delta_p.y ; w = ( p.x - last_pause_p.x ); h = ( p.y - last_pause_p.y ); System.out.println( " rects[ count++ ] = new Rectangle( " + last_pause_p.x + "," + last_pause_p.y + "," + w + "," + h + " ); " ); last_pause_p.x = p.x; last_pause_p.y = p.y; } /* work out width and height of last rectangle */ w = (delta_p.x-last_delta_p.x); h = (delta_p.y-last_delta_p.y); /* if rect is well formed, take a snapshot */ if ( ( w > 0 ) && ( h > 0 ) ){ /* does not work yet, static reported rect = new Rectangle( last_delta_p.x, last_delta_p.y , w , h ); image = robot.createScreenCapture( rect ); saveImage( (fileName+img_cnt) , image ); */ } //System.out.format("*" ); } else { count = 0; } //robot.delay(500); sleep(); } } private static void sleep() { try { Thread.sleep(500); } catch (InterruptedException e) { e.printStackTrace(); } } private static void delta( int x, int y, int delta_x, int delta_y, int base_x, int base_y, int rx,int ry , int w, int h ){ } private static void deltas() { } }