/ image

[Snippets]C# - 仿sikuli式的利用擷取圖片做按鍵精靈

最近有個需求是得去點擊某個程式的一些按鈕,但寫死座標位置使程式沒有彈性不是我的風格,最後就決定結合影像技術的方向來做一個仿sikuli利用截圖來點擊的偽按鍵精靈。以下是一些完成這目的的片段程式碼:

Screenshot

```

int screenWidth = Screen.GetBounds(new Point(0, 0)).Width;

int screenHeight = Screen.GetBounds(new Point(0, 0)).Height;

Bitmap bmpScreenShot = new Bitmap(screenWidth, screenHeight);   // the final image used by memory reference

Graphics gfx = Graphics.FromImage((Image)bmpScreenShot);

gfx.CopyFromScreen(0, 0, 0, 0, new Size(screenWidth, screenHeight));


最後直接使用或回傳bmpScreenShot即可。<!--more-->
<h2><span style="color: #3366ff;">Extract SubImage for Button</span></h2>

Bitmap croppedImage = originalBitmap.Clone(theRect, originalBitmap.PixelFormat);

<h2><span style="color: #3366ff;">Mouse Click</span></h2>
一開始先設定引用一些dll與外部函式,將其放在class內與所有函式最外面:

[DllImport("user32.dll", CharSet = CharSet.Auto, CallingConvention = CallingConvention.StdCall)]

public static extern void mouse_event(uint dwFlags, uint dx, uint dy, uint cButtons, uint dwExtraInfo);

[DllImport("User32")]

public static extern void SetCursorPos(int x, int y); [DllImport("USER32.DLL")]</pre>
private const int MOUSEEVENTF_LEFTDOWN = 0x02;

private const int MOUSEEVENTF_LEFTUP = 0x04;

private const int MOUSEEVENTF_RIGHTDOWN = 0x08;

private const int MOUSEEVENTF_RIGHTUP = 0x10;


點擊的方式這裡是先將滑鼠移到該位置後再做點擊的動作:

SetCursorPos(posX, posY);

mouse_event(MOUSEEVENTF_LEFTDOWN | MOUSEEVENTF_LEFTUP, 0, 0, 0, 0);   // dx, dy from current mouse position

<h2><span style="color: #3366ff;">Keyboard Pressing</span></h2>
一樣先引入一些dll與外部函式的定義:

[DllImport("USER32.DLL")]

public static extern IntPtr FindWindow(string lpClassName, string lpWindowName);

[DllImport("USER32.DLL")]

public static extern bool SetForegroundWindow(IntPtr hWnd);


點擊按鈕,相關對應按鈕代碼可參照<a href="http://msdn.microsoft.com/zh-tw/library/system.windows.forms.sendkeys(v=vs.80).aspx" target="_blank">官方reference</a>。

IntPtr calculatorHandle = FindWindow(null, "提示");

// 先找到你要送 key message 到哪個視窗,第二個參數給視窗的名稱即可

if (calculatorHandle == IntPtr.Zero)  // 這裡檢查有沒有找到

{

return;

}

SetForegroundWindow(calculatorHandle); // 然後 focus 在你要送的視窗是

SendKeys.SendWait("{ENTER}");


註:當然若你不想指定視窗,想在當下最前面的視窗按鍵盤,那麼不用去引入些dll,只要呼叫SendKeys即可。
<h2><span style="color: #3366ff;">SubImage Finding</span></h2>

public static List<Point> GetSubPositions(Bitmap main, Bitmap sub)

{

List<Point> possiblepos = new List<Point>();

int mainwidth = main.Width;

int mainheight = main.Height;

int subwidth = sub.Width;

int subheight = sub.Height;

int movewidth = mainwidth - subwidth;

int moveheight = mainheight - subheight;

BitmapData bmMainData = main.LockBits(new Rectangle(0, 0, mainwidth, mainheight), ImageLockMode.ReadWrite, PixelFormat.Format32bppArgb);

BitmapData bmSubData = sub.LockBits(new Rectangle(0, 0, subwidth, subheight), ImageLockMode.ReadWrite, PixelFormat.Format32bppArgb);

int bytesMain = Math.Abs(bmMainData.Stride) * mainheight;

int strideMain = bmMainData.Stride;

System.IntPtr Scan0Main = bmMainData.Scan0;

byte[] dataMain = new byte[bytesMain];

System.Runtime.InteropServices.Marshal.Copy(Scan0Main, dataMain, 0, bytesMain);

int bytesSub = Math.Abs(bmSubData.Stride) * subheight;

int strideSub = bmSubData.Stride;

System.IntPtr Scan0Sub = bmSubData.Scan0;

byte[] dataSub = new byte[bytesSub];

System.Runtime.InteropServices.Marshal.Copy(Scan0Sub, dataSub, 0, bytesSub);

for (int y = 0; y < moveheight; ++y)

{

for (int x = 0; x < movewidth; ++x)

{

MyColor curcolor = GetColor(x, y, strideMain, dataMain);

foreach (var item in possiblepos.ToArray())

{

int xsub = x - item.X;

int ysub = y - item.Y;

if (xsub >= subwidth || ysub >= subheight || xsub < 0)

continue;

MyColor subcolor = GetColor(xsub, ysub, strideSub, dataSub);

if (!curcolor.Equals(subcolor))

{

possiblepos.Remove(item);

}

}

if (curcolor.Equals(GetColor(0, 0, strideSub, dataSub)))

possiblepos.Add(new Point(x, y));

}

}

System.Runtime.InteropServices.Marshal.Copy(dataSub, 0, Scan0Sub, bytesSub);

sub.UnlockBits(bmSubData);

System.Runtime.InteropServices.Marshal.Copy(dataMain, 0, Scan0Main, bytesMain);

main.UnlockBits(bmMainData);

return possiblepos;

}

private static MyColor GetColor(Point point, int stride, byte[] data)

{

return GetColor(point.X, point.Y, stride, data);

}

private static MyColor GetColor(int x, int y, int stride, byte[] data)

{

int pos = y * stride + x * 4;

byte a = data[pos + 3];

byte r = data[pos + 2];

byte g = data[pos + 1];

byte b = data[pos + 0];

return MyColor.FromARGB(a, r, g, b);

}

struct MyColor

{

byte A;

byte R;

byte G;

byte B;

public static MyColor FromARGB(byte a, byte r, byte g, byte b)

{

MyColor mc = new MyColor();

mc.A = a;

mc.R = r;

mc.G = g;

mc.B = b;

return mc;

}

public override bool Equals(object obj)

{

if (!(obj is MyColor))

return false;

MyColor color = (MyColor)obj;

if (color.A == this.A && color.R == this.R && color.G == this.G && color.B == this.B)

return true;

return false;

}

}


直接呼叫GetSubPositions函式,把要搜尋的畫面image與目標image當做參數傳進去即可得到所有該目標的位置所在。

註:這是網路上提供的方式,是以顏色為基底去做找尋,越複雜的目標圖像準確率會較高,若底色較單純的圖案可能會有找到很多位置的情況,最好選擇的目標image區塊能大點使複雜度提高。最好的方式還是使用影像處理中的<a href="http://en.wikipedia.org/wiki/Template_matching" target="_blank">template matching</a>,在matlab有現成函式可以使用,但C#就得自己刻了。
## ----- 20120916 updated -----
以下新增相關程式碼,即利用<a href="http://code.google.com/p/aforge/">Aforge</a>與<a href="http://file.emgu.com/wiki/index.php/Main_Page">EmguCV</a>做到template matching的功能。
<h2><span style="color: #3366ff;">Aforge</span></h2>
使用方式:下載aforge相關<a href="http://code.google.com/p/aforge/downloads/list">dll</a>、加入相關dll至專案參考內、在程式碼最上方加入"using Aforge.Imaging"即可。

List<Point> template_matching(Bitmap s, Bitmap t)

{

Bitmap sourceImage = (Bitmap)s.Clone(new Rectangle(0, 0, s.Width, s.Height), PixelFormat.Format24bppRgb);

Bitmap template = (Bitmap)t.Clone(new Rectangle(0, 0, t.Width, t.Height), PixelFormat.Format24bppRgb);

ExhaustiveTemplateMatching tm = new ExhaustiveTemplateMatching(0.9f);   // 0.9f is score threshold

TemplateMatch[] matchings = tm.ProcessImage(sourceImage, template);

List<Point> result = new List<Point>();

foreach (TemplateMatch matching in matchings)

{

result.Add(matching.Rectangle.Location);

}

return result;

}


註:aforge的template matching運算很久,在一般配備的雙核心+2g ram下需約跑20秒以上。但好處是不用額外安裝opencv, emgucv等的套件。
<h2><span style="color: #3366ff;">EmguCV</span></h2>
使用方法:安裝<a href="http://sourceforge.net/projects/opencvlibrary/files/opencv-win/2.2/">opencv2.2</a>、安裝<a href="http://sourceforge.net/projects/emgucv/files/emgucv/2.2.0.0/">emgucv2.2</a>、加入相關dll參考至專案中(Emgu.CV, Emgu.Util)、引入namespace參考(using ...)。另可參考本站有關在vs2010使用emgucv的<a href="http://blog.hothero.org/178/opencv-on-vs2010-csharp-using-emgucv">舊文</a>。

List<Point> template_matching(Bitmap s, Bitmap t)

{

List<Point> result = new List<Point>();

Image<Bgr, Byte> sourceImage = new Image<Bgr, Byte>(s);

Image<Bgr, Byte> templateImage = new Image<Bgr, Byte>(t);

Image<Gray, float> resultImage = sourceImage.MatchTemplate(templateImage,Emgu.CV.CvEnum.TM_TYPE.CV_TM_CCOEFF_NORMED);

float[,,] matches = resultImage.Data;

for (int x = 0; x < matches.GetLength(1); x++)

{

for (int y = 0; y < matches.GetLength(0); y++)

{

double matchScore = matches[y, x, 0];

if (matchScore > 0.95)

{

result.Add(new Point(x, y));

}

}

}

return result;

}


註:emgucv的template matching執行約1秒,且很準確。唯一缺點就是需要在執行機器上額外安裝opencv。BTW, 若有相關執行問題,或dll參考失敗,只要把emgu.cv, emgu.util的dll複製至與執行檔相同路徑即可。

*此篇主要著重在程式碼記錄,若有任何不清楚的歡迎留言詢問。
<h2><span style="color: #3366ff;">Reference</span></h2>
<ul>
	<li><span style="line-height: 22px;"><a title="find-subimage-in-larger-image-in-c-sharp" href="http://stackoverflow.com/questions/11343772/find-subimage-in-larger-image-in-c-sharp" target="_blank">Find Subimage in Large Image in C#</a></span></li>
	<li><a href="http://stackoverflow.com/questions/11457679/extract-sub-image-from-an-image-using-c-sharp" target="_blank">Extract sub image from an image using C#</a></li>
	<li><a href="http://roronoa.pixnet.net/blog/post/25312590-c%23-%3A-%E6%A8%A1%E6%93%AC%E6%BB%91%E9%BC%A0%E9%8D%B5%E7%9B%A4%E5%8B%95%E4%BD%9C" target="_blank">模擬滑鼠鍵盤動作</a></li>
</ul>