[Snippets]C# - 仿sikuli式的利用擷取圖片做按鍵精靈
最近有個需求是得去點擊某個程式的一些按鈕，但寫死座標位置使程式沒有彈性不是我的風格，最後就決定結合影像技術的方向來做一個仿sikuli利用截圖來點擊的偽按鍵精靈。以下是一些完成這目的的片段程式碼：
Screenshot
```int screenWidth = Screen.GetBounds(new Point(0, 0)).Width;
int screenHeight = Screen.GetBounds(new Point(0, 0)).Height;
Bitmap bmpScreenShot = new Bitmap(screenWidth, screenHeight); // the final image used by memory reference
Graphics gfx = Graphics.FromImage((Image)bmpScreenShot);
gfx.CopyFromScreen(0, 0, 0, 0, new Size(screenWidth, screenHeight));
最後直接使用或回傳bmpScreenShot即可。<!--more-->
<h2><span style="color: #3366ff;">Extract SubImage for Button</span></h2>
Bitmap croppedImage = originalBitmap.Clone(theRect, originalBitmap.PixelFormat);
<h2><span style="color: #3366ff;">Mouse Click</span></h2>
一開始先設定引用一些dll與外部函式，將其放在class內與所有函式最外面：
[DllImport("user32.dll", CharSet = CharSet.Auto, CallingConvention = CallingConvention.StdCall)]
public static extern void mouse_event(uint dwFlags, uint dx, uint dy, uint cButtons, uint dwExtraInfo);
[DllImport("User32")]
public static extern void SetCursorPos(int x, int y); [DllImport("USER32.DLL")]</pre>
private const int MOUSEEVENTF_LEFTDOWN = 0x02;
private const int MOUSEEVENTF_LEFTUP = 0x04;
private const int MOUSEEVENTF_RIGHTDOWN = 0x08;
private const int MOUSEEVENTF_RIGHTUP = 0x10;
點擊的方式這裡是先將滑鼠移到該位置後再做點擊的動作：
SetCursorPos(posX, posY);
mouse_event(MOUSEEVENTF_LEFTDOWN | MOUSEEVENTF_LEFTUP, 0, 0, 0, 0); // dx, dy from current mouse position
<h2><span style="color: #3366ff;">Keyboard Pressing</span></h2>
一樣先引入一些dll與外部函式的定義：
[DllImport("USER32.DLL")]
public static extern IntPtr FindWindow(string lpClassName, string lpWindowName);
[DllImport("USER32.DLL")]
public static extern bool SetForegroundWindow(IntPtr hWnd);
點擊按鈕，相關對應按鈕代碼可參照<a href="http://msdn.microsoft.com/zh-tw/library/system.windows.forms.sendkeys(v=vs.80).aspx" target="_blank">官方reference</a>。
IntPtr calculatorHandle = FindWindow(null, "提示");
// 先找到你要送 key message 到哪個視窗，第二個參數給視窗的名稱即可
if (calculatorHandle == IntPtr.Zero) // 這裡檢查有沒有找到
{
return;
}
SetForegroundWindow(calculatorHandle); // 然後 focus 在你要送的視窗是
SendKeys.SendWait("{ENTER}");
註：當然若你不想指定視窗，想在當下最前面的視窗按鍵盤，那麼不用去引入些dll，只要呼叫SendKeys即可。
<h2><span style="color: #3366ff;">SubImage Finding</span></h2>
public static List<Point> GetSubPositions(Bitmap main, Bitmap sub)
{
List<Point> possiblepos = new List<Point>();
int mainwidth = main.Width;
int mainheight = main.Height;
int subwidth = sub.Width;
int subheight = sub.Height;
int movewidth = mainwidth - subwidth;
int moveheight = mainheight - subheight;
BitmapData bmMainData = main.LockBits(new Rectangle(0, 0, mainwidth, mainheight), ImageLockMode.ReadWrite, PixelFormat.Format32bppArgb);
BitmapData bmSubData = sub.LockBits(new Rectangle(0, 0, subwidth, subheight), ImageLockMode.ReadWrite, PixelFormat.Format32bppArgb);
int bytesMain = Math.Abs(bmMainData.Stride) * mainheight;
int strideMain = bmMainData.Stride;
System.IntPtr Scan0Main = bmMainData.Scan0;
byte[] dataMain = new byte[bytesMain];
System.Runtime.InteropServices.Marshal.Copy(Scan0Main, dataMain, 0, bytesMain);
int bytesSub = Math.Abs(bmSubData.Stride) * subheight;
int strideSub = bmSubData.Stride;
System.IntPtr Scan0Sub = bmSubData.Scan0;
byte[] dataSub = new byte[bytesSub];
System.Runtime.InteropServices.Marshal.Copy(Scan0Sub, dataSub, 0, bytesSub);
for (int y = 0; y < moveheight; ++y)
{
for (int x = 0; x < movewidth; ++x)
{
MyColor curcolor = GetColor(x, y, strideMain, dataMain);
foreach (var item in possiblepos.ToArray())
{
int xsub = x - item.X;
int ysub = y - item.Y;
if (xsub >= subwidth || ysub >= subheight || xsub < 0)
continue;
MyColor subcolor = GetColor(xsub, ysub, strideSub, dataSub);
if (!curcolor.Equals(subcolor))
{
possiblepos.Remove(item);
}
}
if (curcolor.Equals(GetColor(0, 0, strideSub, dataSub)))
possiblepos.Add(new Point(x, y));
}
}
System.Runtime.InteropServices.Marshal.Copy(dataSub, 0, Scan0Sub, bytesSub);
sub.UnlockBits(bmSubData);
System.Runtime.InteropServices.Marshal.Copy(dataMain, 0, Scan0Main, bytesMain);
main.UnlockBits(bmMainData);
return possiblepos;
}
private static MyColor GetColor(Point point, int stride, byte[] data)
{
return GetColor(point.X, point.Y, stride, data);
}
private static MyColor GetColor(int x, int y, int stride, byte[] data)
{
int pos = y * stride + x * 4;
byte a = data[pos + 3];
byte r = data[pos + 2];
byte g = data[pos + 1];
byte b = data[pos + 0];
return MyColor.FromARGB(a, r, g, b);
}
struct MyColor
{
byte A;
byte R;
byte G;
byte B;
public static MyColor FromARGB(byte a, byte r, byte g, byte b)
{
MyColor mc = new MyColor();
mc.A = a;
mc.R = r;
mc.G = g;
mc.B = b;
return mc;
}
public override bool Equals(object obj)
{
if (!(obj is MyColor))
return false;
MyColor color = (MyColor)obj;
if (color.A == this.A && color.R == this.R && color.G == this.G && color.B == this.B)
return true;
return false;
}
}
直接呼叫GetSubPositions函式，把要搜尋的畫面image與目標image當做參數傳進去即可得到所有該目標的位置所在。
註：這是網路上提供的方式，是以顏色為基底去做找尋，越複雜的目標圖像準確率會較高，若底色較單純的圖案可能會有找到很多位置的情況，最好選擇的目標image區塊能大點使複雜度提高。最好的方式還是使用影像處理中的<a href="http://en.wikipedia.org/wiki/Template_matching" target="_blank">template matching</a>，在matlab有現成函式可以使用，但C#就得自己刻了。
## ----- 20120916 updated -----
以下新增相關程式碼，即利用<a href="http://code.google.com/p/aforge/">Aforge</a>與<a href="http://file.emgu.com/wiki/index.php/Main_Page">EmguCV</a>做到template matching的功能。
<h2><span style="color: #3366ff;">Aforge</span></h2>
使用方式：下載aforge相關<a href="http://code.google.com/p/aforge/downloads/list">dll</a>、加入相關dll至專案參考內、在程式碼最上方加入"using Aforge.Imaging"即可。
List<Point> template_matching(Bitmap s, Bitmap t)
{
Bitmap sourceImage = (Bitmap)s.Clone(new Rectangle(0, 0, s.Width, s.Height), PixelFormat.Format24bppRgb);
Bitmap template = (Bitmap)t.Clone(new Rectangle(0, 0, t.Width, t.Height), PixelFormat.Format24bppRgb);
ExhaustiveTemplateMatching tm = new ExhaustiveTemplateMatching(0.9f); // 0.9f is score threshold
TemplateMatch[] matchings = tm.ProcessImage(sourceImage, template);
List<Point> result = new List<Point>();
foreach (TemplateMatch matching in matchings)
{
result.Add(matching.Rectangle.Location);
}
return result;
}
註：aforge的template matching運算很久，在一般配備的雙核心＋2g ram下需約跑20秒以上。但好處是不用額外安裝opencv, emgucv等的套件。
<h2><span style="color: #3366ff;">EmguCV</span></h2>
使用方法：安裝<a href="http://sourceforge.net/projects/opencvlibrary/files/opencv-win/2.2/">opencv2.2</a>、安裝<a href="http://sourceforge.net/projects/emgucv/files/emgucv/2.2.0.0/">emgucv2.2</a>、加入相關dll參考至專案中（Emgu.CV, Emgu.Util）、引入namespace參考（using ...）。另可參考本站有關在vs2010使用emgucv的<a href="http://blog.hothero.org/178/opencv-on-vs2010-csharp-using-emgucv">舊文</a>。
List<Point> template_matching(Bitmap s, Bitmap t)
{
List<Point> result = new List<Point>();
Image<Bgr, Byte> sourceImage = new Image<Bgr, Byte>(s);
Image<Bgr, Byte> templateImage = new Image<Bgr, Byte>(t);
Image<Gray, float> resultImage = sourceImage.MatchTemplate(templateImage,Emgu.CV.CvEnum.TM_TYPE.CV_TM_CCOEFF_NORMED);
float[,,] matches = resultImage.Data;
for (int x = 0; x < matches.GetLength(1); x++)
{
for (int y = 0; y < matches.GetLength(0); y++)
{
double matchScore = matches[y, x, 0];
if (matchScore > 0.95)
{
result.Add(new Point(x, y));
}
}
}
return result;
}
註：emgucv的template matching執行約1秒，且很準確。唯一缺點就是需要在執行機器上額外安裝opencv。BTW, 若有相關執行問題，或dll參考失敗，只要把emgu.cv, emgu.util的dll複製至與執行檔相同路徑即可。
*此篇主要著重在程式碼記錄，若有任何不清楚的歡迎留言詢問。
<h2><span style="color: #3366ff;">Reference</span></h2>
<ul>
<li><span style="line-height: 22px;"><a title="find-subimage-in-larger-image-in-c-sharp" href="http://stackoverflow.com/questions/11343772/find-subimage-in-larger-image-in-c-sharp" target="_blank">Find Subimage in Large Image in C#</a></span></li>
<li><a href="http://stackoverflow.com/questions/11457679/extract-sub-image-from-an-image-using-c-sharp" target="_blank">Extract sub image from an image using C#</a></li>
<li><a href="http://roronoa.pixnet.net/blog/post/25312590-c%23-%3A-%E6%A8%A1%E6%93%AC%E6%BB%91%E9%BC%A0%E9%8D%B5%E7%9B%A4%E5%8B%95%E4%BD%9C" target="_blank">模擬滑鼠鍵盤動作</a></li>
</ul>