[faces] Face detection

---- Reported by shotwell-maint@gnome.bugs 2010-04-05 12:19:00 -0700 ---- Original Redmine bug id: 1702 Original URL: http://redmine.yorba.org/issues/1702 Searchable id: yorba-bug-1702 Original author: Marcus P S Original description:

Face detection libraries, such as Intel's OpenCV (http://sourceforge.net/projects/opencvlibrary/), enable a relatively simple implementation of face recognition in Shotwell. &nbsp_place_holder;Users may then add names to each of the detected faces, and when OpenCV fails to detect a face, the user should be able to indicate where a face is manually.

Another browsing view, similar to the events view, can then be associated with the named faced, and a separate folder can exist with all the unnamed faces.

Users should also have the choice of ignoring faces which are detected (i.e. choosing to never name a face that is detected, and not have it show up in the unnamed faces folder.

This feature is extremely useful for organizing and browsing family pictures.

I am not aware of any reliable open source face recognition libraries, but this should be an easy first step in that direction. The faces folders can later on be used for training the software.

Related issues: related to shotwell - Feature #3111: tags bound to certain area of the image (Duplicate) related to shotwell - Feature #3894: People view (Fixed) ---- Additional Comments From shotwell-maint@gnome.bugs 2013-08-26 13:49:00 -0700 ---- ### History #### #1 Updated by Marcus P S over 3 years ago There is some sample code for using the Python bindings to OpenCV to highlight faces in images http://creatingwithcode.com/howto/face-detection-in-static-images-with-python/ This can be a useful reference for an initial implemetation. #### #2 Updated by Cosmin Cosma over 3 years ago faces is a must for the best photo manager #### #3 Updated by mnemo - over 3 years ago * **Keywords** changed from _face detection opencv_ to _face detection opencv libface eigenfaces_ Aditya Bhatt is working as a GSoC student on “libface” and is adding face detection to digiKam. This code does face detection using “stock haar cascades” implemented by OpenCV just like the Python sample above. More info here: http://adityabhatt.wordpress.com/2010/05/19/yet-another-gsoc-update-libface- gets-face-recognition/ #### #4 Updated by Marcus P S over 3 years ago I would be nice to have a DBus face detection (and eventually recognition) service. That might be a nice project to do with libface, keep things language agnostic to some extent, and separate from Shotwell. #### #5 Updated by zedtux - over 3 years ago Hello all, I'm starting to have a look to implement a new feature that is linked with this ticket. In a first step, I want to implement the possibility to set who is on the picture and where manually (with a shape). If everything go fast and good, I will have a look to all existing library for face recognition and maybe have a try to implement it. So, wait and see :) #### #6 Updated by zedtux - over 3 years ago * **Status** changed from _Open_ to _Review_ * **Assignee** changed from _Anonymous_ to _zedtux -_ I've took time to think about the best way to implement this kind of feature, and found the same idea as [/ticket/1702#comment:5 marcus.ps]: Make it in a D-Bus service ! :) So, I have start to play with OpenCV (and others but opencv was the best), DBus as a server (already have a project as a DBus client). I encountered many problems with some pictures from the web and from personal library, with bas face detection, even changing parameters. I have work a lot about it and have made a first version of my DBus Face Recognition service daemon. I have a create a new project on Launchpad: !https://launchpad.net/quiestla This first version don't really care about fast execution, but much more on having the highest good results in face recognition ! So, I'm working hard on that, and when it will be finished, in a 1.0 version, I will implement it if necessary into Shotwell. Do not hesitate to test it, check the code if you know Python, DBus, OpenCV etc.. and do not hesitate to contact me for suggestions and discussions. Hope it will work and be implemented shortly ! :) #### #7 Updated by zedtux - over 3 years ago Also if you test it now, there are a lot of bugs (faces detected but not well reported) but I will work on it with good test pictures like this one !http://www.ists.dartmouth.edu/images/ISTSpeople.jpg. #### #8 Updated by Adam Dingle over 3 years ago I don't understand the advantage of implementing this as a separate service process. Why not simply call the OpenCV shared library from the Shotwell process directly? A separate process will add an additional dependency, will complicate installation and startup, and will potentially slow things down since we'll need to cross process boundaries with each request. #### #9 Updated by mnemo - over 3 years ago I'm only guessing here but I guess the idea is that if you tag someone as being person X in Shotwell, then other apps could reuse that info and suddenly your browser or e-mail program could also recognize that person in photos? #### #10 Updated by Marcus P S over 3 years ago I can see that it may not be the fastest way to do face recognition, by as mnemo pointed out, it may be a better way to share face recognition and detection across applications. Then again, if the IPC overhead is too large, Shotwell could implement its own face recog/detect and then share the processed information with whatever OS service may be available. My main motivation for the suggestion was to decouple the development of this feature from Shotwell, and let the Shotwell developers concentrate on other more important features. #### #11 Updated by Adam Dingle over 3 years ago marcus.ps, thanks for the explanation. I think there's no need to try to decouple this feature's development from Shotwell. On the contrary, if you or anyone else if interested in working on this I'd like to encourage communication between you and the Shotwell team. Rather than building an external service which we have to glue on later, I'd be inclined to start thinking today about how to call the OpenCV shared library from Shotwell (do we need to build a Vala binding, for instance?) and about how we want face detection to work in the Shotwell user interface. #### #12 Updated by zedtux - over 3 years ago I'm doing it in that way because it will be done one time, and then, nobody else will have to do it again and again (re-writing the wheel). Replying to [comment:9 adam]: > A separate process will add an additional dependency, will complicate installation and startup, and will potentially slow things down since we'll need to cross process boundaries with each request. I don't agree with you. Face recognition will be disabled until the person install the dependence. It's a cool thing, because in that way, users that don't want face recognition just have to do nothing more. For person who want it it's just a search in the packet manager, and press install… I also imagine (always if everything work great etc…) that the service could be integrated in the OS directly. #### #13 Updated by Adam Dingle over 3 years ago I still don't understand the precise benefit you're looking for in implementing a separate server process. You said that “it will be done one time, and then nobody else will have to do it again and again”. Shared libraries have that exact benefit: I can write a shared library once and then many programs can use it. And, conveniently, OpenCV is already a shared library which we can call directly from Shotwell. If you want to write extra functionality to be shared among programs that perform face recognition, you could either write another shared library which calls OpenCV (in fact, I think libface does exactly this), or you could simply submit patches to OpenCV itself. Shotwell already uses shared libraries for lots of functionality: for decoding RAW photos and for displaying Web pages as part of publishing, for example. Do you think those capabilites should have been implemented via separate server processes instead? I agree that it might be nice if people can optionally enable certain features, and we'd like to have a plugin system that allows exactly that (this is #1603). Once we've implemented that, we could conceivably implement face detection as a plugin. I think that face detection will be useful enough, however, that once it's implemented it should be part of the core Shotwell program and available to all users. #### #14 Updated by Marcus P S over 3 years ago Vala bindings for OpenCV would be nice, I think, and I wish I had the time to contribute to a project like that. I did look into, but I don't think it would be something I could take a lead on at this time. #### #15 Updated by zedtux - over 3 years ago I understand your point adam. I think that shared library are here to give you a, as developer, a lot of classes/methods that you can use in many possible ways to do a lot of things. ( Like OpenCV. It don't only detect faces. ) But here, I'm implementing a specific feature that use some of these classes/methods to do it. Now, another application want to have face detection but to do another thing. It can use my work. And having a DBus daemon or a shared library is not big difference. I will continue to do my DBus daemon for now, because I'm learning a lot of thing about picture manipulations, that I never did before. I plan to do a try, implement a Dbus client to use my daemon and make benches. After that, I can write the same code, but directly in Shotwell in Vala. Is this ok for you ? #### #16 Updated by zedtux - over 3 years ago Just for give to all of you a status, I want to post a result of my daemon :) To do my tests I'm taking picture of people on the net. Here is a source: !http://www.dfas.mil/careers/acareeratdfas/comminvolv/communitypeople.jpg And here the end result of my daemon: http://a.yfrog.com/img684/7900/communitypeoplewithface.jpg It's not clean but cool 14/17 :) #### #17 Updated by Adam Dingle over 3 years ago Yes, that plan sounds OK with me. That's great that you already have code which is detecting some faces! #### #18 Updated by zedtux - over 3 years ago Ok, we have a deal :) #### #19 Updated by zedtux - over 3 years ago I have improve the garbage detector and now in my previous example I have 17/17 !! :) http://img33.imageshack.us/img33/7900/communitypeoplewithface.jpg Now I'm working on other sources to improve face detection. #### #20 Updated by zedtux - over 3 years ago Here another example where the first try work great: Source: http://lac.web.psi.ch/Pictures/LAC_people_20080403.jpg With faces areas: http://img682.imageshack.us/img682/5030/lacpeople20080403withfa.jpg #### #21 Updated by Aditya Bhatt over 3 years ago Replying to [comment:21 zedtux]: Hi, nice to hear that you're getting good accuracy. In the last example, can you tell me: 1. The parameter values for detection (search increment, minimum face size used, etc) 2. The number of cascades used. Are you using a custom-trained cascade or opencv's? 3. Time taken for recognition. Does it increase a lot if the image's pixel count increases? Source code would be good :) Thanks. I'm the developer of libface, btw. #### #22 Updated by zedtux - over 3 years ago Hello all. A short message to tell you that I'm in holidays for 3 weeks o/ **@adityabhatt:** I already have post the project's url (https://launchpad.net/quiestla) where you can find the Python code. You will also find all you answer in it. But, basically, I'm using OpenCV to do face detection, with 5 haarcascades XML files. (Here is the core method where opencv face detection method is called http://b azaar.launchpad.net/~zedtux/quiestla/trunk/annotate/head:/quiestla/common/open cvclient.py#L160) The execution time isn't very short until now, because it wasn't the highest priority. Here they are: * Discover and test opencv **Done** * Discover how to implement a D-Bus service **Done** * Have first good results on random pictures from google **Done** * Increase queue and thread use to do more than one task at the same time **Done** * Improve D-Bus service (method and signals) **Done** * Improve execution time **In progress** * Improve results **Waiting** * Implement into Shotwell **Waiting** My last commit was about execution time improvement by resizing pictures. And it work great ! Execution time is about 1 second, and my other code to clean results is also about 1 second. (If you execute the project, you will see in logs execution time) But I've planned to improve it more. Thanks for your questions, and do not hesitate if you have others or suggestions :) P.S: It's a very good project for my second project in Python ! :) #### #23 Updated by zedtux - over 3 years ago Hi all, After 2 commits, I've decided to write a little documentation about how to test my project. You will find it on the page of the project on Launchpad: https://launchpad.net/quiestla If you find bugs or have any questions, please Launchpad tools for that, thanks ! :) #### #24 Updated by zedtux - over 3 years ago I'm asking me a question about faces areas positions and associated name storage. I've browsed the SQLite Database, and I was thinking that we should have to create a new table like FaceTable that should be like this: CREATE TABLE FaceTable (id INTEGER PRIMARY KEY, primary_photo_id INTEGER, name TEXT, x INTEGER, y INTEGER, width INTEGER, height INTEGER) Am I wrong ? #### #25 Updated by Jim Nelson over 3 years ago This looks reasonable, but stepping back, what are you proposing as far as implementing this? I have an idea of what you're storing in the FaceTable, but how does this interact with the rest of the system (i.e. PhotoTable)? #### #26 Updated by zedtux - over 3 years ago The idea is quite simple: I've create an external project that do all that Shotwell need to know where are faces. So, after importing a folder of photos, in order to show faces and associated names, a FaceManager should connect through DBus to my external project, given the path of pictures where faces should be found. Then, my project emit a signal when stuff is done for a photo, returning the result (faces areas). When Shotwell will receive that result, it should store then in a row of the FaceTable table, containing x and y positions and width and height. More after, the user should update a face area to give the name of the detected face. In the future, I hope to implement the face detection feature to recognize a face and give it's name directly. If you have future questions, do not hesitate to ask me ! :) #### #27 Updated by zedtux - about 3 years ago Hi all, I'm currently facing a problem that stop me in dev. I've start to write a Dbus class like this: [DBus (name = "org.zedroot.QuiEstLa.PeopleLocalisation")] private class PeopleLocalisation : GLib.Object { public signal void a_file_detection_finished (Results[] results); } But the compilation failed with this error messages: > cc -c !`pkg-config --cflags atk gdk-2.0 gee-1.0 gtk+-2.0 glib-2.0 libexif sqlite3 gexiv2 gconf-2.0 libgphoto2 libsoup-2.4 libxml-2.0 unique-1.0 webkit-1.0 gudev-1.0 dbus-glib-1 gdk-x11-2.0 gthread-2.0` -I./vapi -D_PREFIX='“/usr/local”' -D_VERSION='“0.7.1+trunk”' -DGETTEXT_PACKAGE='“shotwell”' -D_LANG_SUPPORT_DIR='“/usr/local/share/locale”' !`./libraw-config --cflags` -O2 -g -pipe -fPIC -DG_UDEV_API_IS_SUBJECT_TO_CHANGE -o src/!FaceManager.o src/FaceManager.c FaceManager.c: In function face_manager_finalize: FaceManager.c:635: error: _tmp1_ undeclared (first use in this function) FaceManager.c:635: error: (Each undeclared identifier is reported only once FaceManager.c:635: error: for each function it appears in.) make: *** [src/!FaceManager.o] Erreur 1 Compilationwork fine if I remove the line [DBus (name = "org.zedroot.QuiEstLa.PeopleLocalisation")] What's wrong ? P.S: I've alsotriedexample like[those](http://live.gnome.org/Vala/DBusClientSamplesand) compilation neverfail. #### #28 Updated by Jim Nelson about 3 years ago I'm not sure what the problem is here, but it appears to be with the DBus implementation and not with Shotwell. I would recommend asking your question on the Vala mailing list or the Vala IRC channel. #### #29 Updated by zedtux - about 3 years ago Hello all, Since this issue, I haven't worked on because here priorities have changed. I don't want to give up. So I suggest that someone that have a biggest knowledge than me in vala write the vala code to send requests to my [QuiEstLa](http://trac.yorba.org/search?q=wiki%3AQuiEstLa) service. I'm available for help, and will watch this ticket to answer to all questions and do quick modifications to my project if necessary. Please, do not hesitate to request my help, and I will take time to continue to work on it. #### #30 Updated by Gunter Königsmann about 3 years ago If the descriptions found under http://launchpad.net/quiestla are correct quiestla still isn't very fast but is already usable… #### #31 Updated by Arthur Lutz about 3 years ago Am interested in this feature too, and am ready to help out testing (at least). #### #32 Updated by Jim Nelson almost 3 years ago Requested downstream: https://bugs.launchpad.net/ubuntu/source/shotwell/bug/693808 #### #33 Updated by Anonymous almost 3 years ago Replying to [comment:27 zedtux]: > I'm asking me a question about faces areas positions and associated name storage. > > I've browsed the SQLite Database, and I was thinking that we should have to create a new table like FaceTable that should be like this: > > > > > %(=caps)CREATE% %(=caps)TABLE% FaceTable (id %(=caps)INTEGER% %(=caps)PRIMARY% %(=caps)KEY%, primary_photo_id %(=caps)INTEGER%, name %(=caps)TEXT%, x %(=caps)INTEGER%, y %(=caps)INTEGER%, width %(=caps)INTEGER%, height %(=caps)INTEGER%) > > > > > > Am I wrong ? I suggest RegionTable. A region may be used for something other than a face. RegionTable will have a field tag_id (not “name”), so a region could be associated with a tag. A tag may represent a person. #### #34 Updated by Lucas Beeler over 2 years ago * **Subject** changed from _Face detection and people view_ to _Face detection_ * **Description** updated (diff) * **Assignee** deleted (_zedtux -_) * **Priority** changed from _Low_ to _Normal_ #### #35 Updated by Lucas Beeler over 2 years ago This ticket was titled "Face detection and people view," but has been retitled "Face detection" because the "people view" functionality has been factored out into ticket #3894. "People view" is the ability to manage faces/people in Shotwell, while "Face detection" is the ability to automatically recognize faces using a pattern recognition/computer vision library. #### #36 Updated by Jim Nelson about 2 years ago * **Description** updated (diff) * **Priority** changed from _Normal_ to _High_ * **Target version** set to _0.12_ We'd like to finish up Faces work for 0.12 and get this feature out there. #### #37 Updated by Valentín Barros about 2 years ago * **Description** updated (diff) Hello! Due to work/studies I have no time to keep actively contributing, but I have two work-in-progress patches (one for this bug and the other for #3897) that I will update and share them with you tomorrow or this Monday, maybe you could use them. #### #38 Updated by Adam Dingle about 2 years ago Thanks, Valentin - we'll look forward to your patches. #### #39 Updated by Valentín Barros about 2 years ago I'm trying to generate the patches but they are created on top of previous modifications I did in my local repository to create patches that have not been revised yet, so if you try to apply them to Shotwell master they will fail. If you don't mind, I prefer to wait until you review #3910, #3916, #3939 and #3991 (listed in the order I've sent them) to fix my local branch and get the correct patches. #### #40 Updated by Adam Dingle about 2 years ago * **Subject** changed from _Face detection_ to _[faces] Face detection_ #### #41 Updated by Frederik Sdun almost 2 years ago I'm working on this, too. My code is based on: https://github.com/Sanva/OpenCV-test-programs I only need to add the Boxes detected by OpenCV. Is there any open source tool which is able to do face recognition and I can read some code? #### #42 Updated by zedtux - almost 2 years ago So no chance to use a decentralized system like my QuiEstLa project (https://launchpad.net/quiestla) instead of creating multiple implementations? #### #43 Updated by Frederik Sdun almost 2 years ago I could use your daemon instead if the shotwell devs are ok with it. I need a reply soon, because I don't want to invest time on sth which will be rejected. #### #44 Updated by zedtux - almost 2 years ago Agree. If the Shotwell project want to implement it, I will be available for fixes and improvements. To give a quick status of the development (stopped since a long time ago because nobody was interested by the project), QuiEstLa is only able to locate faces for now, not recognize faces. #### #45 Updated by Adam Dingle almost 2 years ago As I mentioned in comments 8, 11 and 13 above, we'd prefer to have Shotwell call OpenCV in process rather than by accessing an external service over DBus. zedtux, I know that you've been working on a DBus service, but in comment #15 you wrote "After that, I can write the same code, but directly in Shotwell in Vala." I thought that "the same code" meant both the DBus server and client code, so that DBus would no longer be used. Perhaps you meant only the DBus client code and were intending that Shotwell would still talk to a DBus server. If so, we had a misunderstanding. Again, if someone can point out some technical advantage of using a DBus service over a shared library then we could consider it. But so far I haven't heard anyone say what the advantage would be. #### #46 Updated by Adam Dingle almost 2 years ago * **Target version** deleted (_0.12_) I'm unmarking this for 0.12 since we're just a few weeks from feature freeze and it seems there's more work than that to be done. (We could still consider this for 0.12 if the work were complete by about February 1, though.) #### #47 Updated by zedtux - almost 2 years ago **@adam:** When I said "After that, I can write the same code, but directly in Shotwell in Vala." it means that I was thinking of writing the same code (to detect faces locations) in Vala into Shotwell (as you're expecting). I tried to do it, but Vala is a bit hard to learn. (I wrote C++ code years ago, and I'm not Java/C#/all other complex code syntax developer). So, today I'm not going to implement it into Shotwell myself. I'll could do it if someone can give me some screencasts or good guides to learn Vala. In the other hand, I would like to know why you don't want to use a DBus service? (Just to understand your point). #### #48 Updated by Adam Dingle almost 2 years ago @zedtux: See my comments #8 and #13 above. Calling a shared library is simpler and more efficient than using inter-process communication. It sounds to me like you may want to use a separate process because you don't want to write code in Vala. I think that the decision whether to use a shared library or separate process should be made for architectural reasons not related to a choice of programming language. Also, if you've coded C++ in the past then I don't think Vala should be so hard to learn - it's a simpler language. There's unfortunately no Vala book yet, but the Vala web site has plenty of sample code and tutorials. #### #49 Updated by zedtux - almost 2 years ago @adam: The first reason I suggested a DBus service is because it could be used by any other applications, not only Shotwell, and the the second one is like you said to don't write in Vala. But before to try Vala, I had in my mind to implement this DBus service in Shotwell in Vala. (Write in Shotwell a client code) #### #50 Updated by Adam Dingle almost 2 years ago @zedtux: As I pointed out in comment #13 above, a shared library can also be used by any other application, not only Shotwell. I don't see that a DBus service is any different in this respect. #### #51 Updated by Jim Nelson 10 months ago * **Category** set to _faces_ #### #52 Updated by Alexander Menk 3 months ago Is there any progress on that? Or any other ways to automatically tag images and then re-import such tags to shotwell? #### #53 Updated by Jim Nelson 3 months ago Work on Faces is currently on hold. --- Bug imported by chaz@yorba.org 2013-11-25 21:43 UTC --- This bug was previously known as _bug_ 1702 at http://redmine.yorba.org/show_bug.cgi?id=1702 Unknown version " in product shotwell. Setting version to "!unspecified". Unknown milestone "unknown in product shotwell. Setting to default milestone for this product, "---". Setting qa contact to the default for this product. This bug either had no qa contact or an invalid one. Resolution set on an open status. Dropping resolution