Styjun

I'm trying to detect faces in my iOS camera app, but it doesn't work properly, while it works properly in Camera.app. Notice that:

• The first face isns't detected in my app, only in Camera.app.


• For the third face — the east asian woman — Camera.app correctly draws a rectangle around her face, while my app draws a rectangle that extends far below her face.


• Obama's face isn't detected in my app, only in Camera.app.


• When the camera zooms out from Putin's face, my app draws a rectangle over the right half of his face, cutting it in half, while Camera.app draws a rectangle correctly around his face.

Why is this happening?

My code is as follows. Do you see anything wrong?

First, I create a video output as follows:

let videoOutput = AVCaptureVideoDataOutput()
videoOutput.videoSettings =
 [kCVPixelBufferPixelFormatTypeKey as AnyHashable: 
 Int(kCMPixelFormat_32BGRA)]

session.addOutput(videoOutput)

videoOutput.setSampleBufferDelegate(faceDetector, queue: faceDetectionQueue)

This is the delegate:

class FaceDetector: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate 
 func captureOutput(_ captureOutput: AVCaptureOutput!,
 didOutputSampleBuffer sampleBuffer: CMSampleBuffer!,
 from connection: AVCaptureConnection!) 
 let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
 let features = FaceDetector.ciDetector.features(
 in: CIImage(cvPixelBuffer: imageBuffer))

 let faces = features.map $0.bounds 
 let imageSize = CVImageBufferGetDisplaySize(imageBuffer)

 let faceBounds = faces.map (face: CIFeature) -> CGRect in
 var ciBounds = face.bounds

 ciBounds = ciBounds.applying(
 CGAffineTransform(scaleX: 1/imageSize.width, y: -1/imageSize.height))
 CGRect(x: 0, y: 0, width: 1, height: -1).verifyContains(ciBounds)

 let bounds = ciBounds.applying(CGAffineTransform(translationX: 0, y: 1.0))
 CGRect(x: 0, y: 0, width: 1, height: 1).verifyContains(bounds)
 return bounds
 
 DispatchQueue.main.sync 
 facesUpdated(faceBounds, imageSize)
 
 

 private static let ciDetector = CIDetector(ofType: CIDetectorTypeFace,
 context: nil,
 options: [CIDetectorAccuracy: CIDetectorAccuracyHigh])!

The facesUpdated() callback is as follows:

class PreviewView: UIView 
 private var faceRects = [UIView]()

 private static func makeFaceRect() -> UIView 
 let r = UIView()
 r.layer.borderWidth = FocusRect.borderWidth
 r.layer.borderColor = FocusRect.color.cgColor
 faceRects.append(r)
 addSubview(r)
 return r
 

 private func removeAllFaceRects() 
 for faceRect in faceRects 
 verify(faceRect.superview == self)
 faceRect.removeFromSuperview()
 
 faceRects.removeAll()
 

 private func facesUpdated(_ faces: [CGRect], _ imageSize: CGSize) 
 removeAllFaceRects()

 let faceFrames = faces.map (original: CGRect) -> CGRect in
 let face = original.applying(CGAffineTransform(scaleX: bounds.width, y: bounds.height))
 verify(self.bounds.contains(face))
 return face
 

 for faceFrame in faceFrames 
 let faceRect = PreviewView.makeFaceRect()
 faceRect.frame = faceFrame
 
 

I also tried the following, but they didn't help:

1. Setting the AVCaptureVideoDataOutput's videoSettings to nil.


2. Explicitly setting the CIDetector's orientation to portrait. The phone is in portrait for this test, so it shouldn't matter.


3. Setting and removing CIDetectorTracking: true


4. Setting and removing CIDetectorAccuracy: CIDetectorAccuracyHigh


5. Trying to track only one face, by looking only at the first feature detected.


6. Replacing CVImageBufferGetDisplaySize() with CVImageBufferGetEncodedSize() — they're anyway same, at 1440 x 1080.