MediaPipe 적용

[zozero94]

MediaPipe
MediaPipeGit
BazelTutorial
BazelKotlin

진행 순서

• Bazel 학습
• Bazel 을 통해 안드로이드 앱 빌드
• MediaPipe에 사용된 Bazel 파악하기
• MediaPipe 샘플 실행(Android)
• 개인앱 위에 MediaPipe 올리기

[zozero94]

command-Line-Build

Build

bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu:handtrackinggpu

Install

adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu/handtrackinggpu.apk

[zozero94]

환경변수

export ANDROID_HOME=/Users/zero/Library/Android/sdk
export ANDROID_NDK_HOME=/Users/zero/Library/Android/sdk/ndk/20.0.5594570

[zozero94]

Bazel

WORKSPACE

Bazel의 기본 명세서같은 존재이며 프로젝트 최 상단에 위치해야 한다.

BUILD

BULD 파일은 apt 또는 javac의 클래스 파일로부터 컴파일된 Android 리소스와 같은 빌드 출력 집합과 그 종속성 사이의 관계를 설명한다.
이러한 종속성은 작업영역의 소스 파일(Java, C++) 또는 기타 빌드 출력일 수 있다. BUILD 파일은 Starlark라는 언어로 쓰여져 있다.

android_library

앱 소스 코드와 리소스 파일에서 Android 라이브러리 모듈을 구축하도록 명시한다.

android_binary

Android 애플리케이션 패키지(.apk 파일)를 구축한다.

빌드

현재 디렉토리에서 BUILD에 지정한 android_binary의 name을 호출한다.

bazel build //app/src/main:MyApllication

.DS_Store 파일로 인한 오류같음

파인더에서 접속할때마다 생성된다고 하는데, 폴더의 위치나, 아이콘, 배경화면등의 설정을 담는다고 한다.

다 지워버리기

$sudo find / -name .DS_Store -print -delete

[zozero94]

여러 디펜던시를 엮은 프로젝트를 빌드하려하니 고려할 사항이 많음.
완전 기초 프로젝트를 기반으로 BazelBuild를 다시 해볼것.

• BaseProject Bazel Build (코틀린 디펜던시 고려할 것)
• AAR build

[zozero94]

AAR 빌드 성공

• AndroidAssertUtil Import 작업 import com.google.mediapipe.framework.AndroidAssetUtil;

[zozero94]

[zozero94]

MediaPipe 자체에 모듈화가 너무 많이되어있어 안드로이드 API를 가지고 사용하기 버겁다.

예를들어 카메라 핵심로직을 다 모듈화해두어서 커스텀 할수가 없고 문서가 부족하다..

mediapipe 를 대상으로 만들어진 BlazePose가 릴리즈 되었고 개발자 커뮤니티가 존재하는 MLkit 을 사용해보자.

• MLKit 프로젝트 적용
• 나오는 결과물이 어떻게 찍히나 확인
• 사용 가능 API 정리
• 내부 Sdk와 결합 할 수 있는지 확인
• 해당 리포트 작성

[zozero94]

MLkit Version은 요구사항에 맞는 최저 프레임 (30FPS)에 맞추기 위해 pose-detection sdk를 사용

[zozero94]

샘플앱을 통하여 내부 구조 학습

• LivePreviewActivity 와 CameraXLivePreviewActivity 차이 구분하기

Camera Preview 를 사용하느냐, CameraX Preview를 사용하느냐의 차이

• Enable Live ViewPort 역할 찾아내기

Toggle between blocking camera preview by API processing and result rendering or not

• 화면에 어떻게 그리는지 파악하기

받아온 좌표정보를 통해 화면에 직접 그려주는 형식

[zozero94]

샘의 모델을 적용해보자

• AAR 빌드 정리
• graph 파일 정리
• pbtxt 파일 정리
• example BUILD 정리

[zozero94]

AAR 빌드 정리

• graph(.pbtxt)를 지정해서 binarypb파일을 뽑아내기 위함

1. Example/.../apps/aar_example/BUILD에 정의

   load("//mediapipe/java/com/google/mediapipe:mediapipe_aar.bzl", "mediapipe_aar")
   
   mediapipe_aar(
       name = "mp_person_segmentation_aar",
       calculators = ["//mediapipe/graphs/person_segmentation:mobile_calculators"],
   )
   

   • name : 빌드시 명시해줄 이름
   • calculators : 빌드할 때, 사용할 graph 정보

   # //mediapipe/graphs/person_segmentation/BUILD"
   mediapipe_binary_graph(
       name = "mobile_gpu_binary_graph",
       # 연결할 그래프 (같은 paht에 있음)
       graph = "hair_segmentation_mobile_gpu.pbtxt", 
       # 뽑아 낼 결과물 이름
       output_name = "mobile_gpu.binarypb", 
       deps = [":mobile_calculators"],
   )

2. AAR을 빌드한다.

   bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a \
   //mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example:mp_person_segmentation_aar

   • 추가 옵션 (aar 용량 줄이기)

     bazel build -c  opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a -- 
     linkopt="-s"\  //mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example:mp_person_segmentation_aar

[zozero94]

Graph

1. Android app/libs 생성 후 aar 복사

2. app/assets 에 들어갈 파일 빌드
   //mediapipe/graphs/person_segmentation
   path에 아래 두 파일 넣고 빌드

   1. person_segmentation_mobile_gpu.pbtxt

# MediaPipe graph that performs hair segmentation with TensorFlow Lite on CPU.
# Used in the example in
# mediapipie/examples/desktop/hair_segmentation:hair_segmentation_cpu

# Images on CPU coming into and out of the graph.
input_stream: "input_video"
output_stream: "output_video"

# Throttles the images flowing downstream for flow control. It passes through
# the very first incoming image unaltered, and waits for
# TfLiteTensorsToSegmentationCalculator downstream in the graph to finish
# generating the corresponding hair mask before it passes through another
# image. All images that come in while waiting are dropped, limiting the number
# of in-flight images between this calculator and
# TfLiteTensorsToSegmentationCalculator to 1. This prevents the nodes in between
# from queuing up incoming images and data excessively, which leads to increased
# latency and memory usage, unwanted in real-time mobile applications. It also
# eliminates unnecessarily computation, e.g., a transformed image produced by
# ImageTransformationCalculator may get dropped downstream if the subsequent
# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
# processing previous inputs.
node {
  calculator: "FlowLimiterCalculator"
  input_stream: "input_video"
  input_stream: "FINISHED:hair_mask"
  input_stream_info: {
    tag_index: "FINISHED"
    back_edge: true
  }
  output_stream: "throttled_input_video"
}

# Transforms the input image on CPU to a 512x512 image. To scale the image, by
# default it uses the STRETCH scale mode that maps the entire input image to the
# entire transformed image. As a result, image aspect ratio may be changed and
# objects in the image may be deformed (stretched or squeezed), but the hair
# segmentation model used in this graph is agnostic to that deformation.
node: {
  calculator: "ImageTransformationCalculator"
  input_stream: "IMAGE:throttled_input_video"
  output_stream: "IMAGE:transformed_input_video"
  node_options: {
    [type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
      output_width: 512
      output_height: 512
    }
  }
}

# Caches a mask fed back from the previous round of hair segmentation, and upon
# the arrival of the next input image sends out the cached mask with the
# timestamp replaced by that of the input image, essentially generating a packet
# that carries the previous mask. Note that upon the arrival of the very first
# input image, an empty packet is sent out to jump start the feedback loop.
node {
  calculator: "PreviousLoopbackCalculator"
  input_stream: "MAIN:throttled_input_video"
  input_stream: "LOOP:hair_mask"
  input_stream_info: {
    tag_index: "LOOP"
    back_edge: true
  }
  output_stream: "PREV_LOOP:previous_hair_mask"
}

# # Embeds the hair mask generated from the previous round of hair segmentation
# # as the alpha channel of the current input image.
# node {
#   calculator: "SetAlphaCalculator"
#   input_stream: "IMAGE:transformed_input_video"
#   input_stream: "ALPHA:previous_hair_mask"
#   output_stream: "IMAGE:mask_embedded_input_video"
# }

# Converts the transformed input image on CPU into an image tensor stored in
# TfLiteTensor. The zero_center option is set to false to normalize the
# pixel values to [0.f, 1.f] as opposed to [-1.f, 1.f]. With the
# max_num_channels option set to 4, all 4 RGBA channels are contained in the
# image tensor.
node {
  calculator: "TfLiteConverterCalculator"
  # input_stream: "IMAGE:mask_embedded_input_video"
  input_stream: "IMAGE:transformed_input_video"
  output_stream: "TENSORS:image_tensor"
  node_options: {
    [type.googleapis.com/mediapipe.TfLiteConverterCalculatorOptions] {
      zero_center: false
      # max_num_channels: 4
      max_num_channels: 3
    }
  }
}

# Generates a single side packet containing a TensorFlow Lite op resolver that
# supports custom ops needed by the model used in this graph.
node {
  calculator: "TfLiteCustomOpResolverCalculator"
  output_side_packet: "op_resolver"
  node_options: {
    [type.googleapis.com/mediapipe.TfLiteCustomOpResolverCalculatorOptions] {
      use_gpu: false
    }
  }
}

# Runs a TensorFlow Lite model on CPU that takes an image tensor and outputs a
# tensor representing the hair segmentation, which has the same width and height
# as the input image tensor.
node {
  calculator: "TfLiteInferenceCalculator"
  input_stream: "TENSORS:image_tensor"
  output_stream: "TENSORS:segmentation_tensor"
  input_side_packet: "CUSTOM_OP_RESOLVER:op_resolver"
  node_options: {
    [type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
      # model_path: "mediapipe/models/person_segmentation_4ch.tflite"
      model_path: "mediapipe/models/person_segmentation_3ch.tflite"

      # for developing, only
      # model_path: "mediapipe/models/temp.tflite"  
      use_gpu: false
    }
  }
}

# Decodes the segmentation tensor generated by the TensorFlow Lite model into a
# mask of values in [0, 255], stored in a CPU buffer. It also
# takes the mask generated previously as another input to improve the temporal
# consistency.
node {
  calculator: "TfLiteTensorsToSegmentationCalculator"
  input_stream: "TENSORS:segmentation_tensor"
  input_stream: "PREV_MASK:previous_hair_mask"
  output_stream: "MASK:hair_mask"
  node_options: {
    [type.googleapis.com/mediapipe.TfLiteTensorsToSegmentationCalculatorOptions] {
      tensor_width: 512
      tensor_height: 512
      tensor_channels: 2
      combine_with_previous_ratio: 0.9
      # combine_with_previous_ratio: 1.0
      output_layer_index: 1
    }
  }
}

# Colors the hair segmentation with the color specified in the option.
node {
  calculator: "RecolorCalculator"
  input_stream: "IMAGE:throttled_input_video"
  input_stream: "MASK:hair_mask"
  output_stream: "IMAGE:output_video"
  node_options: {
    [type.googleapis.com/mediapipe.RecolorCalculatorOptions] {
      color { r: 0 g: 0 b: 255 }
      mask_channel: RED
    }
  }
}
      

2. BUILD

load(
    "//mediapipe/framework/tool:mediapipe_graph.bzl",
    "mediapipe_binary_graph",
)

licenses(["notice"])

package(default_visibility = ["//visibility:public"])

cc_library(
    name = "mobile_calculators",
    deps = [
        "//mediapipe/calculators/core:flow_limiter_calculator",
        "//mediapipe/calculators/core:previous_loopback_calculator",
        "//mediapipe/calculators/image:image_transformation_calculator",
        "//mediapipe/calculators/image:recolor_calculator",
        "//mediapipe/calculators/image:set_alpha_calculator",
        "//mediapipe/calculators/tflite:tflite_converter_calculator",
        "//mediapipe/calculators/tflite:tflite_custom_op_resolver_calculator",
        "//mediapipe/calculators/tflite:tflite_inference_calculator",
        "//mediapipe/calculators/tflite:tflite_tensors_to_segmentation_calculator",
        "//mediapipe/gpu:gpu_buffer_to_image_frame_calculator",
        "//mediapipe/gpu:image_frame_to_gpu_buffer_calculator",
    ],
)

cc_library(
    name = "desktop_calculators",
    deps = [
        "//mediapipe/calculators/core:flow_limiter_calculator",
        "//mediapipe/calculators/core:previous_loopback_calculator",
        "//mediapipe/calculators/image:image_transformation_calculator",
        "//mediapipe/calculators/image:recolor_calculator",
        "//mediapipe/calculators/image:set_alpha_calculator",
        "//mediapipe/calculators/tflite:tflite_converter_calculator",
        "//mediapipe/calculators/tflite:tflite_custom_op_resolver_calculator",
        "//mediapipe/calculators/tflite:tflite_inference_calculator",
        "//mediapipe/calculators/tflite:tflite_tensors_to_segmentation_calculator",
    ],
)

mediapipe_binary_graph(
    name = "mobile_gpu_binary_graph",
    graph = "person_segmentation_mobile_gpu.pbtxt",
    output_name = "mobile_gpu.binarypb",
    deps = [":mobile_calculators"],
)

3. 빌드

   bazel build -c opt //mediapipe/graphs/person_segmentation:mobile_gpu_binary_graph

4. 이동 (path to android asset folder)
   bazel-bin/mediapipe/graphs/face_detection/mobile_gpu.binarypb
   mediapipe/models/face_detection_front.tflite

[zozero94]

file.1.mp4

file.mp4