Python Project - Emojify - Create your own emoji with Deep Learning
Posted by Superadmin on August 22 2020 19:10:17

Emojify - Create your own emoji with Deep Learning

BY  · UPDATED · AUGUST 6, 2020

Deep Learning project for beginners – Taking you closer to your Data Science dream

Emojis or avatars are ways to indicate nonverbal cues. These cues have become an essential part of online chatting, product review, brand emotion, and many more. It also lead to increasing data science research dedicated to emoji-driven storytelling.

With advancements in computer vision and deep learning, it is now possible to detect human emotions from images. In this deep learning project, we will classify human facial expressions to filter and map corresponding emojis or avatars.

About the Dataset

The FER2013 dataset ( facial expression recognition) consists of 48*48 pixel grayscale face images. The images are centered and occupy an equal amount of space. This dataset consist of facial emotions of following categories:

Download Dataset: Facial Expression Recognition Dataset

Download Project Code

Before proceeding ahead, please download the source code: Emoji Creator Project Source Code

Create your emoji with Deep Learning

create emoji with deep learning

We will build a deep learning model to classify facial expressions from the images. Then we will map the classified emotion to an emoji or an avatar.

Facial Emotion Recognition using CNN

In the below steps will build a convolution neural network architecture and train the model on FER2013 dataset for Emotion recognition from images.

Download the dataset from the above link. Extract it in the data folder with separate train and test directories.

Make a file train.py and follow the steps:

1. Imports:

  1. import numpy as np
  2. import cv2
  3.  
  4. from keras.emotion_models import Sequential
  5. from keras.layers import Dense, Dropout, Flatten
  6. from keras.layers import Conv2D
  7. from keras.optimizers import Adam
  8. from keras.layers import MaxPooling2D
  9. from keras.preprocessing.image import ImageDataGenerator

2. Initialize the training and validation generators:

  1. train_dir = 'data/train' 
  2. val_dir = 'data/test' 
  3. train_datagen = ImageDataGenerator(rescale=1./255) 
  4. val_datagen = ImageDataGenerator(rescale=1./255) 
  5.  
  6. train_generator = train_datagen.flow_from_directory( 
  7. train_dir,
  8. target_size=(48,48),
  9. batch_size=64,
  10. color_mode="gray_framescale",
  11. class_mode='categorical')
  12. validation_generator = val_datagen.flow_from_directory(
  13. val_dir,
  14. target_size=(48,48),
  15. batch_size=64,
  16. color_mode="gray_framescale",
  17. class_mode='categorical')

3. Build the convolution network architecture:

  1. emotion_model = Sequential()
  2. emotion_model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=(48,48,1)))
  3. emotion_model.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))
  4. emotion_model.add(MaxPooling2D(pool_size=(2, 2)))
  5. emotion_model.add(Dropout(0.25))
  6. emotion_model.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))
  7. emotion_model.add(MaxPooling2D(pool_size=(2, 2)))
  8. emotion_model.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))
  9. emotion_model.add(MaxPooling2D(pool_size=(2, 2)))
  10. emotion_model.add(Dropout(0.25))
  11. emotion_model.add(Flatten())
  12. emotion_model.add(Dense(1024, activation='relu'))
  13. emotion_model.add(Dropout(0.5))
  14. emotion_model.add(Dense(7, activation='softmax'))

4. Compile and train the model:

  1. emotion_model.compile(loss='categorical_crossentropy',optimizer=Adam(lr=0.0001, decay=1e-6),metrics=['accuracy'])
  2. emotion_model_info = emotion_model.fit_generator(
  3. train_generator,
  4. steps_per_epoch=28709 // 64,
  5. epochs=50,
  6. validation_data=validation_generator,
  7. validation_steps=7178 // 64)

5. Save the model weights:

  1. emotion_model.save_weights('model.h5')

6. Using openCV haarcascade xml detect the bounding boxes of face in the webcam and predict the emotions:

  1. cv2.ocl.setUseOpenCL(False)
  2. emotion_dict = {0: "Angry", 1: "Disgusted", 2: "Fearful", 3: "Happy", 4: "Neutral", 5: "Sad", 6: "Surprised"}
  3. cap = cv2.VideoCapture(0)
  4. while True:
  5. ret, frame = cap.read()
  6. if not ret:
  7. break
  8. bounding_box = cv2.CascadeClassifier('/home/shivam/.local/lib/python3.6/site-packages/cv2/data/haarcascade_frontalface_default.xml')
  9. gray_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2gray_frame)
  10. num_faces = bounding_box.detectMultiScale(gray_frame,scaleFactor=1.3, minNeighbors=5)
  11. for (x, y, w, h) in num_faces:
  12. cv2.rectangle(frame, (x, y-50), (x+w, y+h+10), (255, 0, 0), 2)
  13. roi_gray_frame = gray_frame[y:y + h, x:x + w]
  14. cropped_img = np.expand_dims(np.expand_dims(cv2.resize(roi_gray_frame, (48, 48)), -1), 0)
  15. emotion_prediction = emotion_model.predict(cropped_img)
  16. maxindex = int(np.argmax(emotion_prediction))
  17. cv2.putText(frame, emotion_dict[maxindex], (x+20, y-60), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2, cv2.LINE_AA)
  18. cv2.imshow('Video', cv2.resize(frame,(1200,860),interpolation = cv2.INTER_CUBIC))
  19. if cv2.waitKey(1) & 0xFF == ord('q'):
  20. cap.release()
  21. cv2.destroyAllWindows()
  22. break

Code for GUI and mapping with emojis

Create a folder named emojis and save the emojis corresponding to each of the seven emotions in the dataset.

Paste the below code in gui.py and run the file.

  1. import tkinter as tk
  2. from tkinter import *
  3. import cv2
  4. from PIL import Image, ImageTk
  5. import os
  6. import numpy as np
  7. import cv2
  8. from keras.models import Sequential
  9. from keras.layers import Dense, Dropout, Flatten
  10. from keras.layers import Conv2D
  11. from keras.optimizers import Adam
  12. from keras.layers import MaxPooling2D
  13. from keras.preprocessing.image import ImageDataGenerator
  14. emotion_model = Sequential()
  15. emotion_model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=(48,48,1)))
  16. emotion_model.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))
  17. emotion_model.add(MaxPooling2D(pool_size=(2, 2)))
  18. emotion_model.add(Dropout(0.25))
  19. emotion_model.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))
  20. emotion_model.add(MaxPooling2D(pool_size=(2, 2)))
  21. emotion_model.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))
  22. emotion_model.add(MaxPooling2D(pool_size=(2, 2)))
  23. emotion_model.add(Dropout(0.25))
  24. emotion_model.add(Flatten())
  25. emotion_model.add(Dense(1024, activation='relu'))
  26. emotion_model.add(Dropout(0.5))
  27. emotion_model.add(Dense(7, activation='softmax'))
  28. emotion_model.load_weights('model.h5')
  29. cv2.ocl.setUseOpenCL(False)
  30. emotion_dict = {0: " Angry ", 1: "Disgusted", 2: " Fearful ", 3: " Happy ", 4: " Neutral ", 5: " Sad ", 6: "Surprised"}
  31. emoji_dist={0:"./emojis/angry.png",2:"./emojis/disgusted.png",2:"./emojis/fearful.png",3:"./emojis/happy.png",4:"./emojis/neutral.png",5:"./emojis/sad.png",6:"./emojis/surpriced.png"}
  32. global last_frame1
  33. last_frame1 = np.zeros((480, 640, 3), dtype=np.uint8)
  34. global cap1
  35. show_text=[0]
  36. def show_vid():
  37. cap1 = cv2.VideoCapture(0)
  38. if not cap1.isOpened():
  39. print("cant open the camera1")
  40. flag1, frame1 = cap1.read()
  41. frame1 = cv2.resize(frame1,(600,500))
  42. bounding_box = cv2.CascadeClassifier('/home/shivam/.local/lib/python3.6/site-packages/cv2/data/haarcascade_frontalface_default.xml')
  43. gray_frame = cv2.cvtColor(frame1, cv2.COLOR_BGR2GRAY)
  44. num_faces = bounding_box.detectMultiScale(gray_frame,scaleFactor=1.3, minNeighbors=5)
  45. for (x, y, w, h) in num_faces:
  46. cv2.rectangle(frame1, (x, y-50), (x+w, y+h+10), (255, 0, 0), 2)
  47. roi_gray_frame = gray_frame[y:y + h, x:x + w]
  48. cropped_img = np.expand_dims(np.expand_dims(cv2.resize(roi_gray_frame, (48, 48)), -1), 0)
  49. prediction = emotion_model.predict(cropped_img)
  51. maxindex = int(np.argmax(prediction))
  52. cv2.putText(frame1, emotion_dict[maxindex], (x+20, y-60), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2, cv2.LINE_AA)
  53. show_text[0]=maxindex
  54. if flag1 is None:
  55. print ("Major error!")
  56. elif flag1:
  57. global last_frame1
  58. last_frame1 = frame1.copy()
  59. pic = cv2.cvtColor(last_frame1, cv2.COLOR_BGR2RGB)
  60. img = Image.fromarray(pic)
  61. imgtk = ImageTk.PhotoImage(image=img)
  62. lmain.imgtk = imgtk
  63. lmain.configure(image=imgtk)
  64. lmain.after(10, show_vid)
  65. if cv2.waitKey(1) & 0xFF == ord('q'):
  66. exit()
  67. def show_vid2():
  68. frame2=cv2.imread(emoji_dist[show_text[0]])
  69. pic2=cv2.cvtColor(frame2,cv2.COLOR_BGR2RGB)
  70. img2=Image.fromarray(frame2)
  71. imgtk2=ImageTk.PhotoImage(image=img2)
  72. lmain2.imgtk2=imgtk2
  73. lmain3.configure(text=emotion_dict[show_text[0]],font=('arial',45,'bold'))
  75. lmain2.configure(image=imgtk2)
  76. lmain2.after(10, show_vid2)
  77. if __name__ == '__main__':
  78. root=tk.Tk()
  79. img = ImageTk.PhotoImage(Image.open("logo.png"))
  80. heading = Label(root,image=img,bg='black')
  82. heading.pack()
  83. heading2=Label(root,text="Photo to Emoji",pady=20, font=('arial',45,'bold'),bg='black',fg='#CDCDCD')
  85. heading2.pack()
  86. lmain = tk.Label(master=root,padx=50,bd=10)
  87. lmain2 = tk.Label(master=root,bd=10)
  88. lmain3=tk.Label(master=root,bd=10,fg="#CDCDCD",bg='black')
  89. lmain.pack(side=LEFT)
  90. lmain.place(x=50,y=250)
  91. lmain3.pack()
  92. lmain3.place(x=960,y=250)
  93. lmain2.pack(side=RIGHT)
  94. lmain2.place(x=900,y=350)
  96. root.title("Photo To Emoji")
  97. root.geometry("1400x900+100+10")
  98. root['bg']='black'
  99. exitbutton = Button(root, text='Quit',fg="red",command=root.destroy,font=('arial',25,'bold')).pack(side = BOTTOM)
  100. show_vid()
  101. show_vid2()
  102. root.mainloop()

 

Summary

In this deep learning project for beginners, we have built a convolution neural network to recognize facial emotions. We have trained our model on the FER2013 dataset. Then we are mapping those emotions with the corresponding emojis or avatars.

Using OpenCV’s haar cascade xml we are getting the bounding box of the faces in the webcam. Then we feed these boxes to the trained model for classification.